Topology Spread

本节我们将使用AZ topologySpreadConstraints来将Pod分布在不同的AZ

在k8s里，可以使用topology spread来保证高可用，例如将pod分布在不同的AZ、Region。

Karpenter支持的三个topologyKey为：

  • topology.kubernetes.io/zone
  • kubernetes.io/hostname
  • karpenter.sh/capacity-type

首先删除之前的资源：

kubectl delete deployment inflate
kubectl delete nodepools.karpenter.sh default
kubectl delete ec2nodeclasses.karpenter.k8s.aws default

部署NodePool：

mkdir ~/environment/karpenter
cd ~/environment/karpenter
cat <<EoF> topology.yaml
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  creationTimestamp: null
  name: default
spec:
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: Never
  limits:
    cpu: "20"
  template:
    metadata:
      labels:
        eks-immersion-team: my-team
    spec:
      nodeClassRef:
        name: default
      # Requirements that constrain the parameters of provisioned nodes.
      # These requirements are combined with pod.spec.affinity.nodeAffinity rules.
      # Operators { In, NotIn } are supported to enable including or excluding values
      requirements:
        - key: "karpenter.k8s.aws/instance-category"
          operator: In
          values: ["c", "m", "r"]
        - key: "kubernetes.io/arch"
          operator: In
          values: ["amd64"]
        - key: "karpenter.sh/capacity-type" # If not included, the webhook for the AWS cloud provider will default to on-demand
          operator: In
          values: ["on-demand"]
---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2
  role: "KarpenterNodeRole-${CLUSTER_NAME}"
  securityGroupSelectorTerms:
  - tags:
      alpha.eksctl.io/cluster-name: $CLUSTER_NAME
  subnetSelectorTerms:
  - tags:
      alpha.eksctl.io/cluster-name: $CLUSTER_NAME
  tags:
    intent: apps
    managed-by: karpenter
EoF

kubectl apply -f topology.yaml

部署应用

  • 在Pod中使用topologySpreadConstraints参数，Provisioner会根据其中的定义来满足分散的要求。
  • maxskew设置为2。例如AZ a里有4个pod，那么AZ b至少也要有2个，否则他们的差值就大于2
  • whenUnsatisfiable 指定如果 Pod 不满足分布约束，如何处理它。 这里设置的是如果 pod 不满足分布约束，那么调度程序就不会调度 pod。
cd ~/environment/karpenter
cat <<EoF> topology-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate
spec:
  replicas: 3
  selector:
    matchLabels:
      app: inflate
  template:
    metadata:
      labels:
        app: inflate
    spec:
      topologySpreadConstraints:
      - maxSkew: 2
        topologyKey: "topology.kubernetes.io/zone"
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: inflate
      terminationGracePeriodSeconds: 0
      containers:
        - name: inflate
          image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
          resources:
            requests:
              cpu: 1
      nodeSelector:
        eks-immersion-team: my-team
EoF

kubectl apply -f topology-deploy.yaml

可以看到，在两个AZ启动了节点，一个节点上部署了两个Pod，另一个节点上部署了1个Pod：

image-20231028123834632

将replica的数量设置为7：

kubectl scale deployment inflate --replicas 7

现在拉起来了一个新的节点，三个节点的pod数量分别为3、3、1，满足maxSkew的约束：

image-20231028124006158

使用AZ和实例两个约束条件

先删除应用：

kubectl delete deployment inflate

现在添加两个topologyKey限制条件：

  • AZ。maxSkew为1
  • 实例。 maxSkew为1
cd ~/environment/karpenter
cat <<EoF> topology2-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate
spec:
  replicas: 3
  selector:
    matchLabels:
      app: inflate
  template:
    metadata:
      labels:
        app: inflate
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: "topology.kubernetes.io/zone"
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: inflate
      - maxSkew: 1
        topologyKey: "kubernetes.io/hostname"
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: inflate
      terminationGracePeriodSeconds: 0
      containers:
        - name: inflate
          image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
          resources:
            requests:
              cpu: 1
      nodeSelector:
        eks-immersion-team: my-team
EoF

kubectl apply -f topology2-deploy.yaml

image-20231028125314680

加了hostname限制后，开出来三台EC2，且分布在三个AZ。