Topology Spread

本节我们将使用AZ topologySpreadConstraints来将Pod分布在不同的AZ

在k8s里，可以使用topology spread来保证高可用，例如将pod分布在不同的AZ、Region。

Karpenter支持的三个topologyKey为：

  • topology.kubernetes.io/zone
  • kubernetes.io/hostname
  • karpenter.sh/capacity-type

首先删除之前的资源：

kubectl delete deployment inflate
kubectl delete provisioners.karpenter.sh default
kubectl delete awsnodetemplates.karpenter.k8s.aws default

部署Provisioner和Node Template：

mkdir ~/environment/karpenter
cd ~/environment/karpenter
cat <<EoF> topology.yaml
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  # References cloud provider-specific custom resource, see your cloud provider specific documentation
  providerRef:
    name: default

  # Labels are arbitrary key-values that are applied to all nodes
  labels:
    eks-immersion-team: my-team

  # Requirements that constrain the parameters of provisioned nodes.
  # These requirements are combined with pod.spec.affinity.nodeAffinity rules.
  # Operators { In, NotIn } are supported to enable including or excluding values
  requirements:
    - key: "karpenter.k8s.aws/instance-category"
      operator: In
      values: ["c", "m", "r"]
    - key: "kubernetes.io/arch"
      operator: In
      values: ["amd64"]
    - key: "karpenter.sh/capacity-type" # If not included, the webhook for the AWS cloud provider will default to on-demand
      operator: In
      values: ["on-demand"]
  limits:
    resources:
      cpu: "20"
  consolidation:
    enabled: true

---
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: default
spec:
  subnetSelector:
    alpha.eksctl.io/cluster-name: ${CLUSTER_NAME}
  securityGroupSelector:
    aws:eks:cluster-name: ${CLUSTER_NAME}
  tags:
    managed-by: "karpenter"
    intent: "apps"
EoF

kubectl apply -f topology.yaml

部署应用

  • 在Pod中使用topologySpreadConstraints参数，Provisioner会根据其中的定义来满足分散的要求。
  • maxskew设置为2。例如AZ a里有4个pod，那么AZ b至少也要有2个，否则他们的差值就大于2
  • whenUnsatisfiable 指定如果 Pod 不满足分布约束，如何处理它。 这里设置的是如果 pod 不满足分布约束，那么调度程序就不会调度 pod。
cd ~/environment/karpenter
cat <<EoF> topology-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate
spec:
  replicas: 3
  selector:
    matchLabels:
      app: inflate
  template:
    metadata:
      labels:
        app: inflate
    spec:
      topologySpreadConstraints:
      - maxSkew: 2
        topologyKey: "topology.kubernetes.io/zone"
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: inflate
      terminationGracePeriodSeconds: 0
      containers:
        - name: inflate
          image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
          resources:
            requests:
              cpu: 1
      nodeSelector:
        eks-immersion-team: my-team
EoF

kubectl apply -f topology-deploy.yaml

可以看到，在两个AZ启动了节点，一个节点上部署了两个Pod，另一个节点上部署了1个Pod：

image-20231028123834632

将replica的数量设置为7：

kubectl scale deployment inflate --replicas 7

现在拉起来了一个新的节点，三个节点的pod数量分别为3、3、1，满足maxSkew的约束：

image-20231028124006158

使用AZ和实例两个约束条件

先删除应用：

kubectl delete deployment inflate

现在添加两个topologyKey限制条件：

  • AZ。maxSkew为1
  • 实例。 maxSkew为1
cd ~/environment/karpenter
cat <<EoF> topology2-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate
spec:
  replicas: 3
  selector:
    matchLabels:
      app: inflate
  template:
    metadata:
      labels:
        app: inflate
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: "topology.kubernetes.io/zone"
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: inflate
      - maxSkew: 1
        topologyKey: "kubernetes.io/hostname"
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: inflate
      terminationGracePeriodSeconds: 0
      containers:
        - name: inflate
          image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
          resources:
            requests:
              cpu: 1
      nodeSelector:
        eks-immersion-team: my-team
EoF

kubectl apply -f topology2-deploy.yaml

image-20231028125314680

加了hostname限制后，开出来三台EC2，且分布在三个AZ。