Workload Consolidation(工作负载整合) II

Consolidation分类

  1. Empty node consolidation(空节点合并) - 并行删除所有空的节点

  2. Single-Node Consolidation(单节点整合) - 尝试删除单个节点，可能会启动比该节点价格便宜的节点来替换它。

  3. Multi-Node Consolidation(多节点整合) - 尝试并行删除两个或多个节点，可能会启动比被删除节点价格更便宜的节点进来替换。Karpenter 通过选择通过终止来整合总体上对工作负载干扰最少的节点来实现此目的：

  • 运行较少 Pod 的节点

  • 即将过期的节点

  • 具有较低优先级 pod 的节点

Multi Node Consolidation - 多节点整合

先删除上面创建的资源：

kubectl delete deployment inflate
kubectl delete nodepool.karpenter.sh default
kubectl delete ec2nodeclass.karpenter.k8s.aws default

创建NodePool：

mkdir -p ~/environment/karpenter
cd ~/environment/karpenter
cat <<EoF> multinode.yaml
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
spec:
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: Never
  limits:
    cpu: 1k
  template:
    metadata:
      labels:
        eks-immersion-team: my-team
    spec:
      nodeClassRef:
        name: default
      requirements:
      - key: karpenter.k8s.aws/instance-category
        operator: In
        values:
        - c
        - m
      - key: karpenter.k8s.aws/instance-cpu
        operator: Lt
        values:
        - "5"
      - key: kubernetes.io/arch
        operator: In
        values:
        - amd64
      - key: karpenter.sh/capacity-type
        operator: In
        values:
        - on-demand
      - key: kubernetes.io/os
        operator: In
        values:
        - linux

---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2
  role: "KarpenterNodeRole-${CLUSTER_NAME}"
  securityGroupSelectorTerms:
  - tags:
      alpha.eksctl.io/cluster-name: $CLUSTER_NAME
  subnetSelectorTerms:
  - tags:
      alpha.eksctl.io/cluster-name: $CLUSTER_NAME
  tags:
    intent: apps
    managed-by: karpenter
    eks-immersion-team: my-team
EoF

kubectl apply -f multinode.yaml

部署应用, 它有20个replica：

cd ~/environment/karpenter
cat <<EoF> multinode-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate
spec:
  replicas: 20
  selector:
    matchLabels:
      app: inflate
  template:
    metadata:
      labels:
        app: inflate
    spec:
      terminationGracePeriodSeconds: 0
      containers:
        - name: inflate
          image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
          resources:
            requests:
              memory: 1Gi
              cpu: 1
      nodeSelector:
        eks-immersion-team: my-team
EoF

kubectl apply -f multinode-deploy.yaml

看到创建出来7个节点，前6个每个跑着3个pod，最后一个跑着2个pod：

image-20231029002141397

将replica数量缩小到10：

kubectl scale deployment inflate --replicas 10

一开始看到随机下掉10个pod，现在每个节点上Pod分布数量为 1 1 3 1 2 2 0

image-20231029002347834

Karpenter首先将没有pod的节点干掉（最后一个）：

image-20231029002409750

然后它将node进行了整合，又下掉了两个node。最后pod数量分布为3 3 3 1（并且机型分别为3台xlarge和1台large）：

image-20231029002552400

再将replica设置为4：

kubectl scale deployment inflate --replicas 4

每个node上的pod数量先变成2 0 1 1 ， 毫无疑问第二个首先被干掉：

image-20231029002836008

接着进行node整合，最后只需要两个node。pod数量分布为3 1：

image-20231029003019851

不仅如此，第二个节点(分布了1个pod的)类型为c6a.large, 达到了极致的省钱效果:

image-20231029003209921

查看日志：

 kubectl -n karpenter logs -l app.kubernetes.io/name=karpenter

能看到disrupting via consolidation replace/delete事件：

image-20240803205822312