Karpenter > Deep Dive > Node Template

Node Template

每个provisioner必须引用一个AWSNodeTemplate，AWSNodeTemplate里面定义了AWS相关的配置。本节我们将探讨Node Template相关的特性

首先删除之前创建的deploy及provisioner、awsnodetemplates：

kubectl delete deployment inflate
kubectl delete provisioners.karpenter.sh default
kubectl delete awsnodetemplates.karpenter.k8s.aws default

securityGroupSelector及subnetSelector参数介绍

查看Node Template文档： https://karpenter.sh/docs/concepts/node-templates/

会发现只有securityGroupSelector和subnetSelector参数是要强制设置的：

apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: default
spec:
  subnetSelector: { ... }        # required, discovers tagged subnets to attach to instances
  securityGroupSelector: { ... } # required, discovers tagged security groups to attach to instances
  instanceProfile: "..."         # optional, overrides the node's identity from global settings
  amiFamily: "..."               # optional, resolves a default ami and userdata
  amiSelector: { ... }           # optional, discovers tagged amis to override the amiFamily's default
  userData: "..."                # optional, overrides autogenerated userdata with a merge semantic
  tags: { ... }                  # optional, propagates tags to underlying EC2 resources
  metadataOptions: { ... }       # optional, configures IMDS for the instance
  blockDeviceMappings: [ ... ]   # optional, configures storage devices for the instance
  detailedMonitoring: "..."      # optional, configures detailed monitoring for the instance

Provisioner根据subnetSelector中定义的值来查询子网，当有新的节点要创建时，从查询到的对应子网中选择一个，来启动节点
Provisioner根据securityGroupSelector来查询安全组，当有新的节点要创建时，从查询到的对应安全组来绑定到节点上。

第一章我们创建Node Template时，是这样设置的：

apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: default
spec:
  subnetSelector:
    alpha.eksctl.io/cluster-name: ${CLUSTER_NAME}
  securityGroupSelector:
    kubernetes.io/cluster/${CLUSTER_NAME}: owned
EOF

本节着重介绍下Node Template中其他的参数。

使用自定义AMI - amiFamily

先创建一个自定义的Node Template，使用Ubuntu AMI：

cd ~/environment/karpenter
cat << EOF > custom_ami_node_template.yaml
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: custom

spec:
  subnetSelector:
    alpha.eksctl.io/cluster-name: ${CLUSTER_NAME}
  securityGroupSelector:
    kubernetes.io/cluster/${CLUSTER_NAME}: owned
    
  amiFamily: Ubuntu # 这里声明AMI使用Ubuntu
  tags:
    managed-by: "karpenter"
    ami-type: "ubuntu"
    intent: "apps"
EOF
kubectl create -f custom_ami_node_template.yaml

同时创建一个provisoiner：

cd ~/environment/karpenter
cat << EOF > custom_karpenter_provisioner.yaml
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: custom
spec:
  providerRef:
    name: custom
  ttlSecondsAfterEmpty: 30
  labels:
    eks-immersion-team: default
  requirements:
    - key: "node.kubernetes.io/instance-type"
      operator: In
      values: ["m5.large", "m5.xlarge"]
    - key: "kubernetes.io/arch"
      operator: In
      values: ["amd64"]
  limits:
    resources:
      cpu: "50"
EOF
kubectl -f custom_karpenter_provisioner.yaml create

测试Node Template，创建一个具有5个replica的deployment：

cd ~/environment/karpenter
cat <<EOF > basic-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate
spec:
  replicas: 5
  selector:
    matchLabels:
      app: inflate
  template:
    metadata:
      labels:
        app: inflate
    spec:
      terminationGracePeriodSeconds: 0
      containers:
        - name: inflate
          image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
          resources:
            requests:
              cpu: 1
EOF

kubectl apply -f basic-deploy.yaml

查看Karpenter的日志，custom provisoner创建出一台EC2:

kongpingfan:~/environment/karpenter $ kubectl -n karpenter logs -l app.kubernetes.io/name=karpenter | grep controller.provisioner
2023-10-27T14:37:05.237Z        DEBUG   controller.provisioner  209 out of 743 instance types were excluded because they would breach limits  {"commit": "61b3e1e-dirty", "provisioner": "custom"}
2023-10-27T14:37:05.266Z        INFO    controller.provisioner  found provisionable pod(s)      {"commit": "61b3e1e-dirty", "pods": "default/inflate-5947bd9774-np687, default/inflate-5947bd9774-2dwj2, default/inflate-5947bd9774-nn5x6", "duration": "29.025095ms"}
2023-10-27T14:37:05.266Z        INFO    controller.provisioner  computed new machine(s) to fit pod(s)   {"commit": "61b3e1e-dirty", "machines": 1, "pods": 3}
2023-10-27T14:37:05.278Z        INFO    controller.provisioner  created machine {"commit": "61b3e1e-dirty", "provisioner": "custom", "machine": "custom-kfb6s", "requests": {"cpu":"3155m","memory":"120Mi","pods":"7"}, "instance-types": "m5.xlarge"}

查看EC2页面，新启动的节点系统确认是ubuntu：

未能分配的pod被分配到这个节点上：

测试完成后，把deployment、provisioner、node template一起删除：

kubectl delete -f basic-deploy.yaml
kubectl delete -f custom_ami_node_template.yaml
kubectl delete -f custom_karpenter_provisioner.yaml

blockdeviceMappings参数

Node Template中的blockdeviceMappings字段用于指定节点的EBS参数。

在实际生产环境中，我们可能遇到下面场景：

挂载一块额外的盘，用于存储业务日志或容器镜像
调整EBS根卷的大小、IOPS、加密方式

这些场景都可以通过blockdeviceMappings来设置，例如：

apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
spec:
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 20Gi
        volumeType: gp3

测试

进入到karpenter项目目录下：

cd ~/environment/karpenter  || mkdir -p ~/environment/karpenter  && cd ~/environment/karpenter

我们使用blockdeviceMappings参数来创建两块ebs盘，并使用userData来执行脚本，在节点启动的时候检测数据卷并挂载到/data$N目录下。创建一个basic-deploy-blockdevice.yaml 文件，内容如下，注意将第34和36行的${CLUSTER_NAME}做替换：

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  providerRef:
    name: default
  ttlSecondsAfterEmpty: 30

  labels:
    eks-immersion-team: my-team

  requirements:
    - key: "karpenter.k8s.aws/instance-category"
      operator: In
      values: ["c", "m", "r"]
    - key: "kubernetes.io/arch"
      operator: In
      values: ["amd64"]
    - key: "karpenter.sh/capacity-type" # If not included, the webhook for the AWS cloud provider will default to on-demand
      operator: In
      values: ["on-demand"]
  limits:
    resources:
      cpu: "10"
    
---
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: default
spec:
  subnetSelector:
    alpha.eksctl.io/cluster-name: ${CLUSTER_NAME}
  securityGroupSelector:
    kubernetes.io/cluster/${CLUSTER_NAME}: owned
    
  blockDeviceMappings:
    - deviceName: /dev/xvda # root Volume to store OS Binaries
      ebs:
        volumeType: gp3 # EBS Volume Type
        volumeSize: 20Gi # Size of the disk
        deleteOnTermination: true # Disk Retention Policy
    - deviceName: /dev/xvdb # Data Volume to store Images, Logs etc
      ebs:
        volumeType: gp3 # EBS Volume Type
        volumeSize: 100Gi # Size of the disk
        deleteOnTermination: true # Disk Retention Policy
  userData: |
    #!/bin/bash -xe
    exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

    # Mount data volumes to /data$N directory on a Amazon Linux Worker Node (excluding OS volume)
    
    # Identify the device name of the root volume
    root_device=$(mount | awk '$3 == "/" {print $1}')
    
    # Identify the device names of all attached block devices (excluding root volume)
    device_names=$(lsblk -d -n -o NAME | grep -v "$root_device")
    
    # Loop through each device name and mount the corresponding volume to a directory named /data$N
    i=1
    for device_name in $device_names; do
      if ! grep -qs "/dev/$device_name" /proc/mounts; then
        sudo mkfs.xfs "/dev/$device_name"
        sudo mkdir -p "/data${i}"
        sudo mount "/dev/$device_name" "/data${i}"
        echo "Mounted /dev/$device_name to /data${i}"
        ((i++))
      fi
    done
        
  tags:
    managed-by: "karpenter"
    intent: “apps"

部署Node Template和Provisioner：

kubectl apply -f basic-deploy-blockdevice.yaml

provisoner和node template创建完成后，继续创建inflate应用，由于它的nodeSelector = eks-immersion-team: my-team，所以会触发provisoner创建出一个新的节点：

cd ~/environment/karpenter  || mkdir -p ~/environment/karpenter  && cd ~/environment/karpenter 
cat > basic-app-deploy.yaml <<'EOF' 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate
spec:
  replicas: 1
  selector:
    matchLabels:
      app: inflate
  template:
    metadata:
      labels:
        app: inflate
    spec:
      terminationGracePeriodSeconds: 0
      containers:
        - name: inflate
          image: nginx
          volumeMounts:
          - name: my-app-volume
            mountPath: /var/log/nginx
          resources:
            requests:
              cpu: 1
      volumes:
      - name: my-app-volume
        hostPath:
          path: /data1
      nodeSelector:
        eks-immersion-team: my-team
EOF

kubectl apply -f basic-app-deploy.yaml

注意上面的pod将/var/log/nginx挂载到本地磁盘上了。

等待一段时间，确认新的节点被拉起：

kubectl get nodes -l eks-immersion-team=my-team

根据这个NAME在EC2控制台找到这台机器，看到它确实有两个EBS：

使用Session Manager连接到EC2：

执行df -h，看到data1已经挂载上去：

进入到/data1目录，执行ls命令会发现有两个文件error.log和access.log, 这是nginx pod的日志，查看具体内容：

最后清理node template和provisioner：

kubectl delete -f basic-app-deploy.yaml

kubectl delete -f basic-deploy-blockdevice.yaml

spec.tags

Karpenter默认会将这些tag加到创建出来的EC2、EBS：

Name: karpenter.sh/provisioner-name/<provisioner-name>
karpenter.sh/provisioner-name: <provisioner-name>
kubernetes.io/cluster/<cluster-name>: owned

使用spec.tags能添加额外的tag：

spec:
  tags:
    InternalAccountingTag: 1234
    dev.corp.net/app: Calculator
    dev.corp.net/team: MyTeam

可以对Name标签进行覆盖，但是不能对“karpenter.sh”, “karpenter.k8s.aws”, “kubernetes.io/cluster这些进行覆盖，因为Karpenter要用它们来自动发现机器。