Multi Provisioner
可以为不同的应用设置不同的Provisioner，让这些Provisioner一起工作。
多Provisioner的使用场景如下：
- 在同一集群上有不同的团队，在完全独立的容量上运行。
- 一个团队可以使用 BottleRocket 在节点上运行，而另一个团队可以使用 EKSOptimizedAMI。
- 多个Provisioner可用于独立计费节点、使用不同的节点约束（例如团队没有使用GPU）或使用不同的
Deprovisioning设置
建议创建互斥的 Provisioner。 Pod 不应匹配多个 Provisioner。 如果匹配多个Provisioner，Karpenter将使用权重最高的Provisioner, 这个权重在provisioner spec中设置
Multi Provisioner测试
先删除之前创建的资源：
kubectl delete deployment inflate
kubectl delete provisioners.karpenter.sh default
kubectl delete awsnodetemplates.karpenter.k8s.aws default
然后部署一个Provisioner和NodeTemplate，这个provisioner创建出来的机器都同m系列：
mkdir -p ~/environment/karpenter
cd ~/environment/karpenter
cat << EOF > karpenter_multi_provisioner_node_template.yaml
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
name: default
spec:
providerRef:
name: default
ttlSecondsAfterEmpty: 30
labels:
eks-immersion-team: default
requirements:
- key: "node.kubernetes.io/instance-type"
operator: In
values: ["m5.large", "m5.xlarge"]
- key: "kubernetes.io/arch"
operator: In
values: ["amd64"]
limits:
resources:
cpu: "50"
---
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
name: default
spec:
subnetSelector:
alpha.eksctl.io/cluster-name: ${CLUSTER_NAME}
securityGroupSelector:
kubernetes.io/cluster/${CLUSTER_NAME}: owned
tags:
managed-by: "karpenter"
intent: "apps"
EOF
kubectl -f karpenter_multi_provisioner_node_template.yaml create
然后创建另一个custom provisioner，这个provisioner创建出来的机器都是t系列：
mkdir -p ~/environment/karpenter
cd ~/environment/karpenter
cat <<EoF> team-provisioner.yaml
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
name: team-provisioner
spec:
# References cloud provider-specific custom resource, see your cloud provider specific documentation
providerRef:
name: team-provisioner
ttlSecondsAfterEmpty: 30
labels:
eks-immersion-team: team-provisioner
requirements:
- key: "karpenter.k8s.aws/instance-category"
operator: In
values: ["t"]
- key: "kubernetes.io/arch"
operator: In
values: ["amd64"]
limits:
resources:
cpu: "50"
---
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
name: team-provisioner
spec:
subnetSelector:
alpha.eksctl.io/cluster-name: ${CLUSTER_NAME}
securityGroupSelector:
kubernetes.io/cluster/${CLUSTER_NAME}: owned
tags:
managed-by: "karpenter"
intent: "apps"
EoF
kubectl apply -f team-provisioner.yaml
测试多Provisioner
创建一个deployment，它在pod定义中声明了使用t3.large机器：
cd ~/environment/karpenter
cat <<EoF> team-workload.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: inflate
spec:
replicas: 1
selector:
matchLabels:
app: inflate
template:
metadata:
labels:
app: inflate
spec:
containers:
- name: inflate
image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
resources:
requests:
cpu: 1
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node.kubernetes.io/instance-type
operator: In
values:
- t3.large
EoF
kubectl apply -f team-workload.yaml
等待一段时间后，Karpenter拉起机器并部署pod：
kubectl -n karpenter logs -l app.kubernetes.io/name=karpenter | grep controller.provisioner
从日志中也能确认是team-provisioner创建出来的：
2023-10-29T07:52:54.511Z INFO controller.provisioner created machine {"commit": "61b3e1e-dirty", "provisioner": "team-provisioner", "machine": "team-provisioner-4qtw5", "requests": {"cpu":"1155m","memory":"120Mi","pods":"5"}, "instance-types": "t3.large"}
结论： 即使集群有两个provisioner，Karpenter还是选择出了正确的provisioner来创建节点
清理资源：
kubectl delete provisioners.karpenter.sh team-provisioner
kubectl delete awsnodetemplates.karpenter.k8s.aws team-provisioner
kubectl delete deployment inflate
带有权重的Provisioner
Karpenter允许在 .spec.weight里设置Provisioner的权重，这样会按顺序评估。
我们再创建一个custom provisioner， 在里面设置它的权重为2，
mkdir -p ~/environment/karpenter
cd ~/environment/karpenter
cat <<EoF> weight-provisioner.yaml
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
name: weight-provisioner
spec:
# References cloud provider-specific custom resource, see your cloud provider specific documentation
providerRef:
name: weight-provisioner
ttlSecondsAfterEmpty: 30
weight: 2
labels:
eks-immersion-team: weight-provisioner
requirements:
- key: "karpenter.k8s.aws/instance-category"
operator: In
values: ["t"]
- key: "kubernetes.io/arch"
operator: In
values: ["amd64"]
limits:
resources:
cpu: "50"
---
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
name: weight-provisioner
spec:
subnetSelector:
alpha.eksctl.io/cluster-name: ${CLUSTER_NAME}
securityGroupSelector:
kubernetes.io/cluster/${CLUSTER_NAME}: owned
tags:
managed-by: "karpenter"
intent: "apps"
EoF
kubectl apply -f weight-provisioner.yaml
之前的default provisioner，我们没有显式设置它的权重，这样它的权重是默认的1。执行以下命令确认：
kubectl get provisioners -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.weight}{"\n"}{end}'
输出：
default
weight-provisioner 2
创建测试应用：
cd ~/environment/karpenter
cat <<EoF> weight-workload.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: inflate
spec:
replicas: 8
selector:
matchLabels:
app: inflate
template:
metadata:
labels:
app: inflate
spec:
containers:
- name: inflate
image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
resources:
requests:
memory: 1Gi
cpu: 1
EoF
kubectl apply -f weight-workload.yaml
新创建出来的节点全部是t系列，这是因为我们的custom provisioner权重比默认的provisioner要高：
清理资源：
kubectl delete provisioners.karpenter.sh weight-provisioner
kubectl delete awsnodetemplates.karpenter.k8s.aws weight-provisioner
kubectl delete deployment inflate