karpenter-provider-aws 实战从 Cluster Autoscaler 平滑迁移到 Karpenter 的完整步骤与源码解析【免费下载链接】karpenter-provider-awsKarpenter is a Kubernetes Node Autoscaler built for flexibility, performance, and simplicity.项目地址: https://gitcode.com/GitHub_Trending/ka/karpenter-provider-aws导读Kubernetes 集群的节点自动扩缩容正从 Cluster AutoscalerCAS转向更灵活的 Karpenter。本文基于本仓库官方文档 Migrating from Cluster Autoscaler 编写完整覆盖迁移前的 IAM 角色、标签、aws-auth 授权、Helm 部署、NodePool 创建、CAS 下线等每一步的实际命令并结合仓库内的 Helm Chart 源码charts/karpenter/values.yaml与示例配置examples/v1/general-purpose.yaml深入讲解每个参数的作用帮助你把现有 EKS 集群从 CAS 平滑切换为 Karpenter 自动供给节点。一、迁移前提假设该迁移指南基于以下假设对应原文档开头的 assumptions 列表使用一个已存在的 EKS 集群复用现有的VPC 和子网复用现有的安全组节点分布在**一个或多个 EKS 托管节点组managed node group**中工作负载已配置符合 EKS 最佳实践的PodDisruptionBudgetPDB集群已启用OIDC Provider用于 IAM Roles for Service AccountsIRSA本机已安装awsCLI多数步骤也可在控制台中完成本文用命令行简化。首先设置集群名与命名空间变量KARPENTER_NAMESPACEkube-system CLUSTER_NAMEyour cluster name然后从集群配置中提取其他变量。原文档引用了 step01-env.sh其完整内容如下它从 SSM 参数中解析 AL2023 优化 AMI 的版本别名供后续 EC2NodeClass 的amiSelectorTerms使用AWS_PARTITIONaws # if you are not using standard partitions, you may need to configure to aws-cn / aws-us-gov AWS_REGION$(aws configure list | grep region | tr -s | cut -d -f3) OIDC_ENDPOINT$(aws eks describe-cluster --name ${CLUSTER_NAME} \ --query cluster.identity.oidc.issuer --output text) AWS_ACCOUNT_ID$(aws sts get-caller-identity --query Account \ --output text) K8S_VERSION$(aws eks describe-cluster --name ${CLUSTER_NAME} --query cluster.version --output text) ALIAS_VERSION$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2023/x86_64/standard/recommended/image_id --query Parameter.Value | xargs aws ec2 describe-images --query Images[0].Name --image-ids | sed -r s/^.*(v[[:digit:]]).*$/\1/)几点说明AWS_PARTITION默认为aws中国区域需改为aws-cn政务云为aws-us-govALIAS_VERSION通过 SSM 中 EKS 优化 AMI 的镜像 ID 反查镜像名称并提取其中的 Kubernetes 版本如v1.30最终用于al2023${ALIAS_VERSION}这种版本化 AMI 别名保证 Karpenter 供给的节点与集群版本匹配仓库中 pkg/providers/amifamily/al2023.go 负责解析al2023...别名并发现对应 AMI。二、创建 IAM 角色迁移的第一步是创建两个新的 IAM 角色Karpenter 节点角色新节点加入集群时扮演和Karpenter 控制器角色控制器创建实例时扮演。2.1 节点角色 KarpenterNodeRolestep02-node-iam.sh 先生成一个允许ec2.amazonaws.com服务 AssumeRole 的信任策略再创建角色echo { Version: 2012-10-17, Statement: [ { Effect: Allow, Principal: { Service: ec2.amazonaws.com }, Action: sts:AssumeRole } ] } node-trust-policy.json aws iam create-role --role-name KarpenterNodeRole-${CLUSTER_NAME} \ --assume-role-policy-document file://node-trust-policy.json随后 step03-node-policies.sh 为角色附加四个 EKS 工作节点必备策略aws iam attach-role-policy --role-name KarpenterNodeRole-${CLUSTER_NAME} \ --policy-arn arn:${AWS_PARTITION}:iam::aws:policy/AmazonEKSWorkerNodePolicy aws iam attach-role-policy --role-name KarpenterNodeRole-${CLUSTER_NAME} \ --policy-arn arn:${AWS_PARTITION}:iam::aws:policy/AmazonEKS_CNI_Policy aws iam attach-role-policy --role-name KarpenterNodeRole-${CLUSTER_NAME} \ --policy-arn arn:${AWS_PARTITION}:iam::aws:policy/AmazonEC2ContainerRegistryPullOnly aws iam attach-role-policy --role-name KarpenterNodeRole-${CLUSTER_NAME} \ --policy-arn arn:${AWS_PARTITION}:iam::aws:policy/AmazonSSMManagedInstanceCore这四个策略分别对应节点接入 EKS 集群的 kubelet 权限、VPC CNI 的 ENI 网络操作、只读拉取 ECR 镜像、以及 SSM 基础运维权限。2.2 控制器角色 KarpenterControllerRoleIRSAKarpenter 控制器运行在集群内通过IRSA获取 AWS 凭证因此需要一个绑定 OIDC 联合身份的信任策略并限定sub为system:serviceaccount:${KARPENTER_NAMESPACE}:karpenter。step04-controller-iam.sh 完整代码如下cat EOF controller-trust-policy.json { Version: 2012-10-17, Statement: [ { Effect: Allow, Principal: { Federated: arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT#*//} }, Action: sts:AssumeRoleWithWebIdentity, Condition: { StringEquals: { ${OIDC_ENDPOINT#*//}:aud: sts.amazonaws.com, ${OIDC_ENDPOINT#*//}:sub: system:serviceaccount:${KARPENTER_NAMESPACE}:karpenter } } } ] } EOF aws iam create-role --role-name KarpenterControllerRole-${CLUSTER_NAME} \ --assume-role-policy-document file://controller-trust-policy.json注意如果你的工作负载使用其他 IAM 凭证方案例如 kube2iam信任策略的写法会不同。控制器的内联权限策略controller-policy.json是迁移中权限面最宽的部分可以归纳为五类核心供给权限Sid: Karpenterec2:RunInstances、ec2:CreateFleet、ec2:DescribeSubnets/SecurityGroups/LaunchTemplates/Instances/InstanceTypes/InstanceTypeOfferings/SpotPriceHistory、ec2:CreateLaunchTemplate、ec2:DeleteLaunchTemplate、ec2:CreateTags、ec2:DescribeImages、ssm:GetParameter、pricing:GetProducts等受限终止权限Sid: ConditionalEC2Terminationec2:TerminateInstances仅允许终止带karpenter.sh/nodepool标签的实例——这是 Karpenter 的安全边界只能回收自己创建的节点PassRoleSid: PassNodeIAMRole允许控制器在创建实例时把第 2.1 步的节点角色传给 EC2集群端点查询Sid: EKSClusterEndpointLookupeks:DescribeCluster且限定具体集群 ARN用于控制器发现集群 API Server 端点IAM Instance Profile 系列权限AllowScopedInstanceProfileCreationActions等iam:CreateInstanceProfile、iam:TagInstanceProfile、iam:AddRoleToInstanceProfile、iam:RemoveRoleFromInstanceProfile、iam:DeleteInstanceProfile、iam:GetInstanceProfile、iam:ListInstanceProfiles全部附带kubernetes.io/cluster/${CLUSTER_NAME}owned和topology.kubernetes.io/region的请求/资源标签条件将操作范围锁定在本集群。最后通过aws iam put-role-policy --policy-name KarpenterControllerPolicy-${CLUSTER_NAME}挂到角色上。从源码结构看上述权限清单与仓库 Chart 中默认 ClusterRolecharts/karpenter/templates/clusterrole-core.yaml互补前者是 AWS 侧权限后者是 K8s 侧权限Node、NodePool、NodeClaim、Pod、Event 等的 CRUD。三、为子网和安全组添加 karpenter.sh/discovery 标签Karpenter 通过标签发现可用子网和安全组。这是 EC2NodeClass 中subnetSelectorTerms与securityGroupSelectorTerms的匹配条件可参见 examples/v1/general-purpose.yaml 中karpenter.sh/discovery: ${CLUSTER_NAME}的用法。3.1 给所有节点组的子网打标签step05-tag-subnets.sh 遍历集群全部节点组为每组子网添加karpenter.sh/discovery标签for NODEGROUP in $(aws eks list-nodegroups --cluster-name ${CLUSTER_NAME} --query nodegroups --output text); do aws ec2 create-tags \ --tags Keykarpenter.sh/discovery,Value${CLUSTER_NAME} \ --resources $(aws eks describe-nodegroup --cluster-name ${CLUSTER_NAME} \ --nodegroup-name ${NODEGROUP} --query nodegroup.subnets --output text ) done3.2 给安全组打标签step06-tag-security-groups.sh只给第一个节点组的安全组打标签如果有多个节点组或多组安全组需要自行决定 Karpenter 使用哪一组NODEGROUP$(aws eks list-nodegroups --cluster-name ${CLUSTER_NAME} \ --query nodegroups[0] --output text) LAUNCH_TEMPLATE$(aws eks describe-nodegroup --cluster-name ${CLUSTER_NAME} \ --nodegroup-name ${NODEGROUP} --query nodegroup.launchTemplate.{id:id,version:version} \ --output text | tr -s \t ,) # If your EKS setup is configured to use only Cluster security group, then please execute - SECURITY_GROUPS$(aws eks describe-cluster \ --name ${CLUSTER_NAME} --query cluster.resourcesVpcConfig.clusterSecurityGroupId --output text) # If your setup uses the security groups in the Launch template of a managed node group, then : SECURITY_GROUPS$(aws ec2 describe-launch-template-versions \ --launch-template-id ${LAUNCH_TEMPLATE%,*} --versions ${LAUNCH_TEMPLATE#*,} \ --query LaunchTemplateVersions[0].LaunchTemplateData.[NetworkInterfaces[0].Groups||SecurityGroupIds] \ --output text) aws ec2 create-tags \ --tags Keykarpenter.sh/discovery,Value${CLUSTER_NAME} \ --resources ${SECURITY_GROUPS}四、更新 aws-auth ConfigMap新建的节点使用第 2.1 步的节点角色需要让集群承认它们。step07-edit-aws-auth.sh 只是简单地执行kubectl edit configmap aws-auth -n kube-system在mapRoles中追加如下条目替换${AWS_PARTITION}、${AWS_ACCOUNT_ID}、${CLUSTER_NAME}但不要替换{{EC2PrivateDNSName}}占位符- groups: - system:bootstrappers - system:nodes ## If you intend to run Windows workloads, the kube-proxy group should be specified. # For more information, see https://github.com/aws/karpenter/issues/5099. # - eks:kube-proxy-windows rolearn: arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME} username: system:node:{{EC2PrivateDNSName}}迁移完成后aws-auth 的mapRoles应有两个条目一个对应 Karpenter 节点角色一个对应原有节点组的角色——这是新旧两套节点共存的授权基础。五、部署 Karpenter5.1 设置版本并生成部署 YAML先固定要部署的 Karpenter 版本然后用helm template从 OCI 仓库渲染完整部署清单export KARPENTER_VERSIONthe version you wantstep08-generate-chart.shhelm template karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION} --namespace ${KARPENTER_NAMESPACE} \ --set settings.clusterName${CLUSTER_NAME} \ --set settings.interruptionQueue${CLUSTER_NAME} \ --set serviceAccount.annotations.eks\.amazonaws\.com/role-arnarn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterControllerRole-${CLUSTER_NAME} \ --set controller.resources.requests.cpu1 \ --set controller.resources.requests.memory1Gi \ --set controller.resources.limits.cpu1 \ --set controller.resources.limits.memory1Gi karpenter.yaml结合仓库 Chart 的 values.yaml这些参数的含义为settings.clusterName集群名控制器以此匹配发现资源settings.interruptionQueue中断处理所用的 SQS 队列名不设置则中断处理被禁用启用后控制器需要额外权限serviceAccount.annotations.eks.amazonaws.com/role-arn这是 IRSA 落地点——Chart 的 serviceaccount.yaml 会将serviceAccount.annotations渲染到 ServiceAccount 上EKS 据此把 OIDC token 交换为第 2.2 步创建的控制器角色controller.resources.*为控制器容器设置 1 CPU / 1Gi 的请求与限制values.yaml 中默认controller.resources: {}即不设置。5.2 设置节点亲和让控制器跑在存量节点组上编辑karpenter.yaml找到 Karpenter Deployment 的affinity规则。Chart 默认只要求节点不属于任何 Karpenter NodePool即karpenter.sh/nodepool为 DoesNotExist迁移场景下需进一步限定到现有节点组保证过渡期内控制器自身不会依赖 Karpenter 供给的节点affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: karpenter.sh/nodepool operator: DoesNotExist - key: eks.amazonaws.com/nodegroup operator: In values: - ${NODEGROUP} podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - topologyKey: kubernetes.io/hostname每行一个节点组values按实际$NODEGROUP修改。podAntiAffinity的kubernetes.io/hostname约束与 values.yaml 中的默认值一致确保两个副本不落在同一节点上。5.3 创建命名空间、CRD 并部署step09-deploy.sh 按顺序创建命名空间、安装三个 CRD再应用渲染出的karpenter.yamlkubectl create namespace ${KARPENTER_NAMESPACE} || true kubectl create -f \ https://raw.githubusercontent.com/aws/karpenter-provider-aws/v${KARPENTER_VERSION}/pkg/apis/crds/karpenter.sh_nodepools.yaml kubectl create -f \ https://raw.githubusercontent.com/aws/karpenter-provider-aws/v${KARPENTER_VERSION}/pkg/apis/crds/karpenter.k8s.aws_ec2nodeclasses.yaml kubectl create -f \ https://raw.githubusercontent.com/aws/karpenter-provider-aws/v${KARPENTER_VERSION}/pkg/apis/crds/karpenter.sh_nodeclaims.yaml kubectl apply -f karpenter.yaml这三个 CRD 在本仓库中的对应源文件为 karpenter.sh_nodepools.yaml、karpenter.k8s.aws_ec2nodeclasses.yaml 和 karpenter.sh_nodeclaims.yaml也可从仓库直接查看定义。注意脚本通过kubectl create而非apply安装 CRD意味着升级 CRD 时需要删除后重建。六、创建默认 NodePool 与 EC2NodeClass部署完成后需要创建默认 NodePool告诉 Karpenter 为未调度工作负载供给什么样的节点更多场景可参考 examples/v1 目录下的示例。step10-create-nodepool.sh 通过envsubst替换变量后 applyapiVersion: karpenter.sh/v1 kind: NodePool metadata: name: default spec: template: spec: requirements: - key: kubernetes.io/arch operator: In values: [amd64] - key: kubernetes.io/os operator: In values: [linux] - key: karpenter.sh/capacity-type operator: In values: [spot] - key: karpenter.k8s.aws/instance-category operator: In values: [c, m, r] - key: karpenter.k8s.aws/instance-generation operator: Gt values: [2] nodeClassRef: group: karpenter.k8s.aws kind: EC2NodeClass name: default expireAfter: 720h # 30 * 24h 720h limits: cpu: 1000 disruption: consolidationPolicy: WhenEmptyOrUnderutilized consolidateAfter: 1m --- apiVersion: karpenter.k8s.aws/v1 kind: EC2NodeClass metadata: name: default spec: role: KarpenterNodeRole-${CLUSTER_NAME} # replace with your cluster name amiSelectorTerms: - alias: al2023${ALIAS_VERSION} subnetSelectorTerms: - tags: karpenter.sh/discovery: ${CLUSTER_NAME} # replace with your cluster name securityGroupSelectorTerms: - tags: karpenter.sh/discovery: ${CLUSTER_NAME} # replace with your cluster name关键参数逐项说明requirements是节点约束集合决定可供给的实例范围amd64 linux spot容量类型 c/m/r 三大类别 第二代以上实例族对比 examples/v1/general-purpose.yaml 可见示例默认使用on-demand而迁移场景选择了spot以降低成本可按需调整expireAfter: 720h节点最长生命周期 30 天到期触发漂移替换保证定期获取最新 AMIlimits.cpu: 1000该 NodePool 供给实例的 CPU 总量上限是 Karpenter 层面的保护性限流disruption.consolidationPolicy: WhenEmptyOrUnderutilized且consolidateAfter: 1m空节点或低利用率节点在 1 分钟后即可被合并释放与 CAS 的缩到最小值语义对应但更智能EC2NodeClass 的role与amiSelectorTerms.alias分别消费第 2.1 步的节点角色和第一步 SSM 解析出的ALIAS_VERSIONsubnetSelectorTerms/securityGroupSelectorTerms则精确匹配第三步打的karpenter.sh/discovery标签。七、为关键工作负载设置 nodeAffinity可选为防止 coredns、metric-server 等集群关键组件被调度到 Karpenter 供给的节点上可为它们添加指向静态节点组的节点亲和。用kubectl edit deploy ...编辑并添加affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: eks.amazonaws.com/nodegroup operator: In values: - ${NODEGROUP}注意这里没有karpenter.sh/nodepool: DoesNotExist这一条与第五节控制器 Deployment 的亲和互补控制器允许跑在任意非 Karpenter 节点上而关键组件被钉死在指定节点组。八、下线 Cluster Autoscaler8.1 关闭 CAS 控制器Karpenter 跑起来后把 CAS 的副本数缩为零即可停用它step11-scale-cas.shkubectl scale deploy/cluster-autoscaler -n kube-system --replicas08.2 缩减存量节点组将节点组缩到足以支撑 Karpenter 控制器和关键服务的规模单节点组、多可用区建议最小2 台step12-scale-single-ng.shaws eks update-nodegroup-config --cluster-name ${CLUSTER_NAME} \ --nodegroup-name ${NODEGROUP} \ --scaling-config minSize2,maxSize2,desiredSize2多个单可用区节点组每组最小1 台step12-scale-multiple-ng.shfor NODEGROUP in $(aws eks list-nodegroups --cluster-name ${CLUSTER_NAME} \ --query nodegroups --output text); do aws eks update-nodegroup-config --cluster-name ${CLUSTER_NAME} \ --nodegroup-name ${NODEGROUP} \ --scaling-config minSize1,maxSize1,desiredSize1 done注意如果工作负载没有配置 PodDisruptionBudget以下命令会导致工作负载不可用。节点较多或工作负载较多时建议每次缩减几台实例观察过渡过程重点关注副本数不足或未配置 PDB 的工作负载。九、验证 Karpenter 工作节点组节点被驱逐drain时可观察 Karpenter 是否为工作负载创建新节点kubectl logs -f -n ${KARPENTER_NAMESPACE} -l app.kubernetes.io/namekarpenter -c controller旧节点被移除的同时应能看到新节点加入集群kubectl get nodes新节点的标签会包含karpenter.sh/nodepooldefault与node.kubernetes.io/instance-type等信息旧 CAS 节点则带eks.amazonaws.com/nodegroup标签。两套标签并存正是迁移期双引擎状态的可观测体现。十、迁移要点回顾步骤核心动作对应脚本1提取 region / OIDC / AMI 别名等环境变量step01-env.sh2创建 KarpenterNodeRole 并附加 4 个托管策略step02-node-iam.sh、step03-node-policies.sh3创建 IRSA 控制器角色与权限策略step04-controller-iam.sh4子网、安全组打karpenter.sh/discovery标签step05-tag-subnets.sh、step06-tag-security-groups.sh5aws-auth 增加节点角色映射step07-edit-aws-auth.sh6helm template 渲染并修改亲和性后部署step08-generate-chart.sh、step09-deploy.sh7创建 NodePool EC2NodeClassstep10-create-nodepool.sh8CAS 缩零、节点组缩到最小规模step11-scale-cas.sh、step12-scale-single-ng.sh迁移的本质是让 Karpenter 与 CAS 在同一集群内短暂共存——Karpenter 只供给新节点通过karpenter.sh/nodepool标签区分CAS 管理的存量节点逐步缩容最终由 Karpenter 接管全部容量供给。整个过程不改动 VPC、子网与安全组的既有拓扑只增加标签与新 IAM 角色回滚路径清晰恢复 CAS 副本数即可接管新增量。【免费下载链接】karpenter-provider-awsKarpenter is a Kubernetes Node Autoscaler built for flexibility, performance, and simplicity.项目地址: https://gitcode.com/GitHub_Trending/ka/karpenter-provider-aws创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考