70. 配置 kube-proxy以配合 RKE2 使用 nftables iptables 后端

环境RKE2情况In some circumstances, the auto-detection of the iptables backend used bykube-proxymay switch between thelegacyandnftablesbackends. This can result in inconsistent or missing NAT rules, causing network communication issues between pods, services, or nodes.在某些情况下kube 代理所用 iptables 后端的自动检测可能会在传统后端和NFTABLES后端之间切换。这可能导致 NAT 规则不一致或缺失导致 Pod、服务或节点之间的网络通信问题。This article explains how to explicitly configurekube-proxyto use thenftables(modern) backend by setting theIPTABLES_MODEenvironment variable.本文解释了如何通过设置IPTABLES_MODE环境变量显式配置kube-proxy以使用nftables现代后端。You may observe one or more of the following symptoms:你可能会观察到以下一种或多种症状TheiptablesNAT table appears incomplete or missing expected rulesiptablesNAT 表似乎不完整或缺少预期规则Traffic between pods or services intermittently failsPod 或服务之间的流量间歇性失败The nodes OS uses thenftablesbackend (iptables-nftpackage), butkube-proxyis managing rules with thelegacybackend节点的操作系统使用nftables的后端iptables-nft包但kube-proxy则用遗留后端管理规则This can be confirmed on RKE2 nodes where the issue is suspected:在怀疑问题的 RKE2 节点上这一点可以得到确认List all of the kube-proxy pods:列出所有 kube 代理 podsspan stylecolor:#000000span stylebackground-color:#ffffffspan stylebackground-color:#efefefcodekubectl get pods -n kube-system -l componentkube-proxy -o wide/code/span/span/spanUpdate the commands to run the kubectl exec commands on the kube-proxy pod of the related node(s):更新命令在相关节点的 kube-proxy pod 上运行 kubectl 执行命令span stylecolor:#000000span stylebackground-color:#ffffffspan stylebackground-color:#efefefcodekubectl exec -it -n kube-system kube-proxy-xxxx -- /usr/sbin/iptables-legacy-save # Compare with nft kubectl exec -it -n kube-system kube-proxy-xxxx -- /usr/sbin/iptables-nft-save/code/span/span/spanThe output for legacy should either not contain any rules, or only the default chains with no rules. The threshold for auto-detection is 10 rules, so there should at least be fewer than 10 legacy rules遗留输出要么不包含任何规则要么只包含默认链没有规则。自动检测的门槛是10条规则所以遗留规则至少应该少于10条Review any legacy rules, if the rules are not added by any Kubernetes components, these custom rules should be updated to ensure they are added to the NFT backend. Avoiding these legacy rules may be related to the auto-detection switching to the legacy backend审查任何遗留规则如果没有任何 Kubernetes 组件添加这些规则这些自定义规则应更新以确保添加到 NFT 后端。避免这些遗留规则可能与自动检测切换到遗留后端有关解决方案If the auto-detection has switched, as a preventative measure it is recommended to ensure consistent networking behaviour and explicitly pinkube-proxyto use the nftables backend in all nodes in the cluster, this can be accomplished by setting the environment variableIPTABLES_MODEnft.如果自动检测已经切换作为预防措施建议确保网络行为一致并明确将kube-proxy钉在集群所有节点使用 nftables 后端这可以通过设置环境变量IPTABLES_MODEnft实现。Steps (via Rancher UI)步骤通过 Rancher 用户界面In the Rancher UI, go toCluster Management→Clusters在 Rancher 界面中进入集群管理→集群Select the affecteddownstream cluster选择受影响的下游集群Click⋮ → Edit Config点击⋮ →编辑配置ChooseEdit as YAML选择编辑为 YAML 编辑Locate themachineGlobalConfigsection and add the following:找到machineGlobalConfig部分添加以下内容span stylecolor:#000000span stylebackground-color:#ffffffspan stylebackground-color:#efefefcodemachineGlobalConfig: # existing configuration kube-proxy-extra-env: - IPTABLES_MODEnft /code/span/span/spanClickSave点击保存The configuration will roll out to each node, and thekube-proxypods will automatically restart with the updated environment variable配置会推送到每个节点kube 代理单元会自动重启更新后的环境变量Notethe same change can be accomplished on standalone clusters that arent managed by Rancher, by adding the same kube-proxy-extra-env field in the config.yaml of each node. The rke2-agent/server service will need to be restarted to apply the change注意同样的更改也适用于非 Rancher 管理的独立集群只需在每个节点的 config.yaml 中添加相同的 kube-proxy-extra-env 字段 。需要重启 rke2 代理/服务器服务来应用变更Verification 验证After the configuration is applied:配置应用后Check thatkube-proxyis running with the correct environment:检查kube-proxy是否在正确的环境中运行span stylecolor:#000000span stylebackground-color:#ffffffspan stylebackground-color:#efefefcodekubectl -n kube-system exec -it kube-proxy-xxxx -- env | grep IPTABLES_MODE/code/span/span/spanExpected output: 预期输出IPTABLES_MODEnftConfirm that NAT rules are populated and accurate under nftables:确认 NFTABLE 下的 NAT 规则是否被填充且准确span stylecolor:#000000span stylebackground-color:#ffffffspan stylebackground-color:#efefefcodekubectl exec -it -n kube-system kube-proxy-xxxx -- /usr/sbin/iptables-nft -nvL/code/span/span/spanor on the node directly或直接在节点上span stylecolor:#000000span stylebackground-color:#ffffffspan stylebackground-color:#efefefcodenft list ruleset | grep KUBE-/code/span/span/span原因CNIs, runtimes and kube-proxy rely on internal logic to detect the iptables mode (nftorlegacy) in use on the OS. In some corner cases, this auto-detection can fail and use the incorrect backend — for example:CNI、运行时和 kube-proxy 依赖内部逻辑来检测操作系统上正在使用的 iptables 模式NFT或遗留模式。在某些角落案例中这种自动检测可能失败使用错误的后端——例如When both legacy and nft iptables binaries are installed (/usr/sbin/iptablesand/usr/sbin/iptables-legacy) and may be used by different scripts, applications or incorrectly symlinked. As mentioned above, the 10 rule threshold could be met if custom rules are added to the incorrect backed当同时安装了遗留和 NFT iptables 二进制文件/usr/sbin/iptables和/usr/sbin/iptables-legacy可能会被不同脚本、应用程序使用或符号链接错误。如上所述如果在错误的后台中添加自定义规则10 条规则的门槛是可以达到的When the system alternates between iptables providers during OS updates or security patching当系统在操作系统更新或安全补丁期间交替使用 iptables 提供者时If the OS is heavily contended for resources the auto-detection logic may fail during startup and default to the incorrect backend如果操作系统资源被严重占用自动检测逻辑可能在启动时失败默认使用错误的后端This can lead toinconsistent NAT rule management— kube-proxy writes rules to one backend (e.g., legacy) while the kernel actually uses nftables rules already in place.这可能导致NAT 规则管理不一致——kube 代理将规则写入一个后端例如遗留而内核实际上使用已有的 nftables 规则。附加信息https://kubernetes.io/blog/2025/02/28/nftables-kube-proxy/https://docs.rke2.io/reference/linux_agent_config#components访问Rancher-K8S解决方案博主 https://blog.csdn.net/lidw2009