一、背景服务器由于升级原因临时断电断电前手动停止程序并手动关机k8s环境没有理会。恢复供电后问题处理.环境: centos7 docker k8s[rootwdy01 ~]# kubectl get nodeE0727 13:54:51.638488 160568 memcache.go:265] couldn’t get current server API group list: Get “https://10.1.1.1:6443/api?timeout32s”: dial tcp 10.1.1.1:6443: connect: connection refusedE0727 13:54:51.638859 160568 memcache.go:265] couldn’t get current server API group list: Get “https://10.1.1.1:6443/api?timeout32s”: dial tcp 10.1.1.1:6443: connect: connection refusedE0727 13:54:51.640220 160568 memcache.go:265] couldn’t get current server API group list: Get “https://10.1.1.1:6443/api?timeout32s”: dial tcp 10.1.1.1:6443: connect: connection refusedE0727 13:54:51.641583 160568 memcache.go:265] couldn’t get current server API group list: Get “https://10.1.1.1:6443/api?timeout32s”: dial tcp 10.1.1.1:6443: connect: connection refusedE0727 13:54:51.642889 160568 memcache.go:265] couldn’t get current server API group list: Get “https://10.1.1.1:6443/api?timeout32s”: dial tcp 10.1.1.1:6443: connect: connection refusedThe connection to the server 10.1.1.1:6443 was refused - did you specify the right host or port?[rootwdy01 ~]#二、解决过程2.1 先检查集群有效期发现正常[rootwdy01 ~]# kubeadm certs check-expiration[check-expiration]Reading configurationfromthe cluster...[check-expiration]FYI: You can look at this config file withkubectl -n kube-system get cm kubeadm-config -o yaml[check-expiration]Error reading configurationfromthe Cluster.Falling back to default configuration CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf Jul 07,2027 00:21 UTC 344d ca no apiserver Jul 07,2027 00:21 UTC 344d ca no apiserver-etcd-client Jul 07,2027 00:21 UTC 344d etcd-ca no apiserver-kubelet-client Jul 07,2027 00:21 UTC 344d ca no controller-manager.conf Jul 07,2027 00:21 UTC 344d ca no etcd-healthcheck-client Jul 07,2027 00:21 UTC 344d etcd-ca no etcd-peer Jul 07,2027 00:21 UTC 344d etcd-ca no etcd-server Jul 07,2027 00:21 UTC 344d etcd-ca no front-proxy-client Jul 07,2027 00:21 UTC 344d front-proxy-ca no scheduler.conf Jul 07,2027 00:21 UTC 344d ca no CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Jul 02,2035 01:45 UTC 8y no etcd-ca Jul 02,2035 01:45 UTC 8y no front-proxy-ca Jul 02,2035 01:45 UTC 8y no2.2 查看kubelet日志Jul2714:22:56 wdy01 systemd[1]: kubelet.service holdofftimeover, scheduling restart. Jul2714:22:56 wdy01 systemd[1]: Stopped kubelet: The Kubernetes Node Agent. Jul2714:22:56 wdy01 systemd[1]: Started kubelet: The Kubernetes Node Agent. Jul2714:22:56 wdy01 kubelet[237447]: Flag --container-runtime-endpoint has been deprecated, This parameter should besetvia the configfilespecified by the Kubelets--configflag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/formoreinformation. Jul2714:22:56 wdy01 kubelet[237447]: Flag --pod-infra-container-image has been deprecated, will be removedina future release. Image garbage collector will get sandbox image information from CRI. Jul2714:22:56 wdy01 kubelet[237447]: I072714:22:56.600080237447server.go:203]--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtimeJul2714:22:56 wdy01 kubelet[237447]: I072714:22:56.605137237447server.go:467]Kubelet versionkubeletVersionv1.2x.2Jul2714:22:56 wdy01 kubelet[237447]: I072714:22:56.605175237447server.go:469]Golang settingsGOGCGOMAXPROCSGOTRACEBACKJul2714:22:56 wdy01 kubelet[237447]: I072714:22:56.606111237447server.go:895]Client rotation is on, will bootstrap in backgroundJul2714:22:56 wdy01 kubelet[237447]: I072714:22:56.607831237447certificate_store.go:130]Loading cert/key pair from/var/lib/kubelet/pki/kubelet-client-current.pem.Jul2714:22:56 wdy01 kubelet[237447]: I072714:22:56.608791237447dynamic_cafile_content.go:157]Starting controllernameclient-ca-bundle::/etc/kubernetes/pki/ca.crtJul2714:22:56 wdy01 kubelet[237447]: W072714:22:56.609247237447logging.go:59][core][Channel#1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Jul2714:22:56 wdy01 kubelet[237447]:Addr:/var/run/cri-dockerd.sock, Jul2714:22:56 wdy01 kubelet[237447]:ServerName:/var/run/cri-dockerd.sock, Jul2714:22:56 wdy01 kubelet[237447]:Attributes:null, Jul2714:22:56 wdy01 kubelet[237447]:BalancerAttributes:null, Jul2714:22:56 wdy01 kubelet[237447]:Type:0, Jul2714:22:56 wdy01 kubelet[237447]:Metadata:null Jul2714:22:56 wdy01 kubelet[237447]:}. Err: connection error: desctransport: Error while dialing: dial unix /var/run/cri-dockerd.sock: connect: connection refusedJul2714:22:56 wdy01 kubelet[237447]: E072714:22:56.610057237447run.go:74]command failederrfailed to run Kubelet: validate service connection: validate CRI v1 runtime API for endpoint\unix:///var/run/cri-dockerd.sock\: rpc error: code Unavailable desc connection error: desc \transport: Error while dialing: dial unix /var/run/cri-dockerd.sock: connect: connection refused\Jul2714:22:56 wdy01 systemd[1]: kubelet.service: main process exited,codeexited,status1/FAILURE Jul2714:22:56 wdy01 systemd[1]: Unit kubelet.service entered failed state. Jul2714:22:56 wdy01 systemd[1]: kubelet.service failed.如上日志发现kubelet 启动失败核心原因dial unix /var/run/cri-dockerd.sock: connect: connection refusedcri-dockerd 服务没启动 / 未安装kubelet 无法对接 Docker 的 CRI 接口直接崩溃退出2.3 解决1查看服务状态systemctl status cri-dockerd2查看套接字文件是否存在ls -l /var/run/cri-dockerd.sock3发现断电重启后cri-dockerd服务丢失了我下载过 cri-dockerd-0.3.14-3.el7.x86_64.rpm执行rpm -ivh cri-dockerd-0.3.14-3.el7.x86_64.rpm安装。4重启并确认systemctl daemon-reload# 开机自启并启动systemctlenable--nowcri-docker# 查看状态确认runningsystemctl status cri-docker# 检查socket文件是否生成ls/var/run/cri-dockerd.sock[rootwdy02 ~]# ls /var/run/cri-dockerd.sock/var/run/cri-dockerd.sock[rootwdy02 ~]# systemctl status cri-docker● cri-docker.service - CRI InterfaceforDocker Application Container Engine Loaded: loaded(/usr/lib/systemd/system/cri-docker.service;enabled;vendor preset: disabled)Active: active(running)since Thu2026-08-2711:14:06 CST;1h 26min ago Docs: https://docs.mirantis.com Main PID:231334(cri-dockerd)Tasks:38Memory:46.6M CGroup: /system.slice/cri-docker.service └─231334 /usr/bin/cri-dockerd --container-runtime-endpoint fd:// --pod-infra-container-imageregistry.aliyuncs.com/google_containers/... Aug2711:31:06 wdy02 cri-dockerd[231334]: delegateAdd: netconf sent to delegate plugin: Aug2711:32:02 wdy02 cri-dockerd[231334]:{cniVersion:0.3.1,hairpinMode:true,ipMasq:false,ipam:{ranges:[[{subnet:10.244... msgWi Aug2711:32:03 wdy02 cri-dockerd[231334]: map[string]interface{}{cniVersion:0.3.1,hairpinMode:true,ipMasq:false,ipam:map...IP{0xa, Aug2711:32:03 wdy02 cri-dockerd[231334]: delegateAdd: netconf sent to delegate plugin: Aug2712:17:16 wdy02 cri-dockerd[231334]:{cniVersion:0.3.1,hairpinMode:true,ipMasq:false,ipam:{ranges:[[{subnet:10.244... msgWi Aug2712:17:16 wdy02 cri-dockerd[231334]: map[string]interface{}{cniVersion:0.3.1,hairpinMode:true,ipMasq:false,ipam:map...IP{0xa, Aug2712:17:16 wdy02 cri-dockerd[231334]: delegateAdd: netconf sent to delegate plugin: Aug2712:17:17 wdy02 cri-dockerd[231334]:{cniVersion:0.3.1,hairpinMode:true,ipMasq:false,ipam:{ranges:[[{subnet:10.244... msgWi Aug2712:17:17 wdy02 cri-dockerd[231334]: map[string]interface{}{cniVersion:0.3.1,hairpinMode:true,ipMasq:false,ipam:map...IP{0xa, Aug2712:17:17 wdy02 cri-dockerd[231334]: delegateAdd: netconf sent to delegate plugin: Hint: Some lines were ellipsized, use-lto showinfull.5重启并确认修复 kubelet 配置消除 cri-dockerd.sock 连接拒绝报错vi/var/lib/kubelet/config.yaml追加一行没有就加有就确认一致containerRuntimeEndpoint: unix:///var/run/cri-dockerd.sock6) 重启kubelet重启kubeletsystemctl daemon-reload systemctl restart kubelet# 实时观察日志journalctl-ukubelet-f如有必要重启kube-apiserver、kube-controller-manage、kube-scheduler# 如果是docker作为容器的话可执行如下命令。其余容器方法类似dockerps|grepkube-apiserver|grep-vpause|awk{print $1}|xargs-idockerrestart{}dockerps|grepkube-controller-manage|grep-vpause|awk{print $1}|xargs-idockerrestart{}dockerps|grepkube-scheduler|grep-vpause|awk{print $1}|xargs-idockerrestart{}三、补充补充1根据实际情况docker必须保障正常。systemctl restart dockersystemctl enable docker补充2如启动后发现worker节点也没有正常ready则worker节点也需要排查并安装cri-dockerd。四、结果如下图执行kubectl get nodes查看集群状态已正常。五、新问题后端程序链接nacos报错java.net.NoRouteToHostException: No route to host (Host unreachable)测试发现宿主机worker是可以连通01部署的nacos的。但程序pod内不通。关闭防火墙也不行。最后发现02 节点上 docker/flannel 残留 iptables nat 规则 拦截了 Pod 访问宿主机网段的回包清空 iptables 后立刻恢复连通。最后是在02 清空所有 iptables 规则natfilter 解决的。# 清空过滤链iptables-F# 清空nat转发链关键iptables-tnat-F# 设置默认转发允许iptables-PFORWARD ACCEPT## 补充1过了一段时间不知什么原因又出现NoRouteToHostException了最后是这样解决的首先是docker配置文件添加编辑/etc/docker/daemon.json增加ip-forward:false## 修改 docker 配置防止重启又生成 DOCKER 链不让docker乱改 filter 表 FORWARD 链、不注入 DOCKER 隔离防火墙规则内核 IP 转发仍然开启calico/kube‑proxy 全权接管集群网络。 bash{exec-opts:[native.cgroupdriversystemd],data-root:/UData/docker,insecure-registries:[ip:81],registry-mirrors:[https://docker.1panel.live],log-driver:json-file,log-opts:{max-size:100m,max-file:5},ip-forward:false}重启 docker 与 cri‑dockerdsystemctl daemon-reload systemctl restartdockercri-docker删除旧的规则1)先删除 FORWARD 链中跳转的引用关键解决 Too many links# 删除FORWARD里跳转到DOCKER‑USERiptables-DFORWARD-jDOCKER-USER# 删除FORWARD里跳转到DOCKER‑ISOLATION‑STAGE‑1iptables-DFORWARD-jDOCKER-ISOLATION-STAGE-1# 删除FORWARD里跳转到FLANNEL‑FWDiptables-DFORWARD-jFLANNEL-FWD# 删除firewalld残留跳转iptables-DFORWARD-jFORWARD_direct iptables-DFORWARD-jFORWARD_IN_ZONES_SOURCE iptables-DFORWARD-jFORWARD_IN_ZONES iptables-DFORWARD-jFORWARD_OUT_ZONES_SOURCE iptables-DFORWARD-jFORWARD_OUT_ZONES2)现在就可以清空并删除这些链# docker链iptables-FDOCKER-USERiptables-XDOCKER-USERiptables-FDOCKER-ISOLATION-STAGE-1 iptables-XDOCKER-ISOLATION-STAGE-1 iptables-FDOCKER iptables-XDOCKER# flanneliptables-FFLANNEL-FWD iptables-XFLANNEL-FWD# firewalld残留iptables-FFORWARD_direct iptables-XFORWARD_direct iptables-FFORWARD_IN_ZONES iptables-XFORWARD_IN_ZONES iptables-FFORWARD_IN_ZONES_SOURCE iptables-XFORWARD_IN_ZONES_SOURCE iptables-FFORWARD_OUT_ZONES iptables-XFORWARD_OUT_ZONES iptables-FFORWARD_OUT_ZONES_SOURCE iptables-XFORWARD_OUT_ZONES_SOURCE3)删除那两条致命 REJECT 规则不用‑m reject按行序号删除先查看行号iptables-LFORWARD-n--line-numbers输出会显示每一行的 num 编号找到两条 REJECT 行的行号例如假设行号是 20、24# 替换为你实际看到的行号先删大序号再删小序号iptables-DFORWARD24iptables-DFORWARD20注意删除行号的时候先删数字大的行因为删掉前面行后面行号会变化。4清理完核对结果[rootwoker02 ~]# iptables -L FORWARD -n --line-numbers Chain FORWARD (policy ACCEPT) num target prot opt source destination 1 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 2 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 3 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 4 KUBE-PROXY-FIREWALL all -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes load balancer firewall */ 5 KUBE-FORWARD all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding rules */ 6 KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes service portals */ 7 KUBE-EXTERNAL-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes externally-visible service portals */ 8 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 9 ACCEPT all -- 0.0.0.0/0 192.168.122.0/24 ctstate RELATED,ESTABLISHED 10 ACCEPT all -- 192.168.122.0/24 0.0.0.0/0 11 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 12 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 13 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 14 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 15 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 16 FLANNEL-FWD all -- 0.0.0.0/0 0.0.0.0/0 /* flanneld forward */END