ETCD 集群写满导致集群退出

发布时间:2026/7/8 3:29:37

ETCD 集群写满导致集群退出 [roothost-192.168.16-74 ~]# etcdctl --endpointshttp://127.0.0.1:2379 endpoint status --write-outjson | jq .[0].Status.header.revision 7407956 # 假设返回 123456压缩到该 revision保留最新版本 [roothost-192.168.16-74 ~]# etcdctl --endpointshttp://127.0.0.1:2379 compact 7407956 compacted revision 7407956 #整理碎片 etcdctl --endpointshttp://127.0.0.1:2379 defrag [roothost-192.168.16-74 ~]# etcdctl --endpointshttp://127.0.0.1:2379 --dial-timeout10s endpoint status --write-outtable --------------------------------------------------------------------------------------------- | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX | --------------------------------------------------------------------------------------------- | http://127.0.0.1:2379 | 60069b4d95cd9901 | 3.3.11 | 1.4 GB | true | 6986 | 14471872 | --------------------------------------------------------------------------------------------- #每个节点都要执行。 [roothost-192.168.16-75 ~]# ETCDCTL_API3 etcdctl --endpointshttp://127.0.0.1:2379 --dial-timeout10s endpoint status --write-outtable --------------------------------------------------------------------------------------------- | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX | --------------------------------------------------------------------------------------------- | http://127.0.0.1:2379 | e9fb6ab6aa554f62 | 3.3.11 | 2.1 GB | false | 6986 | 14472089 | --------------------------------------------------------------------------------------------- [roothost-192.168.16-75 ~]# ETCDCTL_API3 etcdctl --endpointshttp://192.168.16.74:2379,http://192.168.16.75:2379,http://192.168.16.76:2379 defrag Finished defragmenting etcd member[http://192.168.16.74:2379] Finished defragmenting etcd member[http://192.168.16.75:2379] Finished defragmenting etcd member[http://192.168.16.76:2379] ETCDCTL_API3 etcdctl --endpointshttp://192.168.16.74:2379,http://192.168.16.75:2379,http://192.168.16.76:2379 --dial-timeout10s endpoint status --write-outtable [roothost-192.168.16-75 ~]# ETCDCTL_API3 etcdctl --endpointshttp://192.168.16.74:2379,http://192.168.16.75:2379,http://192.168.16.76:2379 --dial-timeout10s endpoint status --write-outtable ------------------------------------------------------------------------------------------------ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX | ------------------------------------------------------------------------------------------------ | http://192.168.16.74:2379 | 60069b4d95cd9901 | 3.3.11 | 25 kB | true | 6986 | 14472154 | | http://192.168.16.75:2379 | e9fb6ab6aa554f62 | 3.3.11 | 25 kB | false | 6986 | 14472154 | | http://192.168.16.76:2379 | e7dbae353e23179e | 3.3.11 | 25 kB | false | 6986 | 14472154 | ------------------------------------------------------------------------------------------------ #--quota-backend-bytes8589934592 #配置ETCD 的总磁盘可使用8G;默认2G; cat /usr/lib/systemd/system/etcd.service [Unit] DescriptionEtcd Server Afternetwork.target Afternetwork-online.target Wantsnetwork-online.target [Service] Typenotify WorkingDirectory/var/lib/etcd/ EnvironmentFile-/etc/etcd/etcd.conf Useretcd # set GOMAXPROCS to number of processors ExecStart/bin/bash -c GOMAXPROCS$(nproc) /usr/bin/etcd --name\${ETCD_NAME}\ --data-dir\${ETCD_DATA_DIR}\ --listen-client-urls\${ETCD_LISTEN_CLIENT_URLS}\ --quota-backend-bytes8589934592 Restarton-failure LimitNOFILE65536 [Install] WantedBymulti-user.target #patroni依然报错。 Jul 07 16:17:43 host-192.168.16-75 patroni[61280]: raise _raise_for_data(ret or data, response.status) Jul 07 16:17:43 host-192.168.16-75 patroni[61280]: patroni.dcs.etcd3.Unknown: Unknown error: etcdserver: mvcc: database space exceeded, code: 2 Jul 07 16:17:43 host-192.168.16-75 patroni[61280]: 2026-07-07 16:17:43,356 INFO: waiting on etcd #查看警告. [roothost-192.168.16-75 ~]# ETCDCTL_API3 etcdctl --endpointshttp://192.168.16.74:2379,http://192.168.16.75:2379,http://192.168.16.76:2379 alarm list memberID:16707138786359973790 alarm:NOSPACE #解除告警让 etcd 允许写入 [roothost-192.168.16-75 ~]# ETCDCTL_API3 etcdctl --endpointshttp://192.168.16.74:2379,http://192.168.16.75:2379,http://192.168.16.76:2379 alarm disarm memberID:16707138786359973790 alarm:NOSPACE #重新查看集群已恢复正常。 [roothost-192.168.16-75 ~]# systemctl status patroni ● patroni.service - Runners to orchestrate a high-availability PostgreSQL Loaded: loaded (/etc/systemd/system/patroni.service; disabled; vendor preset: disabled) Active: active (running) since Tue 2026-07-07 16:17:42 CST; 5min ago Main PID: 61280 (patroni) CGroup: /system.slice/patroni.service ├─61280 /usr/bin/python3 /usr/local/bin/patroni /etc/patroni.yml ├─61362 /data/pgsql/13/bin/postgres -D /data/pgsql/13/data --config-file/data/pgsql/13/data/postgresql.conf --listen_addresses0.0.0.0 --port5432 --cluster_nameyace_pgsql... ├─61364 postgres: yace_pgsql: logger ├─61365 postgres: yace_pgsql: startup recovering 0000000E000000110000004D ├─61372 postgres: yace_pgsql: checkpointer ├─61373 postgres: yace_pgsql: background writer ├─61374 postgres: yace_pgsql: stats collector ├─61375 postgres: yace_pgsql: walreceiver streaming 11/4D02B1F0 └─61383 postgres: yace_pgsql: postgres postgres 127.0.0.1(52950) idle Jul 07 16:21:57 host-192.168.16-75 patroni[61280]: localhost:5432 - rejecting connections Jul 07 16:21:58 host-192.168.16-75 patroni[61280]: localhost:5432 - rejecting connections Jul 07 16:21:59 host-192.168.16-75 patroni[61280]: localhost:5432 - accepting connections Jul 07 16:21:59 host-192.168.16-75 patroni[61280]: 2026-07-07 16:21:59,258 INFO: Lock owner: pg01; I am pg02 Jul 07 16:21:59 host-192.168.16-75 patroni[61280]: 2026-07-07 16:21:59,259 INFO: establishing a new patroni connection to the postgres cluster Jul 07 16:21:59 host-192.168.16-75 patroni[61280]: 2026-07-07 16:21:59,306 INFO: no action. I am (pg02), a secondary, and following a leader (pg01) Jul 07 16:22:03 host-192.168.16-75 patroni[61280]: 2026-07-07 16:22:03,527 INFO: no action. I am (pg02), a secondary, and following a leader (pg01) Jul 07 16:22:13 host-192.168.16-75 patroni[61280]: 2026-07-07 16:22:13,493 INFO: no action. I am (pg02), a secondary, and following a leader (pg01) Jul 07 16:22:23 host-192.168.16-75 patroni[61280]: 2026-07-07 16:22:23,524 INFO: no action. I am (pg02), a secondary, and following a leader (pg01) Jul 07 16:22:33 host-192.168.16-75 patroni[61280]: 2026-07-07 16:22:33,490 INFO: no action. I am (pg02), a secondary, and following a leader (pg01)

相关新闻