【基于 K8S+NFS 动态存储实战部署 Redis-Cluster 集群（含三主三从配置与访问配置）】

提示本文原创作品良心制作干货为主简洁清晰一看就会文章目录前言一、整体流程概述二、NFSK8S部署redis-cluster集群实战2.1 安装NFS2.2 创建SC2.3 创建SA并授权2.4 部署自动创建PV的pod服务2.5 创建Redis的Sts2.6 测试集群内Redis域名解析连通性2.7 安装redis-trib2.8 配置Redis三主三从2.9 创建用于访问Redis的SVC前言在Kubernetes中搭建Redis Cluster是实现高可用、可扩展缓存服务的关键实践。本文从底层存储配置出发完整演示基于NFS动态供给的Redis集群部署流程帮助你快速构建一套生产可用的分布式缓存系统本文实验基于 K8s PV/PVC 动态存储前置知识对此尚不熟悉的同学可先参阅此篇文章https://blog.csdn.net/m0_63756214/article/details/160648532?spm1001.2014.3001.5502一、整体流程概述1准备一台NFS服务器为 K8s 提供统一后端存储供 PV/PVC 挂载使用2创建StorageClass、ServiceAccount 并授权部署自动供给 PV/PVC 的 NFS Provisioner Pod3创建ConfigMap存储自定义 redis.conf 配置文件创建Headless Service为 Redis 有状态应用提供固定域名解析4部署StatefulSet引用 ConfigMap 与 Service配置 6 副本通过 StorageClass 自动申请 NFS 持久化存储5如果是centos系统则创建一个ubuntu的pod在 Ubuntu 容器内安装集群管理工具redis-trib将 6 个 Redis 节点配置为三主三从高可用集群如果是ubuntu系统直接安装集群管理工具redis-trib执行相关命令二、NFSK8S部署redis-cluster集群实战2.1 安装NFS## 准备一台机器安装NFS后续的PV/PVC会到NFS中拿空间## 我这台nfs的ip是192.168.13.137rootNFS:~# apt -y install nfs-kernel-serverrootNFS:~# systemctl start nfs-kernel-serverrootNFS:~# mkdir /datarootNFS:~# vim /etc/exports/data*(rw,sync,no_root_squash,no_subtree_check) rootNFS:~# chmod 777 /datarootNFS:~# systemctl restart nfs-kernel-serverrootNFS:~# systemctl enable nfs-kernel-server## 在K8S的每个node节点都安装nfs客户端不然会挂载失败rootk8s-node1:~# apt -y install nfs-commonrootk8s-node2:~# apt -y install nfs-common2.2 创建SCrootk8s-master1:~# vim storageclass-nfs.yamlapiVersion:storage.k8s.io/v1kind:StorageClassmetadata:name:managed-nfs-storage## 指定外部NFS供给器的名字provisioner:k8s-sigs.io/nfs-subdir-external-provisioner rootk8s-master1:~# kubectl apply -f storageclass-nfs.yaml2.3 创建SA并授权## 创建RBACrootk8s-master1:~# vim rbac.yaml## ServiceAccountNFS provisioner pod 使用的身份apiVersion:v1kind:ServiceAccountmetadata:name:nfs-client-provisioner# 需要指定命名空间namespace:default---## ClusterRole定义集群维度的权限apiVersion:rbac.authorization.k8s.io/v1kind:ClusterRolemetadata:name:nfs-client-provisioner-ClusterRolerules:-apiGroups:[]resources:[nodes]# 需要获取节点信息用于拓扑感知verbs:[get,list,watch]-apiGroups:[]resources:[persistentvolumes]# 需要全量操作PVverbs:[get,list,watch,create,delete]-apiGroups:[]resources:[persistentvolumeclaims]# 需要监控PVC并更新其状态verbs:[get,list,watch,update]-apiGroups:[storage.k8s.io]resources:[storageclasses]# 需要获取storageclasses信息verbs:[get,list,watch]-apiGroups:[]resources:[events]# 需要记录事件verbs:[create,update,patch]---## ClusterRoleBinding将ServiceAccount绑定到ClusterRoleapiVersion:rbac.authorization.k8s.io/v1kind:ClusterRoleBindingmetadata:name:nfs-clinet-provisioner-ClusterRoleBindingsubjects:-kind:ServiceAccountname:nfs-client-provisioner# 必须添加ServiceAccount所在命名空间namespace:defaultroleRef:kind:ClusterRolename:nfs-clinet-provisioner-ClusterRoleapiGroup:rbac.authorization.k8s.io---## Role: 命名空间维度的权限可以和clusterrole合并apiVersion:rbac.authorization.k8s.io/v1kind:Rolemetadata:name:nfs-clinet-provisioner-Rolenamespace:defaultrules:-apiGroups:[]resources:[endpoints]# 某些provisioner实现会使用endpoints做leader electionverbs:[get,list,watch,create,update,patch]---apiVersion:rbac.authorization.k8s.io/v1kind:RoleBindingmetadata:name:nfs-clinet-provisioner-RoleBindingnamespace:defaultsubjects:-kind:ServiceAccountname:nfs-client-provisioner# 必须添加ServiceAccount所在命名空间namespace:defaultroleRef:kind:Rolename:nfs-clinet-provisioner-RoleapiGroup:rbac.authorization.k8s.iorootk8s-master1:~# kubectl apply -f rbac.yaml2.4 部署自动创建PV的pod服务rootk8s-master1:~# vim deploy-nfs.yamlapiVersion:apps/v1kind:Deploymentmetadata:name:nfs-client-provisionernamespace:defaultspec:replicas:1selector:matchLabels:app:nfs-client-provisionerstrategy:# 升级策略type:Recreate# 当需要更新Pod时K8S会先删除现有的Pod再创建新的Pod这适用于不能同时运行多个副本的有状态服务template:metadata:labels:app:nfs-client-provisionerspec:# 指定pod运行时使用的sa名称,其绑定的RBAC权限决定了provisioner能够访问哪些K8S API资源serviceAccountName:nfs-client-provisionercontainers:-name:nfs-client-provisioner# 这里使用阿里云镜像仓库中的nfs-subdir-external-provisioner镜像版本 v4.0.0该镜像实现了nfs动态供给器image:registry.cn-beijing.aliyuncs.com/pylixm/nfs-subdir-external-provisioner:v4.0.0volumeMounts:-name:nfs-root# 将nfs共享挂载到容器内的 /persistentvolumes 路径,不建议改provisioner 会在该目录下为每个pvc创建子目录mountPath:/persistentvolumesenv:-name:PROVISIONER_NAME# 必须与StorageClass中的 provisioner 字段完全一致这样StorageClass才能调用该供给器value:k8s-sigs.io/nfs-subdir-external-provisioner-name:NFS_SERVERvalue:192.168.13.137# nfs服务器的ip地址-name:NFS_PATHvalue:/data# nfs服务器上导出的共享目录路径volumes:-name:nfs-rootnfs:server:192.168.13.137path:/datarootk8s-master1:~# kubectl apply -f deploy-nfs.yaml2.5 创建Redis的Stsrootk8s-master1:~# cat redis-cm-svc-pod.yaml# 1. Redis 配置文件 ConfigMapapiVersion:v1kind:ConfigMapmetadata:name:redis-confdata:redis.conf:|port 6379 cluster-enabled yes cluster-config-file /var/lib/redis/nodes.conf cluster-node-timeout 5000 appendonly yes dir /var/lib/redis # 数据存放目录 protected-mode no daemonize no pidfile /var/run/redis.pid logfile ---# 2. Redis Headless Service集群必须apiVersion:v1kind:Servicemetadata:name:redis-servicelabels:app:redisspec:ports:-name:redis-portport:6379clusterIP:None# 不分配clusterIPselector:app:redisappCluster:redis-cluster---# 3. Redis StatefulSet 有状态部署apiVersion:apps/v1kind:StatefulSetmetadata:name:redisspec:# 指定关联的Headless Service名称必须与Service同名serviceName:redis-service# 创建6个Redis实例3主3从replicas:6selector:matchLabels:app:redisappCluster:redis-clustertemplate:metadata:labels:app:redisappCluster:redis-clusterspec:containers:-name:redisimage:redis:latest# 启动命令执行redis-servercommand:-redis-server# 启动参数指定配置文件 关闭保护模式args:-/etc/redis/redis.conf# 加载自定义配置---protected-mode# 关闭保护模式允许集群通信-no# 资源限制CPU与内存申请resources:requests:cpu:100mmemory:200Mi# 声明容器内部监听了什么端口,不提供流量转发与访问暴露ports:-containerPort:6379# redis客户端连接端口name:redisprotocol:TCP-containerPort:16379# redis 集群节点间通信的端口name:clusterprotocol:TCPvolumeMounts:-name:redis-config# 挂载配置文件mountPath:/etc/redis/redis.confsubPath:redis.conf-name:redis-data# 挂载数据文件mountPath:/var/lib/redis# 挂载ConfigMap配置volumes:-name:redis-configconfigMap:name:redis-conf# 动态存储申请配合之前的 NFS StorageClassvolumeClaimTemplates:-metadata:name:redis-dataspec:accessModes:[ReadWriteOnce]# 访问模式单节点读写resources:requests:storage:1Gi# 申请大小# 这里使用上面创建的 StorageClassstorageClassName:managed-nfs-storagerootk8s-master1:~# kubectl apply -f redis-cm-svc-pod.yamlrootk8s-master1:~# kubectl get svcNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 none443/TCP 37d redis-service ClusterIP None none6379/TCP 17s## 通过创建时间可以看出sts在创建pod的时候不是同时启动的而是一个一个的顺序启动rootk8s-master1:~# kubectl get pod -o wide | grep redisredis-0 1/1 Running 0 3m53s 10.244.36.108 k8s-node1 nonenoneredis-1 1/1 Running 0 3m31s 10.244.169.183 k8s-node2 nonenoneredis-2 1/1 Running 0 2m51s 10.244.36.109 k8s-node1 nonenoneredis-3 1/1 Running 0 2m26s 10.244.169.184 k8s-node2 nonenoneredis-4 1/1 Running 0 2m1s 10.244.36.110 k8s-node1 nonenoneredis-5 1/1 Running 0 98s 10.244.169.185 k8s-node2 nonenone## 查看nfs的/data目录会产生数据rootNFS:~# ls /data/default-redis-data-redis-0-pvc-992e88f2-1460-4b48-8096-491dc31fa274 default-redis-data-redis-3-pvc-57b4f9bf-50b6-4057-a844-fed705b1ec8f default-redis-data-redis-1-pvc-d9414620-ec81-4464-906a-473e54882199 default-redis-data-redis-4-pvc-eab88346-8fc9-4467-ad71-9f044ec412c2 default-redis-data-redis-2-pvc-299b0bc2-c6c8-4a3f-b068-f65b365d9e2d default-redis-data-redis-5-pvc-644ffafe-9d26-4852-81ea-e46e9082f481 rootNFS:~# ls /data/default-redis-data-redis-0-pvc-992e88f2-1460-4b48-8096-491dc31fa274/appendonlydir nodes.conf rootNFS:~#2.6 测试集群内Redis域名解析连通性StatefulSet 中的每个 Pod 会获得形如 pod-name.service-name.namespace.svc.cluster.local 的稳定 DNS 名称例如redis-0.redis.default.svc.cluster.local即使 Pod 重新调度该域名也不会改变我们可以进入 busybox 测试容器对 Redis 各节点域名进行连通性验证确保 Kubernetes 集群内部 DNS 解析正常、Pod 网络通信正为后续 Redis 集群组建提供稳定的网络环境rootk8s-master1:~# kubectl run busybox --imagebusybox -it -- /bin/shIf you dont see a command prompt,try pressing enter. /# ping redis-0.redis.default.svc.cluster.localping:bad address redis-0.redis.default.svc.cluster.local /# ping redis-0.redis-service.default.svc.cluster.local # ping该域名解析出来的就是redis-0对应的ip 10.244.36.108PING redis-0.redis-service.default.svc.cluster.local (10.244.36.108):56 data bytes64 bytes from 10.244.36.108:seq0 ttl63 time0.648 ms64 bytes from 10.244.36.108:seq1 ttl63 time0.084 ms64 bytes from 10.244.36.108:seq2 ttl63 time0.086 ms ^C---redis-0.redis-service.default.svc.cluster.local ping statistics---3 packets transmitted,3 packets received,0% packet loss round-trip min/avg/max 0.084/0.272/0.648 ms /# ping redis-4.redis-service.default.svc.cluster.local # ping该域名解析出来的就是redis-4对应的ip 10.244.36.110PING redis-4.redis-service.default.svc.cluster.local (10.244.36.110):56 data bytes64 bytes from 10.244.36.110:seq0 ttl63 time0.099 ms64 bytes from 10.244.36.110:seq1 ttl63 time0.107 ms64 bytes from 10.244.36.110:seq2 ttl63 time0.169 ms ^C---redis-4.redis-service.default.svc.cluster.local ping statistics---3 packets transmitted,3 packets received,0% packet loss round-trip min/avg/max 0.099/0.125/0.169 ms /#2.7 安装redis-trib需要先运行一个ubuntu的pod然后在ubuntu中安装redis-trib命令然后使用redis-tribrootk8s-master1:~# kubectl run -it ubuntu --imageubuntu:18.04 --restartNever /bin/bashIf you dont see a command prompt,try pressing enter.## 把软件下载源替换成阿里云镜像源rootubuntu:/# cat /etc/apt/sources.list EOFdeb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiversedeb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiversedeb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiversedeb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiversedeb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiversedeb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiversedeb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiversedeb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiversedeb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiversedeb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverseEOF rootubuntu:/# apt update## 安装相关命令rootubuntu:/# apt install -y python2.7 python-pip redis-tools dnsutilsrootubuntu:/# pip install redis-trib0.5.1## 查这个域名在 K8s 里的ip地址可以测试一下是否正确rootubuntu:/# dig short redis-0.redis-service.default.svc.cluster.local10.244.36.108 rootubuntu:/# dig short redis-5.redis-service.default.svc.cluster.local10.244.169.185 rootubuntu:/#2.8 配置Redis三主三从在Ubuntu pod中设置redis三主三从## 设置redis-0 redis-1 redis-2 为masterrootubuntu:/# redis-trib.py create \dig short redis-0.redis-service.default.svc.cluster.local:6379 \dig short redis-1.redis-service.default.svc.cluster.local:6379 \dig short redis-2.redis-service.default.svc.cluster.local:6379## 设置redis-3为redis-0的slaverootubuntu:/# redis-trib.py replicate --master-addr dig short redis-0.redis-service.default.svc.cluster.local:6379 --slave-addr dig short redis-3.redis-service.default.svc.cluster.local:6379## 设置redis-4为redis-1的slaverootubuntu:/# redis-trib.py replicate --master-addr dig short redis-1.redis-service.default.svc.cluster.local:6379 --slave-addr dig short redis-4.redis-service.default.svc.cluster.local:6379## 设置redis-5为redis-2的slaverootubuntu:/# redis-trib.py replicate --master-addr dig short redis-2.redis-service.default.svc.cluster.local:6379 --slave-addr dig short redis-5.redis-service.default.svc.cluster.local:6379## 退出ubuntu pod然后随意进入一个redis pod查看集群信息rootk8s-master1:~# kubectl exec -it redis-0 /bin/bashkubectl exec[POD][COMMAND]is DEPRECATED and will be removed in a future version. Use kubectl exec[POD]--[COMMAND]instead. rootredis-0:/data# redis-cli -c127.0.0.1:6379cluster nodes de0431085ec24f3445be041d33911d1ada4d7bda 10.244.169.184:637916379 slave c7f02678e65c8a5100dd3a96f82840efff27a3fc 0 1778838395000 3 connected e193aed56f924e576e6788050b7d285e9756e480 10.244.36.110:637916379 slave aac66b559d8f6b738861eff8e5bcdf7ee49789e7 0 1778838396871 1 connected 104beb94505d8a6ef14995f1759dd1db7204f2f0 10.244.169.185:637916379 slave 6232737d86561370f931c3b18a3f1170f7173b08 0 1778838396368 2 connected c7f02678e65c8a5100dd3a96f82840efff27a3fc 10.244.36.108:637916379 myself,master-0 0 3 connected 0-5461aac66b559d8f6b738861eff8e5bcdf7ee49789e7 10.244.169.183:637916379 master-0 1778838395000 1 connected 10923-163836232737d86561370f931c3b18a3f1170f7173b08 10.244.36.109:637916379 master-0 1778838396000 2 connected 5462-10922127.0.0.1:63792.9 创建用于访问Redis的SVC在上面我们创建了用于实现sts的headless service但该svc没有cluster ip因此不能用于外接访问所以我们还需要创建一个svc来为redis集群提供访问和负载均衡rootk8s-master1:~# vim redis-svc.yamlapiVersion:v1kind:Servicemetadata:name:redis-access-servicelabels:app:redisspec:type:NodePortports:-name:redis-portprotocol:TCPport:6379# 集群内访问端口targetPort:6379# 映射Pod的6379端口nodePort:30079# 外部固定访问端口范围30000-32767selector:# 多标签为AND关系精准匹配当前Redis集群Podapp:redisappCluster:redis-clusterrootk8s-master1:~# kubectl apply -f redis-svc.yamlrootk8s-master1:~# kubectl get svc | grep redisredis-access-service NodePort 10.106.169.154 none6379:30079/TCP 3m7s redis-service ClusterIP None none6379/TCP 50m rootk8s-master1:~# kubectl describe endpoints redis-access-serviceName:redis-access-serviceNamespace:defaultLabels:appredisAnnotations:endpoints.kubernetes.io/last-change-trigger-time:2026-05-18T00:16:02ZSubsets:Addresses:10.244.169.129,10.244.169.132,10.244.169.133,10.244.36.116,10.244.36.117,10.244.36.120NotReadyAddresses:nonePorts:Name Port Protocol----------------redis-port 6379 TCPEvents:none## pod内部登录redisK8s Service 域名只在集群内部 Pod 中生效宿主机无法直接解析## 进入业务 Pod 使用服务名 redis-access-service:6379rootk8s-master1:~# kubectl exec -it redis-0 /bin/bashkubectl exec[POD][COMMAND]is DEPRECATED and will be removed in a future version. Use kubectl exec[POD]--[COMMAND]instead. rootredis-0:/data# redis-cli -h redis-access-service -p 6379 # 或者redis-cli -h redis-access-service.default.svc.cluster.local -p 6379redis-access-service:6379ping PONG redis-access-service:6379quit rootredis-0:/data# exitexit rootk8s-master1:~### 宿主机访问redis使用 NodePort 端口 30079rootk8s-master1:~# redis-cli -h 127.0.0.1 -p 30079127.0.0.1:30079ping PONG 127.0.0.1:30079quit## K8S集群外部访问redis使用 NodePort 端口 30079rootNFS:~# redis-cli -h 192.168.13.139 -p 30079192.168.13.139:30079ping PONG 192.168.13.139:30079quit rootNFS:~#至此K8S部署redis-cluster集群完成注文中若有疏漏欢迎大家指正赐教。本文为100%原创转载请务必标注原创作者尊重劳动成果。求赞、求关注、求评论你的支持是我更新的最大动力评论区等你