K8s实战指南：从集群部署到应用编排的完整流程解析

1. Kubernetes集群部署实战第一次接触Kubernetes时我被它复杂的组件和配置吓到了。但经过多次实践后发现只要掌握正确的方法从零搭建生产级K8s集群并没有想象中困难。下面分享我用Ansible自动化部署集群的完整过程帮你避开我踩过的那些坑。1.1 环境准备与Ansible配置在开始部署前我们需要准备至少3台Linux主机2C4G配置起步。我习惯用CentOS 7.9作为基础系统因为它对Kubernetes的兼容性较好。假设我们已经有三台主机master10.0.0.1作为控制节点node110.0.0.2和node210.0.0.3作为工作节点首先在所有节点上安装Ansibleyum install -y epel-release yum install -y ansible创建Ansible工作目录并配置主机清单mkdir /k8s-deploy cd /k8s-deploy cat hosts EOF [k8s_master] master ansible_host10.0.0.1 [k8s_nodes] node1 ansible_host10.0.0.2 node2 ansible_host10.0.0.3 [k8s:children] k8s_master k8s_nodes EOF1.2 系统基础配置通过Ansible统一配置所有节点的系统参数非常重要。创建pre-install.yml文件--- - name: 基础系统配置 hosts: k8s tasks: - name: 关闭SELinux selinux: state: disabled - name: 关闭Swap shell: | swapoff -a sed -i / swap / s/^\(.*\)$/#\1/g /etc/fstab - name: 加载内核模块 modprobe: name: {{ item }} state: present loop: - br_netfilter - ip_vs - ip_vs_rr - ip_vs_sh - nf_conntrack - name: 配置内核参数 sysctl: name: {{ item.name }} value: {{ item.value }} state: present reload: yes loop: - { name: net.bridge.bridge-nf-call-ip6tables, value: 1 } - { name: net.bridge.bridge-nf-call-iptables, value: 1 } - { name: net.ipv4.ip_forward, value: 1 }执行配置ansible-playbook -i hosts pre-install.yml1.3 容器运行时安装Kubernetes 1.24版本默认不再包含Docker支持我们可以选择containerd作为运行时。创建containerd-install.yml--- - name: 安装containerd hosts: k8s tasks: - name: 安装依赖 yum: name: - yum-utils - device-mapper-persistent-data - lvm2 state: present - name: 添加Docker仓库 get_url: url: https://download.docker.com/linux/centos/docker-ce.repo dest: /etc/yum.repos.d/docker-ce.repo - name: 安装containerd yum: name: containerd.io state: present - name: 配置containerd copy: src: files/containerd-config.toml dest: /etc/containerd/config.toml - name: 启动containerd systemd: name: containerd state: started enabled: yescontainerd配置文件示例files/containerd-config.tomlversion 2 [plugins.io.containerd.grpc.v1.cri.registry.mirrors.docker.io] endpoint [https://registry-1.docker.io] [plugins.io.containerd.grpc.v1.cri.containerd] snapshotter overlayfs2. Kubernetes核心组件部署2.1 使用kubeadm初始化集群在master节点执行初始化命令前需要先安装kubeadm、kubelet和kubectlcat EOF /etc/yum.repos.d/kubernetes.repo [kubernetes] nameKubernetes baseurlhttps://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled1 gpgcheck1 repo_gpgcheck1 gpgkeyhttps://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF yum install -y kubelet-1.25.0 kubeadm-1.25.0 kubectl-1.25.0 systemctl enable --now kubelet初始化master节点注意替换实际IPkubeadm init \ --apiserver-advertise-address10.0.0.1 \ --image-repository registry.aliyuncs.com/google_containers \ --kubernetes-version v1.25.0 \ --service-cidr10.96.0.0/12 \ --pod-network-cidr10.244.0.0/16初始化成功后按照提示配置kubectlmkdir -p $HOME/.kube cp -i /etc/kubernetes/admin.conf $HOME/.kube/config chown $(id -u):$(id -g) $HOME/.kube/config2.2 网络插件安装我推荐使用Calico作为网络插件它对网络策略的支持更好kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml验证节点状态kubectl get nodes # 应该看到master节点状态为Ready2.3 工作节点加入集群在master节点上获取加入命令kubeadm token create --print-join-command然后在每个工作节点上执行输出的join命令格式类似kubeadm join 10.0.0.1:6443 --token xxxx.xxxxxxxxxxxx \ --discovery-token-ca-cert-hash sha256:xxxxxxxx...3. 应用部署与配置管理3.1 部署第一个应用让我们从简单的Nginx开始创建nginx-deployment.yamlapiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.23 ports: - containerPort: 80应用配置kubectl apply -f nginx-deployment.yaml查看部署状态kubectl get pods -l appnginx3.2 使用Service暴露应用创建service.yamlapiVersion: v1 kind: Service metadata: name: nginx-service spec: selector: app: nginx ports: - protocol: TCP port: 80 targetPort: 80 type: NodePort应用Service配置后可以通过节点IP和随机端口访问Nginxkubectl get svc nginx-service3.3 使用ConfigMap管理配置将Nginx配置存储在ConfigMap中apiVersion: v1 kind: ConfigMap metadata: name: nginx-config data: nginx.conf: | user nginx; worker_processes auto; error_log /var/log/nginx/error.log notice; events { worker_connections 1024; } http { include /etc/nginx/mime.types; default_type application/octet-stream; sendfile on; keepalive_timeout 65; server { listen 80; server_name localhost; location / { root /usr/share/nginx/html; index index.html index.htm; } } }然后在Deployment中挂载这个ConfigMapspec: containers: - name: nginx volumeMounts: - name: config-volume mountPath: /etc/nginx/nginx.conf subPath: nginx.conf volumes: - name: config-volume configMap: name: nginx-config4. 高级部署策略4.1 使用Secret管理敏感信息创建数据库密码Secretkubectl create secret generic db-secret \ --from-literalusernameadmin \ --from-literalpasswordS3cret!123在Pod中使用Secretenv: - name: DB_USERNAME valueFrom: secretKeyRef: name: db-secret key: username - name: DB_PASSWORD valueFrom: secretKeyRef: name: db-secret key: password4.2 部署有状态应用MySQL有状态应用需要PersistentVolume先创建storage-class.yamlapiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: local-storage provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer然后创建mysql-statefulset.yamlapiVersion: apps/v1 kind: StatefulSet metadata: name: mysql spec: serviceName: mysql replicas: 1 selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - name: mysql image: mysql:5.7 env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: db-secret key: password ports: - containerPort: 3306 volumeMounts: - name: mysql-persistent-storage mountPath: /var/lib/mysql volumeClaimTemplates: - metadata: name: mysql-persistent-storage spec: accessModes: [ ReadWriteOnce ] storageClassName: local-storage resources: requests: storage: 10Gi4.3 使用Helm部署复杂应用Helm是Kubernetes的包管理工具非常适合部署复杂应用。以Redis集群为例# 添加Bitnami仓库 helm repo add bitnami https://charts.bitnami.com/bitnami # 安装Redis集群 helm install my-redis bitnami/redis \ --set cluster.enabledtrue \ --set cluster.slaveCount2 \ --set passwordredis-password5. 集群运维与监控5.1 节点维护与隔离当需要对节点进行维护时可以先隔离节点kubectl drain node1 --ignore-daemonsets维护完成后重新加入集群kubectl uncordon node15.2 部署监控系统使用Prometheus-Operator监控集群helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm install prometheus prometheus-community/kube-prometheus-stack访问Grafanakubectl port-forward svc/prometheus-grafana 3000:805.3 日志收集方案部署EFKElasticsearchFluentdKibana日志系统helm install efk elastic/elasticsearch helm install fluent-bit fluent/fluent-bit helm install kibana elastic/kibana6. 生产环境最佳实践6.1 资源限制与配额为命名空间设置资源配额apiVersion: v1 kind: ResourceQuota metadata: name: compute-resources spec: hard: requests.cpu: 4 requests.memory: 8Gi limits.cpu: 8 limits.memory: 16Gi pods: 106.2 Pod安全策略创建Pod安全策略apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: restricted spec: privileged: false allowPrivilegeEscalation: false requiredDropCapabilities: - ALL volumes: - configMap - emptyDir - secret hostNetwork: false hostIPC: false hostPID: false runAsUser: rule: MustRunAsNonRoot seLinux: rule: RunAsAny supplementalGroups: rule: MustRunAs ranges: - min: 1 max: 65535 fsGroup: rule: MustRunAs ranges: - min: 1 max: 655356.3 备份与恢复使用Velero进行集群备份velero install \ --provider aws \ --bucket my-k8s-backups \ --secret-file ./credentials-velero \ --use-volume-snapshotsfalse \ --backup-location-config regionminio,s3ForcePathStyletrue,s3Urlhttp://minio.example.com创建定时备份velero schedule create daily-backup \ --scheduleevery 24h \ --include-namespacesproduction