一、为什么选择 Prometheus在传统架构中Zabbix、Nagios 等监控工具是主流但在 Kubernetes 环境中这些工具难以应对动态、短生命周期、大规模容器的监控需求。Prometheus 应运而生成为 CNCF 毕业项目之一专为云原生设计。核心优势多维数据模型支持标签labels实现灵活的数据查询与聚合。PromQL强大的查询语言支持实时分析和告警。无需依赖外部存储内置时序数据库。服务发现机制自动发现 K8s 中的 Pod、Service、Node。Pull 模型主动拉取 Metrics便于控制数据采集节奏。二、Prometheus 在 K8s 中的架构流程textNode Exporter (节点级) cAdvisor (容器级) → Prometheus (采集存储) → Grafana (可视化) → AlertManager (告警)组件说明组件功能Node Exporter采集节点 CPU、内存、磁盘、网络等指标cAdvisor集成在 Kubelet 中采集容器资源使用情况Prometheus拉取 Metrics存储时序数据支持 PromQLGrafana可视化仪表盘AlertManager告警路由、分组、静默、抑制三、环境准备克隆 Prometheus Operator 项目bashgit clone -b release-0.10 https://github.com/prometheus-operator/kube-prometheus.git cd kube-prometheus四、安装 Prometheus Operator核心bashkubectl apply --server-side -f manifests/setup--server-side避免 kubectl 的冲突问题直接由 API Server 处理声明式配置。删除 Operatorbashkubectl delete --ignore-not-foundtrue -f manifests/setup五、安装 Prometheus Stack全家桶bashkubectl apply --server-side -f manifests/包含Prometheus、AlertManager、Grafana、ServiceMonitors、告警规则等。删除 Stackbashkubectl delete --ignore-not-foundtrue -f manifests/ -f manifests/setup六、查看组件状态bashkubectl get pod -n monitoring七、暴露 Grafana 服务NodePort编辑 Servicebashkubectl edit svc grafana -n monitoring修改type: NodePort并指定nodePort: 32082yamlports: - name: http nodePort: 32082 port: 3000 protocol: TCP targetPort: http type: NodePort访问地址http://任意节点IP:32082默认账号密码admin/admin八、暴露 Prometheus 服务NodePortbashkubectl edit svc prometheus-k8s -n monitoring修改type: NodePortnodePort: 32370访问地址http://任意节点IP:32370九、配置 Grafana 数据源若未自动添加手动配置URLhttp://prometheus-k8s:9090保存并测试十、导入监控 Dashboard推荐模板模板 ID说明13105Kubernetes 中文版集群监控7249Kubernetes Cluster893Docker 和系统监控14731Node Exporter 主机监控导入方式Grafana →→ Import → 输入 ID → Load → Import十一、监控 MySQLExporter 实战1. 部署 MySQLbashkubectl create deploy mysql --imagemysql:5.7.23 kubectl set env deploy/mysql MYSQL_ROOT_PASSWORDpwd1232. 暴露 Servicebashkubectl expose deployment mysql --type NodePort --port33063. 创建 MySQL Exporteryaml# mysql-exporter.yaml apiVersion: apps/v1 kind: Deployment metadata: name: mysql-exporter namespace: monitoring spec: replicas: 1 selector: matchLabels: k8s-app: mysql-exporter template: metadata: labels: k8s-app: mysql-exporter spec: containers: - name: mysql-exporter image: registry.cn-beijing.aliyuncs.com/dotbalo/mysql-exporter env: - name: DATA_SOURCE_NAME value: exporter:exporter(mysql.default:3306)/ ports: - containerPort: 9104 --- apiVersion: v1 kind: Service metadata: name: mysql-exporter namespace: monitoring labels: k8s-app: mysql-exporter spec: selector: k8s-app: mysql-exporter ports: - name: api port: 9104bashkubectl apply -f mysql-exporter.yaml4. 创建 ServiceMonitoryaml# mysql-sm.yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: mysql-exporter namespace: monitoring labels: k8s-app: mysql-exporter spec: jobLabel: k8s-app endpoints: - port: api interval: 30s selector: matchLabels: k8s-app: mysql-exporterbashkubectl apply -f mysql-sm.yaml5. 导入 MySQL Dashboard模板 ID6239十二、钉钉告警集成企业级必备1. 添加钉钉机器人群设置 → 智能群助手 → 添加机器人 → 自定义关键词FIRING2. 部署 webhook-dingtalkbashwget https://github.com/timonwong/prometheus-webhook-dingtalk/releases/download/v2.0.0/prometheus-webhook-dingtalk-2.0.0.linux-amd64.tar.gz tar xf prometheus-webhook-dingtalk-2.0.0.linux-amd64.tar.gz mv prometheus-webhook-dingtalk-2.0.0.linux-amd64 /usr/local/dingtalk3. 配置 config.ymlyamltargets: webhook2: url: https://oapi.dingtalk.com/robot/send?access_token你的token secret: 你的secret4. 启动服务bashcat /etc/systemd/system/prometheus-webhook-dingtalk.service EOF [Unit] DescriptionPrometheus Webhook DingTalk Afternetwork.target [Service] ExecStart/usr/local/dingtalk/prometheus-webhook-dingtalk WorkingDirectory/usr/local/dingtalk Restartalways [Install] WantedBymulti-user.target EOF systemctl start prometheus-webhook-dingtalk5. 修改 AlertManager 配置编辑alertmanager-secret.yaml添加yamlreceivers: - name: webhook webhook_configs: - url: http://dingtalk节点IP:8060/dingtalk/webhook2/send send_resolved: truebashkubectl replace -f alertmanager-secret.yaml6. 暴露 AlertManagerbashkubectl edit svc alertmanager-main -n monitoring修改type: NodePort查看端口bashkubectl get svc -n monitoring alertmanager-main访问http://任意节点IP:30586十三、补充知识点博客亮点✅ 1. Prometheus 告警状态机状态说明Inactive未触发Pending已触发等待 for 时间Firing触发并发送到 AlertManager✅ 2. Prometheus vs Prometheus Operator方式优点缺点手动部署灵活可控配置复杂需手动维护Operator自动化管理K8s 原生黑盒化调试困难✅ 3. ServiceMonitor 核心字段selector.matchLabels匹配 Exporter Service 的标签endpoints.portExporter 的端口名称interval采集间隔✅ 4. Exporter 设计模式适用于无法直接暴露 Metrics 的应用MySQL、Redis、Kafka、Nginx 等通过 Sidecar 或独立 Pod 采集并暴露。✅ 5. Prometheus 高可用方案双实例 数据分片Thanos 或 VictoriaMetrics 实现长期存储与全局视图AlertManager 集群化避免告警重复十四、常见问题排查问题解决方法Target 状态为 DOWN检查 Service 端口、Endpoint、网络策略Grafana 无数据确认 Prometheus 数据源 URL 是否正确钉钉未收到告警检查 AlertManager 配置、机器人关键词、网络连通性Prometheus 内存高减少采集频率、优化 PromQL、限制标签基数
云原生环境 Prometheus 企业级监控实战指南
发布时间：2026/6/5 22:15:14
一、为什么选择 Prometheus在传统架构中Zabbix、Nagios 等监控工具是主流但在 Kubernetes 环境中这些工具难以应对动态、短生命周期、大规模容器的监控需求。Prometheus 应运而生成为 CNCF 毕业项目之一专为云原生设计。核心优势多维数据模型支持标签labels实现灵活的数据查询与聚合。PromQL强大的查询语言支持实时分析和告警。无需依赖外部存储内置时序数据库。服务发现机制自动发现 K8s 中的 Pod、Service、Node。Pull 模型主动拉取 Metrics便于控制数据采集节奏。二、Prometheus 在 K8s 中的架构流程textNode Exporter (节点级) cAdvisor (容器级) → Prometheus (采集存储) → Grafana (可视化) → AlertManager (告警)组件说明组件功能Node Exporter采集节点 CPU、内存、磁盘、网络等指标cAdvisor集成在 Kubelet 中采集容器资源使用情况Prometheus拉取 Metrics存储时序数据支持 PromQLGrafana可视化仪表盘AlertManager告警路由、分组、静默、抑制三、环境准备克隆 Prometheus Operator 项目bashgit clone -b release-0.10 https://github.com/prometheus-operator/kube-prometheus.git cd kube-prometheus四、安装 Prometheus Operator核心bashkubectl apply --server-side -f manifests/setup--server-side避免 kubectl 的冲突问题直接由 API Server 处理声明式配置。删除 Operatorbashkubectl delete --ignore-not-foundtrue -f manifests/setup五、安装 Prometheus Stack全家桶bashkubectl apply --server-side -f manifests/包含Prometheus、AlertManager、Grafana、ServiceMonitors、告警规则等。删除 Stackbashkubectl delete --ignore-not-foundtrue -f manifests/ -f manifests/setup六、查看组件状态bashkubectl get pod -n monitoring七、暴露 Grafana 服务NodePort编辑 Servicebashkubectl edit svc grafana -n monitoring修改type: NodePort并指定nodePort: 32082yamlports: - name: http nodePort: 32082 port: 3000 protocol: TCP targetPort: http type: NodePort访问地址http://任意节点IP:32082默认账号密码admin/admin八、暴露 Prometheus 服务NodePortbashkubectl edit svc prometheus-k8s -n monitoring修改type: NodePortnodePort: 32370访问地址http://任意节点IP:32370九、配置 Grafana 数据源若未自动添加手动配置URLhttp://prometheus-k8s:9090保存并测试十、导入监控 Dashboard推荐模板模板 ID说明13105Kubernetes 中文版集群监控7249Kubernetes Cluster893Docker 和系统监控14731Node Exporter 主机监控导入方式Grafana →→ Import → 输入 ID → Load → Import十一、监控 MySQLExporter 实战1. 部署 MySQLbashkubectl create deploy mysql --imagemysql:5.7.23 kubectl set env deploy/mysql MYSQL_ROOT_PASSWORDpwd1232. 暴露 Servicebashkubectl expose deployment mysql --type NodePort --port33063. 创建 MySQL Exporteryaml# mysql-exporter.yaml apiVersion: apps/v1 kind: Deployment metadata: name: mysql-exporter namespace: monitoring spec: replicas: 1 selector: matchLabels: k8s-app: mysql-exporter template: metadata: labels: k8s-app: mysql-exporter spec: containers: - name: mysql-exporter image: registry.cn-beijing.aliyuncs.com/dotbalo/mysql-exporter env: - name: DATA_SOURCE_NAME value: exporter:exporter(mysql.default:3306)/ ports: - containerPort: 9104 --- apiVersion: v1 kind: Service metadata: name: mysql-exporter namespace: monitoring labels: k8s-app: mysql-exporter spec: selector: k8s-app: mysql-exporter ports: - name: api port: 9104bashkubectl apply -f mysql-exporter.yaml4. 创建 ServiceMonitoryaml# mysql-sm.yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: mysql-exporter namespace: monitoring labels: k8s-app: mysql-exporter spec: jobLabel: k8s-app endpoints: - port: api interval: 30s selector: matchLabels: k8s-app: mysql-exporterbashkubectl apply -f mysql-sm.yaml5. 导入 MySQL Dashboard模板 ID6239十二、钉钉告警集成企业级必备1. 添加钉钉机器人群设置 → 智能群助手 → 添加机器人 → 自定义关键词FIRING2. 部署 webhook-dingtalkbashwget https://github.com/timonwong/prometheus-webhook-dingtalk/releases/download/v2.0.0/prometheus-webhook-dingtalk-2.0.0.linux-amd64.tar.gz tar xf prometheus-webhook-dingtalk-2.0.0.linux-amd64.tar.gz mv prometheus-webhook-dingtalk-2.0.0.linux-amd64 /usr/local/dingtalk3. 配置 config.ymlyamltargets: webhook2: url: https://oapi.dingtalk.com/robot/send?access_token你的token secret: 你的secret4. 启动服务bashcat /etc/systemd/system/prometheus-webhook-dingtalk.service EOF [Unit] DescriptionPrometheus Webhook DingTalk Afternetwork.target [Service] ExecStart/usr/local/dingtalk/prometheus-webhook-dingtalk WorkingDirectory/usr/local/dingtalk Restartalways [Install] WantedBymulti-user.target EOF systemctl start prometheus-webhook-dingtalk5. 修改 AlertManager 配置编辑alertmanager-secret.yaml添加yamlreceivers: - name: webhook webhook_configs: - url: http://dingtalk节点IP:8060/dingtalk/webhook2/send send_resolved: truebashkubectl replace -f alertmanager-secret.yaml6. 暴露 AlertManagerbashkubectl edit svc alertmanager-main -n monitoring修改type: NodePort查看端口bashkubectl get svc -n monitoring alertmanager-main访问http://任意节点IP:30586十三、补充知识点博客亮点✅ 1. Prometheus 告警状态机状态说明Inactive未触发Pending已触发等待 for 时间Firing触发并发送到 AlertManager✅ 2. Prometheus vs Prometheus Operator方式优点缺点手动部署灵活可控配置复杂需手动维护Operator自动化管理K8s 原生黑盒化调试困难✅ 3. ServiceMonitor 核心字段selector.matchLabels匹配 Exporter Service 的标签endpoints.portExporter 的端口名称interval采集间隔✅ 4. Exporter 设计模式适用于无法直接暴露 Metrics 的应用MySQL、Redis、Kafka、Nginx 等通过 Sidecar 或独立 Pod 采集并暴露。✅ 5. Prometheus 高可用方案双实例 数据分片Thanos 或 VictoriaMetrics 实现长期存储与全局视图AlertManager 集群化避免告警重复十四、常见问题排查问题解决方法Target 状态为 DOWN检查 Service 端口、Endpoint、网络策略Grafana 无数据确认 Prometheus 数据源 URL 是否正确钉钉未收到告警检查 AlertManager 配置、机器人关键词、网络连通性Prometheus 内存高减少采集频率、优化 PromQL、限制标签基数
