k8s集群部署Prometheus和Grafana

k8s集群部署Prometheus和Grafana一、部署node-exporter一创建命名空间二部署node-exporter二、部署Prometheus一创建账号并授权二创建configmap存储卷三部署prometheus四创建svc三、部署Grafana一准备镜像二创建Grafana的pod资源三创建Grafana的svc资源四访问Grafana四、部署kube-state-metrics一部署kube-state-metrics一、部署node-exporter一创建命名空间kubectl create ns monitor-sa二部署node-exporter获取镜像这里采用私服的方式进行镜像的pull和push操作。如不会搭建Harbor私服请参考另一篇文章k8s部署EFK日志管理系统dockerpull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/prom/node-exporter:v0.16.0dockertag swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/prom/node-exporter:v0.16.0192.168.224.130/library/node-exporter:v0.16.0dockerlogin-uadmin(Harbor账号)-pHarbor12345(Harbor密码)192.168.224.130(Harbor IP地址)dockerpush192.168.224.130/library/node-exporter:v0.16.0定义yaml文件使用yaml文件以DaemonSet副本控制器在每个节点上部署node-exportervimnode-export.yaml--- apiVersion: apps/v1 kind: DaemonSet#可以保证 k8s 集群的每个节点都运行完全一样的 podmetadata: name: node-exporter namespace: monitor-sa labels: name: node-exporter spec: selector: matchLabels: name: node-exporter template: metadata: labels: name: node-exporter spec: hostPID:true#允许Pod使用宿主机的 PIDhostIPC:true#允许Pod使用宿主机的IPChostNetwork:true#允许Pod使用宿主机的网络命名空间containers: - name: node-exporter image:192.168.224.130/library/node-exporter:v0.16.0#指定镜像这里需要修改成自己私服地址ports: - containerPort:9100#指定容器暴露端口resources: requests: cpu:0.15#这个容器运行至少需要0.15核cpusecurityContext: privileged:true#开启特权模式args:#传递给容器的命令行参数---path.procfs- /host/proc ---path.sysfs- /host/sys - --collector.filesystem.ignored-mount-points -^/(sys|proc|dev|host|etc)($|/)volumeMounts:#定义了容器内挂载卷的路径- name: dev mountPath: /host/dev - name: proc mountPath: /host/proc - name: sys mountPath: /host/sys - name: rootfs mountPath: /rootfs tolerations: - key:node-role.kubernetes.io/control-planeoperator:Existseffect:NoSchedulevolumes:#定义了 Pod 中要使用的卷- name: proc hostPath: path: /proc - name: dev hostPath: path: /dev - name: sys hostPath: path: /sys - name: rootfs hostPath: path: /镜像拉取需要改成自己搭建的私服地址创建服务kubectl apply-fnode-export.yamlkubectl get pod-nmonitor-sa-owide每个节点都运行一个node-exporter二、部署Prometheus一创建账号并授权创建一个管理账号并绑定admin用户admin用户拥有所有资源的所有权限kubectl create serviceaccount monitor-nmonitor-sa kubectl create clusterrolebinding monitor-clusterrolebinding-nmonitor-sa--clusterrolecluster-admin--serviceaccountmonitor-sa:monitor二创建configmap存储卷vimprometheus-cfg.yamlapiVersion: v1 kind: ConfigMap metadata: labels: app: prometheus name: prometheus-config namespace: monitor-sa data: prometheus.yml:|global:#指定prometheus的全局配置比如采集间隔抓取超时时间等scrape_interval: 15s#采集目标主机监控数据的时间间隔默认为1mscrape_timeout: 10s#数据采集超时时间默认10sevaluation_interval: 1m#触发告警生成alert的时间间隔默认是1mscrape_configs:#配置数据源称为target每个target用job_name命名。- job_name:kubernetes-nodekubernetes_sd_configs:#指定的是k8s的服务发现- role:node#使用node角色它使用默认的kubelet提供的http端口来发现集群中每个node节点relabel_configs:#重新标记修改目标地址- source_labels:[__address__]#配置的原始标签匹配地址regex:(.*):10250#匹配带有10250端口的urlreplacement:${1}:9100#把匹配到的ip:10250的ip保留类似与sed的后向引用target_label: __address__#新生成的url是${1}获取到的ip:9100action: replace - action: labelmap regex: __meta_kubernetes_node_label_(.)- job_name:kubernetes-node-cadvisorkubernetes_sd_configs: - role:nodescheme: https#定义了与cAdvisor服务通信时使用HTTPS协议tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt#CA证书文件的路径bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - action: labelmap#把匹配到的标签保留regex: __meta_kubernetes_node_label_(.)#保留匹配到的具有__meta_kubernetes_node_label的标签- target_label: __address__ replacement: kubernetes.default.svc:443 - source_labels:[__meta_kubernetes_node_name]regex:(.)#把原始标签中__meta_kubernetes_node_name值匹配到target_label: __metrics_path__#获取__metrics_path__对应的值replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor - job_name:kubernetes-apiserverkubernetes_sd_configs: - role: endpoints scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels:[__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]action: keep regex: default;kubernetes;https - job_name:kubernetes-service-endpointskubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels:[__meta_kubernetes_service_annotation_prometheus_io_scrape]action: keep regex:true- source_labels:[__meta_kubernetes_service_annotation_prometheus_io_scheme]action: replace target_label: __scheme__ regex:(https?)- source_labels:[__meta_kubernetes_service_annotation_prometheus_io_path]action: replace target_label: __metrics_path__ regex:(.)- source_labels:[__address__, __meta_kubernetes_service_annotation_prometheus_io_port]action: replace target_label: __address__ regex:([^:])(?::\d)?;(\d)replacement:$1:$2- action: labelmap regex: __meta_kubernetes_service_label_(.)- source_labels:[__meta_kubernetes_namespace]action: replace target_label: kubernetes_namespace - source_labels:[__meta_kubernetes_service_name]action: replace target_label: kubernetes_name创建服务kubectl apply-fprometheus-cfg.yamlkubectl get configmaps-nmonitor-sa三部署prometheus拉取镜像这里采用私服的方式进行镜像的pull和push操作。如不会搭建Harbor私服请参考另一篇文章k8s部署EFK日志管理系统dockerpull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/prom/prometheus:v2.52.0dockertag swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/prom/prometheus:v2.52.0192.168.224.130/library/prometheus:v2.52.0dockerlogin-uadmin(Harbor账号)-pHarbor12345(Harbor密码)192.168.224.130(Harbor IP地址)dockerpush192.168.224.130/library/prometheus:v2.52.0定义yaml文件vimprometheus-deploy.yamlapiVersion: apps/v1 kind: Deployment metadata: name: prometheus-server#部署的名称namespace: monitor-sa#指定命名空间labels: app: prometheus spec: replicas:1selector: matchLabels: app: prometheus component: server template: metadata: labels: app: prometheus component: server annotations: prometheus.io/scrape:false#Pod的注解表示Prometheus不从此端点抓取数据spec: nodeName: node01#指定Pod应调度到的节点名称serviceAccountName: monitor#指定Pod使用的ServiceAccount的名称containers: - name: prometheus image:192.168.224.130/library/prometheus:v2.52.0#要使用的容器镜像和标签imagePullPolicy: IfNotPresent command: - prometheus#启动命令---config.file/etc/prometheus/prometheus.yml#指定加载的配置文件---storage.tsdb.path/prometheus#数据存储目录---storage.tsdb.retention720h#数据保存时长- --web.enable-lifecycle#开启热加载ports: - containerPort:9090protocol: TCP volumeMounts: - mountPath: /etc/prometheus/prometheus.yml name: prometheus-config subPath: prometheus.yml - mountPath: /prometheus/ name: prometheus-storage-volume volumes: - name: prometheus-config configMap:#指定挂载的configMap资源name: prometheus-config items: - key: prometheus.yml path: prometheus.yml mode: 0644 - name: prometheus-storage-volume hostPath: path: /data type: Directory镜像需要改成自己的私服地址。在node1节点创建挂载目录[rootnode1 ~]#mkdir /data chmod 777 /data创建服务kubectl apply-fprometheus-deploy.yamlkubectl get pod-owide-nmonitor-sa四创建svcvimprometheus-svc.yamlapiVersion: v1 kind: Service metadata: name: prometheus namespace: monitor-sa labels: app: prometheus spec: type: NodePort ports: - port:9090targetPort:9090protocol: TCP nodePort:31000selector: app: prometheus component: serverkubectl apply-fprometheus-svc.yamlkubectl get svc-nmonitor-sa使用web浏览器访问节点的31000端口点击Targets查看加入监控的服务。三、部署Grafana一准备镜像这里采用私服的方式进行镜像的pull和push操作。如不会搭建Harbor私服请参考另一篇文章k8s部署EFK日志管理系统dockerpull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/grafana/grafana:7.3.7dockertag swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/grafana/grafana:7.3.7192.168.224.130/library/grafana:7.3.7dockerlogin-uadmin-pHarbor12345192.168.224.130dockerpush192.168.224.130/library/grafana:7.3.7二创建Grafana的pod资源vimgrafana.yamlapiVersion: apps/v1 kind: Deployment metadata: name: monitoring-grafana namespace: kube-system spec: replicas:1selector: matchLabels: task: monitoring k8s-app: grafana template: metadata: labels: task: monitoring k8s-app: grafana spec: containers: - name: grafana image: grafana/grafana:5.0.4 ports: - containerPort:3000protocol: TCP volumeMounts:#定义了如何挂载卷到容器中- mountPath: /etc/ssl/certs name: ca-certificates readOnly:true- mountPath: /var/lib/grafana/ name: grafana-storage env:#定义容器内的环境变量- name: INFLUXDB_HOST value: monitoring-influxdb#指定了Grafana连接到的InfluxDB数据库的主机名或IP地址- name: GF_SERVER_HTTP_PORT value:3000#指定了Grafana HTTP服务器监听的端口- name: GF_AUTH_BASIC_ENABLED value:false#用于启用或禁用Grafana的基本认证false表示禁用- name: GF_AUTH_ANONYMOUS_ENABLED value:true#用于启用或禁用Grafana的匿名访问true表示启用- name: GF_AUTH_ANONYMOUS_ORG_ROLE value: Admin#当匿名访问被启用时这个环境变量定义了匿名用户所拥有的组织角色Admin表示具有管理员权限- name: GF_SERVER_ROOT_URL value: /#定义了Grafana服务器的根URLvolumes:#定义了要挂载到Pod中的卷- name: ca-certificates hostPath: path: /etc/ssl/certs - name: grafana-storage emptyDir:{}创建资源kubectl apply-fgrafana.yaml三创建Grafana的svc资源vimgrafana-service.yamlapiVersion: v1 kind: Service metadata: labels: kubernetes.io/cluster-service:truekubernetes.io/name: monitoring-grafana name: monitoring-grafana namespace: kube-system spec: ports: - port:80targetPort:3000selector: k8s-app: grafana type: NodePort[rootmaster01 prometheus]#service/monitoring-grafana createdkubectl apply-fgrafana-service.yamlkubectl get svc monitoring-grafana-nkube-system四访问Grafana登录grafana浏览器访问http://192.168.224.130:31860登陆 grafana默认账号admin密码admin配置grafana配置数据源Prometheus 。【Name】设置成 Prometheus【Type】选择 Prometheus 【URL】设置成 http://10.102.14.76:9090#使用service的集群内部端口配置服务端地址,kubectl get svc -n monitor-sa查看点击 【Save Test】导入监控模板官方链接https://grafana.com/dashboards?dataSourceprometheussearchkubernetes选择适合的面板点击 Copy ID 或者 Download JSON3.1 导入监控模板监控状态点击左侧号选择【Import】点击【Upload .json File】导入下载的模板文件【Prometheus】选择 Prometheus点击【Import】可以在监控模板中挑选适合的进行导入。四、部署kube-state-metricskube-state-metrics是一个Kubernetes集群中的重要组件它的主要作用是收集和暴露Kubernetes资源的状态指标以便于监控和分析一部署kube-state-metrics创建sa账号并授权创建serviceaccount用户账号并授予指定组的权限使其对k8s资源有管理的权限能够收集到资源信息vimkube-state-metrics-rbac.yamlapiVersion: v1 kind: ServiceAccount metadata: name: kube-state-metrics namespace: kube-system ---#创建集群角色并指定角色权限apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: kube-state-metrics rules: - apiGroups:[]resources:[nodes,pods,services,resourcequotas,replicationcontrollers,limitranges,persistentvolumeclaims,persistentvolumes,namespaces,endpoints]verbs:[list,watch]- apiGroups:[extensions]resources:[daemonsets,deployments,replicasets]verbs:[list,watch]- apiGroups:[apps]resources:[statefulsets]verbs:[list,watch]- apiGroups:[batch]resources:[cronjobs,jobs]verbs:[list,watch]- apiGroups:[autoscaling]resources:[horizontalpodautoscalers]verbs:[list,watch]---#对用户与集群角色进行绑定使创建的用户有一定的权限去管理并获取资源信息apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kube-state-metrics roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kube-state-metrics subjects: - kind: ServiceAccount name: kube-state-metrics namespace: kube-system创建并查看信息kubectl apply-fkube-state-metrics-rbac.yamlkubectl get sa kube-state-metrics-nkube-systemkubectl get clusterrolebindings kube-state-metrics-nkube-system安装kube-state-metrics组件拉取镜像这里采用私服的方式进行镜像的pull和push操作。如不会搭建Harbor私服请参考另一篇文章k8s部署EFK日志管理系统dockerpull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/quay.io/coreos/kube-state-metrics:v1.9.7dockertag swr.cn-north-4.myhuaweicloud.com/ddn-k8s/quay.io/coreos/kube-state-metrics:v1.9.7192.168.224.130/library/kube-state-metrics:v1.9.7dockerlogin-uadmin-pHarbor12345192.168.224.130dockerpush192.168.224.130/library/kube-state-metrics:v1.9.7创建服务vimkube-state-metrics-deploy.yamlapiVersion: apps/v1 kind: Deployment metadata: name: kube-state-metrics namespace: kube-system spec: replicas:1selector: matchLabels: app: kube-state-metrics template: metadata: labels: app: kube-state-metrics spec: serviceAccountName: kube-state-metrics containers: - name: kube-state-metrics image: quay.io/coreos/kube-state-metrics:v1.9.0 ports: - containerPort:8080kubectl apply-fkube-state-metrics-deploy.yaml创建service资源vimkube-state-metrics-svc.yamlapiVersion: v1 kind: Service metadata: annotations: prometheus.io/scrape:true#注解信息表示从此端点抓取数据name: kube-state-metrics namespace: kube-system labels: app: kube-state-metrics spec: ports: - name: kube-state-metrics port:8080protocol: TCP selector: app: kube-state-metricskubectl apply-fkube-state-metrics-svc.yamlkubectl get svc kube-state-metrics-owide-nkube-system