Kubernetes 日志管理最佳实践

Kubernetes 日志管理最佳实践一、前言哥们别整那些花里胡哨的。日志管理是 Kubernetes 运维的重要组成部分今天直接上硬货教你如何构建完善的日志管理系统。二、日志方案对比方案适用场景优势劣势原生日志简单场景配置简单存储有限ELK Stack大规模日志功能丰富资源消耗大Loki轻量级日志性能优异功能有限Fluent Bit高性能资源占用小配置复杂三、实战配置1. Fluent Bit 配置apiVersion: apps/v1 kind: DaemonSet metadata: name: fluent-bit namespace: logging spec: selector: matchLabels: app: fluent-bit template: metadata: labels: app: fluent-bit spec: containers: - name: fluent-bit image: fluent/fluent-bit:latest ports: - containerPort: 2020 volumeMounts: - name: config mountPath: /fluent-bit/etc/ - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true volumes: - name: config configMap: name: fluent-bit-config - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers2. Fluent Bit 配置文件apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit-config namespace: logging data: fluent-bit.conf: | [SERVICE] Flush 1 Log_Level info Daemon off Parsers_File parsers.conf [INPUT] Name tail Path /var/log/containers/*.log Parser docker Tag kube.* Refresh_Interval 5 Rotate_Wait 30 Mem_Buf_Limit 5MB Skip_Long_Lines On [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc:443 Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token Kube_Tag_Prefix kube.var.log.containers. Merge_Log On Keep_Log Off K8S-Logging.Parser On K8S-Logging.Exclude On [OUTPUT] Name es Match * Host elasticsearch Port 9200 Index fluent-bit Type flb_type Logstash_Format On Retry_Limit False3. Loki 配置apiVersion: apps/v1 kind: Deployment metadata: name: loki namespace: logging spec: replicas: 1 selector: matchLabels: app: loki template: metadata: labels: app: loki spec: containers: - name: loki image: grafana/loki:latest ports: - containerPort: 3100 volumeMounts: - name: config mountPath: /etc/loki/ - name: storage mountPath: /loki volumes: - name: config configMap: name: loki-config - name: storage persistentVolumeClaim: claimName: loki-storage --- apiVersion: v1 kind: Service metadata: name: loki namespace: logging spec: selector: app: loki ports: - port: 3100 targetPort: 31004. Grafana 配置apiVersion: apps/v1 kind: Deployment metadata: name: grafana namespace: logging spec: replicas: 1 selector: matchLabels: app: grafana template: metadata: labels: app: grafana spec: containers: - name: grafana image: grafana/grafana:latest ports: - containerPort: 3000 env: - name: GF_SECURITY_ADMIN_PASSWORD valueFrom: secretKeyRef: name: grafana-secret key: password volumeMounts: - name: storage mountPath: /var/lib/grafana volumes: - name: storage persistentVolumeClaim: claimName: grafana-storage --- apiVersion: v1 kind: Service metadata: name: grafana namespace: logging spec: selector: app: grafana ports: - port: 3000 targetPort: 3000 type: LoadBalancer四、日志管理优化1. 日志轮转apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit-config namespace: logging data: fluent-bit.conf: | [SERVICE] Flush 1 Log_Level info Daemon off Parsers_File parsers.conf [INPUT] Name tail Path /var/log/containers/*.log Parser docker Tag kube.* Refresh_Interval 5 Rotate_Wait 30 Mem_Buf_Limit 5MB Skip_Long_Lines On2. 日志过滤apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit-config namespace: logging data: fluent-bit.conf: | [FILTER] Name grep Match kube.* Exclude log .*debug.* [FILTER] Name record_modifier Match kube.* Record environment production Remove_key log_processed3. 日志监控apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: fluent-bit-metrics namespace: monitoring spec: selector: matchLabels: app: fluent-bit endpoints: - port: metrics interval: 15s --- apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: fluent-bit-alerts namespace: monitoring spec: groups: - name: fluent-bit rules: - alert: FluentBitErrorRateHigh expr: rate(fluentbit_output_errors_total[5m]) 0 for: 5m labels: severity: warning annotations: summary: Fluent Bit error rate high description: Fluent Bit is experiencing high error rate五、常见问题1. 日志丢失解决方案配置适当的缓冲区确保存储可靠性监控日志采集状态2. 日志查询慢解决方案优化索引策略使用合适的存储配置合理的保留策略3. 资源消耗大解决方案优化日志采集配置过滤不必要的日志使用轻量级日志方案六、最佳实践总结集中管理使用集中式日志管理系统实时采集配置实时日志采集日志过滤过滤不必要的日志存储策略配置合理的存储和保留策略监控告警监控日志系统状态可视化使用 Grafana 实现日志可视化七、总结Kubernetes 日志管理是运维工作的重要组成部分。按照本文的最佳实践你可以构建一个高效、可靠的日志管理系统炸了