当前位置: 首页>前端>正文

prometheus 设置cpu占用报警 prometheus告警功能

prometheus监控K8S 监控告警功能

alertmanager邮箱告警配置

首先开通SMTP服务,QQ邮箱:设置–帐号–开通POP3/SMTP服务,记住生成的密码(其它邮箱同理)

prometheus 设置cpu占用报警 prometheus告警功能,prometheus 设置cpu占用报警 prometheus告警功能_Pod,第1张

编辑prometheus的values.yaml文件,配置邮箱告警

[root@master01 prometheus]# vi prometheus/values.yaml 
# 定位到alertmanager的配置文件,
alertmanagerFiles:
  alertmanager.yml:
    global:
      resolve_timeout: 5m
      # 邮箱告警配置
      smtp_hello: 'prometheus'
      smtp_from: 'xxx@qq.com'
      smtp_smarthost: 'smtp.qq.com:465'  # 其它邮箱请填写相应的host
      smtp_auth_username: 'xxx@qq.com'
      smtp_auth_password: 'xxxxxxxxxxxxxxxx'
      smtp_require_tls: false            # qq邮箱需要设定
    templates:
    - '/etc/config/*.tmpl'               # 指定告警模板路径
    receivers:
      - name: email
        email_configs:
        - to: 'x@qq.com'                 #接收报警邮箱地址
          headers: {"subject":'{{ template "email.header" . }}'}
          html: '{{ template "email.html" . }}'
          send_resolved: true            # 发送报警解除邮件
    route:
      group_wait: 5s                     # 分组等待时间
      group_interval: 5s                 # 上下两组发送告警的间隔时间
      receiver: email
      repeat_interval: 5m                # 重复发送告警时间
    inhibit_rules:                       # 告警抑制:当多级别规则同时生效时,只发送最高级别的告警
    - source_match:
        severity: 'critical'
      target_match:
        severity: 'warning'
      equal: ['alertname']
template_email.tmpl: |-           # 告警模版
    {{ define "email.header" }}
    {{ if eq .Status "firing"}}[Warning]: {{ range .Alerts }}{{ .Annotations.summary }} {{ end }}{{ end }}
    {{ if eq .Status "resolved"}}[Resolved]: {{ range .Alerts }}{{ .Annotations.resolve_summary }} {{ end }}{{ end }}
    {{ end }}

    {{ define "email.html" }}
    {{ if gt (len .Alerts.Firing) 0 -}}
    <font color="#FF0000"><h3>[Warning]:</h3></font>
    {{ range .Alerts }}
    告警级别:{{ .Labels.severity }}   <br>
    告警类型:{{ .Labels.alertname }}   <br>
    故障主机: {{ .Labels.instance }}   <br>
    告警主题: {{ .Annotations.summary }}   <br>
    告警详情: {{ .Annotations.description }}   <br>
    触发时间: {{ (.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}   <br>
    {{- end }}
    {{- end }}

    {{ if gt (len .Alerts.Resolved) 0 -}}
    <font color="#66CDAA"><h3>[Resolved]:</h3></font>
    {{ range .Alerts }}
    告警级别:{{ .Labels.severity }}   <br>
    告警类型:{{ .Labels.alertname }}   <br>
    故障主机: {{ .Labels.instance }}   <br>
    告警主题: {{ .Annotations.resolve_summary }}   <br>
    告警详情: {{ .Annotations.resolve_description }}   <br>
    触发时间: {{ (.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}   <br>
    恢复时间: {{ (.EndsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}   <br>
    {{- end }}
    {{- end }}
    {{- end }}

prometheus告警规则配置

接着配置告警规则,以“物理节点状态”为例,先在prometheus控制台测试此条告警规则,确保输出有效:up{component="node-exporter"}

prometheus 设置cpu占用报警 prometheus告警功能,prometheus 设置cpu占用报警 prometheus告警功能_配置文件_02,第2张

在values中配置告警规则(就在alertmanger的配置文件下方)

serverFiles:
  alerting_rules.yml:
    groups:
      - name: 物理节点状态-监控告警
        rules:
          - alert: Node-up
            expr: up {component="node-exporter"} == 0
            for: 2s
            labels:
              severity: critical
            annotations:
              summary: "服务器{{ $labels.kubernetes_node }}已停止运行!"
              description: "检测到服务器{{ $labels.kubernetes_node }}已异常停止,IP: {{ $labels.instance }},请排查!"
              resolve_summary: "服务器{{ $labels.kubernetes_node }}已恢复运行!"
              resolve_description: "服务器{{ $labels.kubernetes_node }}已恢复运行,IP: {{ $labels.instance }}。"

更新prometheus

编辑完成后,更新prometheus

每次增加规则中都需要upgrage,更新后pod中的“configmap-reload”容器会重载配置文件,可能需要等待几分钟

[root@master01 ]# helm upgrade prometheus -n prometheus .

查询alertmanger的配置文件是否更新(server同理):

[root@master01 prometheus]# kubectl logs -f -n prometheus prometheus-alertmanager-7757db759b-9nq9c prometheus-alertmanager

prometheus 设置cpu占用报警 prometheus告警功能,prometheus 设置cpu占用报警 prometheus告警功能_服务器_03,第3张

查询server的configmap(alertmanger同理)

[root@master01 prometheus]# kubectl get configmap -n prometheus

[root@master01 prometheus]# kubectl describe configmaps -n prometheus prometheus-server

prometheus 设置cpu占用报警 prometheus告警功能,prometheus 设置cpu占用报警 prometheus告警功能_配置文件_04,第4张

在alert控制台查看告警规则是否生效:

prometheus 设置cpu占用报警 prometheus告警功能,prometheus 设置cpu占用报警 prometheus告警功能_prometheus_05,第5张

告警测试

关掉其中一台节点测试

prometheus 设置cpu占用报警 prometheus告警功能,prometheus 设置cpu占用报警 prometheus告警功能_配置文件_06,第6张

可观察到Alert中,该告警规则状态由Inactive转到Pending再到Firing,而当状态转为Firing,将发送告警邮件

Pending到Firing的变化默认为1分钟,若想缩短时间,请修改value.yaml中的server.global.scrape_interval字段,如15s

prometheus 设置cpu占用报警 prometheus告警功能,prometheus 设置cpu占用报警 prometheus告警功能_服务器_07,第7张

告警邮件

prometheus 设置cpu占用报警 prometheus告警功能,prometheus 设置cpu占用报警 prometheus告警功能_配置文件_08,第8张

附告警规则
部分参考自阿里云Prometheus监控报警规则

K8S组件状态
target状态
sum by (instance,job) (up)
CPU使用率
round (sum by (instance,job) (rate(process_cpu_seconds_total[2m]) * 100)) > 80
句柄数
sum by (instance,job) (process_open_fds) > 600
虚拟内存
sum by (instance,job) (round(process_virtual_memory_bytes/1024/1024)) > 4096
集群资源状态
资源限制:总CPU资源过载(超过80%)
集群 CPU 过度使用,CPU 已经过度使用无法容忍节点故障

round(sum(kube_pod_container_resource_requests{resource=“cpu”}) / sum(kube_node_status_allocatable{resource=“cpu”})*100) > 80
资源限制:总内存资源过载(超过80%)
集群内存过度使用,内存已经过度使用无法容忍节点故障

round(sum(kube_pod_container_resource_requests{resource=“memory”}) / sum(kube_node_status_allocatable{resource=“memory”})100) > 80
KubeletTooManyPods(Pod过多)
max(max(kubelet_running_pods) by(instance) * on(instance) group_left(node) kubelet_node_name) by(node) / max(kube_node_status_capacity{resource=“pods”} != 1) by(node) > 0.9
Node资源状态
CPU使用率
round((1 - avg(rate(node_cpu_seconds_total{component=“node-exporter”,mode=“idle”}[5m])) by (instance)) * 100)
内存
round((1 - (node_memory_MemAvailable_bytes / (node_memory_MemTotal_bytes)))
100)
剩余容量
(round((node_filesystem_avail_bytes{fstype=~“ext4|xfs”} / node_filesystem_size_bytes{fstype=~“ext4|xfs”}) * 100 < 30 ) and node_filesystem_readonly{fstype=~“ext4|xfs”} == 0)
预测剩余容量
( round(predict_linear(node_filesystem_avail_bytes{fstype=~“ext4|xfs”}[24h], 7246060)/1024/1024/1024) < 30 and node_filesystem_readonly{fstype=~“ext4|xfs”} == 0 )
节点磁盘的IO使用率
100-(avg(rate(node_disk_io_time_seconds_total[2m])) by(instance)
100) < 80
NodeNetworkReceiveErrs(Node网络接收错误)
sum (increase(node_network_receive_errs_total[2m])) by (instance) > 10
NodeNetworkTransmitErrs(Node网络传输错误)
sum (increase(node_network_transmit_errs_total[2m])) by (instance) > 10
入网流量带宽
持续5分钟高于100M

((sum(rate (node_network_receive_bytes_total{device!~‘tap.|veth.|br.|docker.|virbr*|lo*’}[5m])) by (instance)) /102400) > 100
出网流量带宽
((sum(rate (node_network_transmit_bytes_total{device!~‘tap.|veth.|br.|docker.|virbr*|lo*’}[5m])) by (instance))/102400) > 100
TCP_ESTABLISHED过高
node_netstat_Tcp_CurrEstab > 1000
Pod
PodCpu75(Pod的CPU使用率大于75%)
!=“POD”,!=“”}[2m]

{name=~“.+”}:筛选,避免重复指标

round(100*(sum(rate(container_cpu_usage_seconds_total {name=~“.+”}[2m])) by (namespace,pod) / sum(kube_pod_container_resource_limits{resource=“cpu”}) by (namespace,pod))) > 75
PodMemory75(Pod的内存使用率大于75%)
100 * sum(container_memory_working_set_bytes{name=~“.+”}) by (namespace,pod) / sum(kube_pod_container_resource_limits {resource=“memory”}) by (namespace,pod) > 75
pod_status_no_running(Pod的状态为未运行)
sum (kube_pod_status_phase{phase!=“Running”}) by (namespace,pod,phase) > 0
PodMem4GbRestart(Pod的内存大于4096MB)
(sum (container_memory_working_set_bytes{name=~“.+”})by (namespace,pod,container_name) /1024/1024) > 4096
PodRestart(Pod重启)
{pod!~“aws-load-balancer-controller.*”}

sum (round(increase (kube_pod_container_status_restarts_total[5m]))) by (namespace,pod) > 0
KubePodCrashLooping(Pod出现循环崩溃)
rate(kube_pod_container_status_restarts_total{app_kubernetes_io_name=“kube-state-metrics”}[15m]) * 60 * 5 > 0
KubePodNotReady(Pod未准备好)
sum by (namespace, pod) (max by(namespace, pod) (kube_pod_status_phase{app_kubernetes_io_name=“kube-state-metrics”, phase=~“Pending|Unknown”}) * on(namespace, pod) group_left(owner_kind) max by(namespace, pod, owner_kind) (kube_pod_owner{owner_kind!=“Job”})) > 0
KubeContainerWaiting(容器等待)
sum by (namespace, pod,) (kube_pod_container_status_waiting{app_kubernetes_io_name=“kube-state-metrics”}) > 0
Deployment
KubeDeploymentGenerationMismatch(出现部署集版本不匹配)
kube_deployment_status_observed_generation{app_kubernetes_io_name=“kube-state-metrics”} != kube_deployment_metadata_generation{app_kubernetes_io_name=“kube-state-metrics”}
KubeDeploymentReplicasMismatch(出现部署集副本不匹配)
( kube_deployment_spec_replicas{app_kubernetes_io_name=“kube-state-metrics”} != kube_deployment_status_replicas_available{app_kubernetes_io_name=“kube-state-metrics”} ) and ( changes(kube_deployment_status_replicas_updated{app_kubernetes_io_name=“kube-state-metrics”}[5m]) == 0 )
检测到部署集有更新
sum by (namespace, deployment) (changes(kube_deployment_status_observed_generation{app_kubernetes_io_name=“kube-state-metrics”}[5m])) > 0
StatefulSet
KubeStatefulSetGenerationMismatch(状态集版本不匹配)
kube_statefulset_status_observed_generation{app_kubernetes_io_name=“kube-state-metrics”} != kube_statefulset_metadata_generation{app_kubernetes_io_name=“kube-state-metrics”}
KubeStatefulSetReplicasMismatch(状态集副本不匹配)
( kube_statefulset_status_replicas_ready{app_kubernetes_io_name=“kube-state-metrics”} != kube_statefulset_status_replicas{app_kubernetes_io_name=“kube-state-metrics”} ) and ( changes(kube_statefulset_status_replicas_updated{app_kubernetes_io_name=“kube-state-metrics”}[5m]) == 0 )
检测到状态集有更新
sum by (namespace, statefulset) (changes(kube_statefulset_status_observed_generation{app_kubernetes_io_name=“kube-state-metrics”}[5m]))
PV&PVC
KubePersistentVolumeFillingUp(块存储PVC容量即将不足)
sum by (namespace,persistentvolumeclaim) (round(kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes*100)) < 20
KubePersistentVolumeErrors(PV容量出错)
sum by (persistentvolume) (kube_persistentvolume_status_phase{phase=~“Failed|Pending”,app_kubernetes_io_name=“kube-state-metrics”}) > 0
KubePersistentVolumeFillingUp(PVC空间耗尽预测)
通过PVC资源使用6小时变化率预测 接下来4天的磁盘使用率

(kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes ) < 0.4 and predict_linear(kubelet_volume_stats_available_bytes[6h], 4 * 24



https://www.xamrdz.com/web/2tq1962312.html

相关文章: