KubeStateMetricsDown: what to do when it fires

Alert rule

alert: KubeStateMetricsDown
expr: up{job=~"kube-state-metrics.*"} == 0
for: 5m
severity: critical

Alert message: kube-state-metrics is down ({{ $labels.instance }})

What it means

Prometheus has been unable to scrape kube-state-metrics for 5 minutes. All alerts about Pods, Deployments, StatefulSets, DaemonSets, Jobs, HPAs and nodes are blind during this period.

Common causes

The kube-state-metrics Pod crashed or is being rescheduled, a network policy blocks the scrape, the Service/ServiceMonitor name changed, the cluster ran out of resources.

Diagnostics

kubectl -n <ns> get pods -l app.kubernetes.io/name=kube-state-metrics
kubectl -n <ns> describe pod -l app.kubernetes.io/name=kube-state-metrics
kubectl -n <ns> logs -l app.kubernetes.io/name=kube-state-metrics --tail=100
kubectl -n <ns> get servicemonitor,endpoints | grep -i state-metrics

How to fix

Fix the cause of the Pod failure (OOM, RBAC permissions, image), restore the ServiceMonitor/Service to a working state, raise the limits if memory is short.

When it is OK to silence

During a kube-prometheus-stack upgrade.

Related alerts

• KubeletDown
• KubePodCrashLooping
• KubePodNotReady
• KubePodImagePullBackOff

The full pack of 44 tested rules + Alertmanager → Telegram: 790 ₽ (≈ $9, card payment in rubles)
Open in Telegram · Free Monitoring Lite kit · All runbook alerts