KubeletDown: what to do when it fires

Alert rule

alert: KubeletDown
expr: up{job="kubelet"} == 0
for: 5m
severity: critical

Alert message: Kubelet is down ({{ $labels.instance }})

What it means

Prometheus has not received kubelet metrics from a node for 5 minutes. The node may have failed, or kubelet may have stopped, or the network to port 10250 may be broken.

Common causes

The node is down, kubelet crashed, the kubelet certificate expired, a network block, the node was removed from the cluster but is still in the targets.

Diagnostics

kubectl get nodes -o wide
kubectl describe node <node>
ssh <node> 'systemctl status kubelet --no-pager; journalctl -u kubelet -n 100 --no-pager'

How to fix

Restart kubelet (systemctl restart kubelet), restore the node, check the certificates. If the node was removed on purpose, remove it from the scrape targets.

When it is OK to silence

When taking a node out of the cluster (drain).

Related alerts

• KubePodCrashLooping
• KubeStateMetricsDown
• KubePodNotReady
• KubePodImagePullBackOff
• KubeContainerOomKilled

The full pack of 44 tested rules + Alertmanager → Telegram: 790 ₽ (≈ $9, card payment in rubles)
Open in Telegram · Free Monitoring Lite kit · All runbook alerts