KubeNodeNotReady: what to do when it fires

Alert rule

alert: KubeNodeNotReady
expr: kube_node_status_condition{condition="Ready",status="true"} == 0
for: 5m
severity: critical

Alert message: Node {{ $labels.node }} is NotReady

What it means

A node has not reported the Ready state for 5 minutes. Kubernetes stops scheduling Pods on it and will begin evicting them after some time.

Common causes

The node is down or hung, kubelet is stopped, the network between the node and the API server, a shortage of resources on the node, a container runtime error.

Diagnostics

kubectl get nodes -o wide
kubectl describe node <node>                    # Conditions and Events
kubectl get pods -A -o wide --field-selector spec.nodeName=<node>
ssh <node> 'systemctl status kubelet containerd --no-pager; journalctl -u kubelet -n 100 --no-pager'
ssh <node> 'df -h; free -h; uptime'

How to fix

Restart kubelet/the container runtime, free disk and memory, reboot or replace the node. If necessary, run kubectl drain <node> --ignore-daemonsets --delete-emptydir-data.

When it is OK to silence

During planned work on the node.

Related alerts

• KubeNodePressure
• KubePvcCritical
• KubeHpaMaxedOut
• KubePvcAlmostFull
• KubeApiCertExpiresSoon

The full pack of 44 tested rules + Alertmanager → Telegram: 790 ₽ (≈ $9, card payment in rubles)
Open in Telegram · Free Monitoring Lite kit · All runbook alerts