NodeLoadHigh: what to do when it fires

Alert rule

alert: NodeLoadHigh
expr: node_load5 / on (instance) count by (instance) (node_cpu_seconds_total{mode="idle"}) > 2
for: 15m
severity: warning

Alert message: High load average ({{ $labels.instance }})

What it means

The 5-minute load average has been more than 2 times the number of cores for 15 minutes. Load includes both processes waiting for CPU and processes in uninterruptible sleep (usually waiting for disk/NFS).

Common causes

CPU overload, a slow disk or network (NFS), processes in the D state, too many parallel tasks.

Diagnostics

uptime
nproc
top -b -n 1 | head -n 15
ps -eo state,pid,cmd | awk '$1 ~ /D/' | head          # processes in D state
iostat -x 1 3

How to fix

If %CPU is high, act as in NodeCpuHigh; if there are many D-state processes, look for a disk/NFS problem, as in NodeIowaitHigh.

When it is OK to silence

If it is a known batch workload outside working hours.

Related alerts

• NodeCpuHigh
• NodeIowaitHigh
• NodeNetworkErrors
• NodeSwapHigh
• NodeClockSkew

The full pack of 44 tested rules + Alertmanager → Telegram: 790 ₽ (≈ $9, card payment in rubles)
Open in Telegram · Free Monitoring Lite kit · All runbook alerts