NodeCpuHigh: what to do when it fires

Alert rule

alert: NodeCpuHigh
expr: 100 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100 > 90
for: 15m
severity: warning

Alert message: High CPU usage ({{ $labels.instance }})

What it means

Average CPU load across all cores has been above 90% for 15 minutes.

Common causes

A traffic peak, a runaway process, a heavy backup/compression/build, a miner after a compromise, too few cores for the load.

Diagnostics

top -b -n 1 | head -n 15
ps aux --sort=-%cpu | head -n 10
pidstat 1 5               # sysstat package
docker stats --no-stream

How to fix

Stop the culprit or limit it (nice, cpulimit, cgroup/Docker limits), optimise queries, add cores/nodes. Check an unknown process with high load for signs of compromise.

When it is OK to silence

For hosts meant for batch computation, where 100% CPU is normal: exclude them or raise the threshold.

Related alerts

• NodeLoadHigh
• NodeSwapHigh
• NodeIowaitHigh
• NodeMemoryCritical
• NodeNetworkErrors

The full pack of 44 tested rules + Alertmanager → Telegram: 790 ₽ (≈ $9, card payment in rubles)
Open in Telegram · Free Monitoring Lite kit · All runbook alerts