NodeIowaitHigh: what to do when it fires

Alert rule

alert: NodeIowaitHigh
expr: avg by (instance) (rate(node_cpu_seconds_total{mode="iowait"}[5m])) * 100 > 30
for: 15m
severity: warning

Alert message: CPU idle waiting on disk ({{ $labels.instance }})

What it means

The CPU spends more than 30% of its time waiting for disk operations to complete. The disk (or network storage) is the bottleneck.

Common causes

Heavy database queries, a backup/copy, a degraded disk or RAID, overloaded cloud storage, active swapping, an unavailable NFS.

Diagnostics

iostat -x 1 5              # sysstat package: watch %util and await
iotop -oPa                 # who is writing/reading
vmstat 1 5                 # wa, si, so columns
dmesg -T | grep -i -E 'i/o error|ata|nvme|blk_update' | tail
smartctl -H /dev/sda       # smartmontools package

How to fix

Stop or limit the culprit (ionice -c3), spread the load across disks, optimise queries/indexes, replace a failing disk, increase the IOPS of a cloud volume.

When it is OK to silence

During a scheduled backup or a RAID rebuild.

Related alerts

• NodeNetworkErrors
• NodeLoadHigh
• NodeClockSkew
• NodeCpuHigh
• NodeClockNotSynced

The full pack of 44 tested rules + Alertmanager → Telegram: 790 ₽ (≈ $9, card payment in rubles)
Open in Telegram · Free Monitoring Lite kit · All runbook alerts