NodeMemoryCritical: what to do when it fires

Alert rule

alert: NodeMemoryCritical
expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 4
for: 5m
severity: critical

Alert message: Memory nearly exhausted, OOM-killer risk ({{ $labels.instance }})

What it means

Less than 4% of memory is available: the kernel will soon start killing processes (OOM killer), and the system will go into swap and slow down.

Common causes

The same as in NodeMemoryLow, but more acute: a sharp leak, an unexpected load spike.

Diagnostics

free -h
ps aux --sort=-%mem | head -n 10
dmesg -T | grep -i -E 'out of memory|killed process' | tail
journalctl -k --since "1 hour ago" | grep -i oom
vmstat 1 5

How to fix

Stop or restart the main consumer, take load off temporarily, add memory or swap as a temporary measure. Then look for the root cause.

When it is OK to silence

Not until the system has stabilised.

Related alerts

• NodeMemoryLow
• NodeSwapHigh
• NodeCpuHigh
• NodeInodesLow
• NodeLoadHigh

The full pack of 44 tested rules + Alertmanager → Telegram: 790 ₽ (≈ $9, card payment in rubles)
Open in Telegram · Free Monitoring Lite kit · All runbook alerts