NodeDiskSpaceCritical: what to do when it fires

Alert rule

alert: NodeDiskSpaceCritical
expr: node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs|devtmpfs|ramfs|iso9660|autofs|fuse.*|nfs.*|cifs"}
  / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs|devtmpfs|ramfs|iso9660|autofs|fuse.*|nfs.*|cifs"} * 100 < 5
and node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs|devtmpfs|ramfs|iso9660|autofs|fuse.*|nfs.*|cifs"} > 0
and node_filesystem_readonly{fstype!~"tmpfs|overlay|squashfs|devtmpfs|ramfs|iso9660|autofs|fuse.*|nfs.*|cifs"} == 0
for: 5m
severity: critical

Alert message: Disk almost full: {{ $labels.mountpoint }} ({{ $labels.instance }})

What it means

Less than 5% free space is left on the partition. Writes may start failing with errors at any moment.

Common causes

The same as for NodeDiskSpaceLow, but the headroom is used up: fast log growth, a runaway application writing to a file, a database overflow.

Diagnostics

df -hT
du -xh --max-depth=2 / 2>/dev/null | sort -h | tail -n 20
ls -lhS /var/log | head
journalctl --disk-usage
docker system df

How to fix

Free space quickly: truncate the largest log (: > /var/log/<file> rather than rm if a process holds the file open), journalctl --vacuum-size=200M, docker system prune (read carefully what will be removed), delete old backups. Then find the source of the growth and extend the volume.

When it is OK to silence

Not until space has been freed. The exception is a read-only partition or an image that is not expected to have free space (for example a mounted iso/squashfs, if it is not filtered out).

Related alerts

The full pack of 44 tested rules + Alertmanager → Telegram: 790 ₽ (≈ $9, card payment in rubles)
Open in Telegram · Free Monitoring Lite kit · All runbook alerts