NodeDiskSpaceLow: what to do when it fires

Alert rule

alert: NodeDiskSpaceLow
expr: node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs|devtmpfs|ramfs|iso9660|autofs|fuse.*|nfs.*|cifs"}
  / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs|devtmpfs|ramfs|iso9660|autofs|fuse.*|nfs.*|cifs"} * 100 < 15
and node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs|devtmpfs|ramfs|iso9660|autofs|fuse.*|nfs.*|cifs"} > 0
and node_filesystem_readonly{fstype!~"tmpfs|overlay|squashfs|devtmpfs|ramfs|iso9660|autofs|fuse.*|nfs.*|cifs"} == 0
for: 10m
severity: warning

Alert message: Low disk space on {{ $labels.mountpoint }} ({{ $labels.instance }})

What it means

Less than 15% free space is left on the partition (space available to unprivileged processes is counted). There are no real consequences yet, but the headroom is shrinking.

Common causes

log growth without rotation;
accumulated Docker images and layers, dumps, backups, package manager caches;
database growth;
a deleted file still held open by a running process (the space is not freed).

Diagnostics

df -hT
du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -n 15
du -xh --max-depth=1 /var 2>/dev/null | sort -h | tail -n 10
journalctl --disk-usage
docker system df          # if Docker is on the host
lsof +L1 | head           # deleted but still open files

How to fix

Set up log rotation (logrotate, journalctl --vacuum-size=500M), delete what is not needed (apt clean / dnf clean all, docker image prune), move data to another disk or extend the volume. Restart processes that hold deleted files open.

When it is OK to silence

For partitions that fill up normally and are cleaned by the application itself (a cache, for example), it is better to exclude that mountpoint in the rule expression. A temporary silence is acceptable if a disk extension is already planned.

Related alerts

The full pack of 44 tested rules + Alertmanager → Telegram: 790 ₽ (≈ $9, card payment in rubles)
Open in Telegram · Free Monitoring Lite kit · All runbook alerts