NodeDiskWillFill: what to do when it fires

Alert rule

alert: NodeDiskWillFill
expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs|devtmpfs|ramfs|iso9660|autofs|fuse.*|nfs.*|cifs"}[6h], 24 * 3600) < 0
and node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs|devtmpfs|ramfs|iso9660|autofs|fuse.*|nfs.*|cifs"}
  / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs|devtmpfs|ramfs|iso9660|autofs|fuse.*|nfs.*|cifs"} * 100 < 40
and node_filesystem_readonly{fstype!~"tmpfs|overlay|squashfs|devtmpfs|ramfs|iso9660|autofs|fuse.*|nfs.*|cifs"} == 0
for: 1h
severity: warning

Alert message: Disk {{ $labels.mountpoint }} will fill up within a day ({{ $labels.instance }})

What it means

Based on the linear trend over the last 6 hours (predict_linear), the partition will run out of space within the next 24 hours. The rule fires only if less than 40% is free, to avoid noise on large, mostly empty disks.

Common causes

abnormal growth of logs or the database;
a running export, backup or data import;
a leak of temporary files.

Diagnostics

df -h <mountpoint>
du -xh --max-depth=2 <mountpoint> 2>/dev/null | sort -h | tail -n 20
find <mountpoint> -xdev -type f -mmin -60 -size +100M -exec ls -lh {} +

Compare the node_filesystem_avail_bytes graph over the last 6–24 hours: is the growth steady or a sudden jump?

How to fix

Find and stop the source of the growth, add rotation, extend the disk. If the write is a one-off (an import), estimate whether the space will last until the operation finishes.

When it is OK to silence

If the growth is a known one-off (a migration, an import) and there is clearly enough space. In that case set the silence for a few hours, not days.

Related alerts

The full pack of 44 tested rules + Alertmanager → Telegram: 790 ₽ (≈ $9, card payment in rubles)
Open in Telegram · Free Monitoring Lite kit · All runbook alerts