NodeDown: what to do when it fires

Alert rule

alert: NodeDown
expr: up{job=~"node.*"} == 0
for: 3m
severity: critical

Alert message: Host {{ $labels.instance }} is down

What it means

Prometheus has been unable to scrape node-exporter for 3 minutes (up == 0). This is not necessarily a server outage: only the exporter or the network path to it may be down.

Common causes

• the server is powered off, hung or rebooting;
• a network problem or firewall rule between Prometheus and the host (port 9100);
• the node-exporter service is stopped or crashed;
• the wrong address in scrape_configs.

Diagnostics

ping -c 3 <host>
curl -sS -m 5 http://<host>:9100/metrics | head
ssh <host> 'systemctl status node_exporter --no-pager'
ssh <host> 'journalctl -u node_exporter -n 50 --no-pager'
ssh <host> 'uptime; dmesg -T | tail -n 30'

The unit name depends on how it was installed (node_exporter, prometheus-node-exporter).

How to fix

If the server is unreachable, deal with the hosting provider or the hypervisor console. If only the exporter is down, start it: systemctl restart node_exporter, and check the firewall (ss -ltnp | grep 9100).

When it is OK to silence

During planned work or when decommissioning the host (better still, remove the target from the Prometheus configuration).

Related alerts

• NodeDiskSpaceLow
• NodeDiskSpaceCritical
• NodeDiskWillFill
• NodeInodesLow

The full pack of 44 tested rules + Alertmanager → Telegram: 790 ₽ (≈ $9, card payment in rubles)
Open in Telegram · Free Monitoring Lite kit · All runbook alerts