NodeRebooted: what to do when it fires

Alert rule

alert: NodeRebooted
expr: time() - node_boot_time_seconds < 600
for: 1m
severity: warning

Alert message: Host {{ $labels.instance }} was recently rebooted

What it means

Less than 10 minutes have passed since the host last booted. This is an informational alert: it exists so that you notice unexpected reboots that would otherwise go unnoticed (the host is visible again and NodeDown has no time to fire).

Common causes

A planned kernel update, a hardware or hypervisor failure, a kernel panic, an OOM with reset, a power outage, a watchdog.

Diagnostics

uptime -s
last -x reboot shutdown | head
journalctl --list-boots | tail -n 5
journalctl -b -1 -n 100 --no-pager        # end of the previous boot
systemctl --failed

How to fix

If the reboot was planned, nothing. If not, find the cause in the journal of the previous boot and make sure the services came back up (systemctl --failed).

When it is OK to silence

During scheduled updates that involve a reboot.

Related alerts

• NodeClockNotSynced
• NodeClockSkew
• NodeNetworkErrors
• NodeIowaitHigh

The full pack of 44 tested rules + Alertmanager → Telegram: 790 ₽ (≈ $9, card payment in rubles)
Open in Telegram · Free Monitoring Lite kit · All runbook alerts