BlackboxExporterDown: what to do when it fires

Alert rule

alert: BlackboxExporterDown
expr: up{job=~"blackbox.*"} == 0
for: 3m
severity: critical

Alert message: Probe {{ $labels.instance }} is not running

What it means

Prometheus has received no response from blackbox-exporter when probing a target (up == 0) for 3 minutes. Website checks are not being performed, so the other blackbox alerts are silent.

Common causes

The blackbox-exporter service is stopped, the wrong exporter address in relabel_configs (the replacement parameter), a firewall, an error in the module configuration.

Diagnostics

systemctl status prometheus-blackbox-exporter --no-pager     # or your unit name
curl -sS -m 10 'http://<exporter>:9115/probe?module=http_2xx&target=https://example.com' | head -n 20
journalctl -u prometheus-blackbox-exporter -n 50 --no-pager

For Docker/Kubernetes, use docker logs / kubectl logs of the corresponding Pod.

How to fix

Start the exporter, fix the address and port (9115 by default) in the Prometheus configuration, check the syntax of blackbox.yml.

When it is OK to silence

During an exporter upgrade.

Related alerts

• BlackboxProbeFailed
• BlackboxSlowResponse
• BlackboxHttpStatusNotOk
• BlackboxSslExpiresSoon

The full pack of 44 tested rules + Alertmanager → Telegram: 790 ₽ (≈ $9, card payment in rubles)
Open in Telegram · Free Monitoring Lite kit · All runbook alerts