BlackboxProbeFailed: what to do when it fires

Alert rule

alert: BlackboxProbeFailed
expr: probe_success{job=~"blackbox.*"} == 0
for: 3m
severity: critical

Alert message: Site or service is down: {{ $labels.instance }}

What it means

The probe of a target has been failing (probe_success == 0) for 3 minutes in a row. The site or service is unavailable or responds differently from what the probe module expects.

Common causes

The service is down, DNS problems, a closed port or firewall, an expired/broken TLS certificate, a timeout, a wrong response code, the expected response text did not match.

Diagnostics

curl -sS -v -m 10 -o /dev/null https://example.com
dig +short example.com
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null | openssl x509 -noout -dates
curl -sS -m 10 'http://<exporter>:9115/probe?module=http_2xx&target=https://example.com&debug=true'

The debug=true parameter makes blackbox-exporter return a detailed probe log.

How to fix

Based on the diagnostics: bring the service up, fix DNS or the certificate, open the port. If there are false positives due to a timeout, increase timeout in the blackbox module.

When it is OK to silence

During planned work or a site migration.

Related alerts

• BlackboxSlowResponse
• BlackboxExporterDown
• BlackboxHttpStatusNotOk
• BlackboxSslExpiresSoon
• BlackboxSslExpiresCritical

The full pack of 44 tested rules + Alertmanager → Telegram: 790 ₽ (≈ $9, card payment in rubles)
Open in Telegram · Free Monitoring Lite kit · All runbook alerts