BlackboxSlowResponse: what to do when it fires

Alert rule

alert: BlackboxSlowResponse
expr: avg_over_time(probe_duration_seconds{job=~"blackbox.*"}[5m]) > 2 and probe_success{job=~"blackbox.*"} == 1
for: 10m
severity: warning

Alert message: Slow response: {{ $labels.instance }}

What it means

The average probe duration over 5 minutes has exceeded 2 seconds for 10 minutes, while the probe succeeds. The site responds, but slowly.

Common causes

Load on the server or database, slow external APIs, network or DNS problems, caching errors, large pages.

Diagnostics

curl -sS -o /dev/null -w 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n' https://example.com

Then look at the state of the server itself: NodeCpuHigh, NodeIowaitHigh, and the application and database logs.

How to fix

Depending on which phase is slow in curl: DNS, connection, TLS or waiting for the application's response. Optimise queries, add a cache, add resources.

When it is OK to silence

If the 2 s threshold is too strict for this service, raise it in the rule. A silence is acceptable for the duration of a known load.

Related alerts

• BlackboxHttpStatusNotOk
• BlackboxProbeFailed
• BlackboxSslExpiresSoon
• BlackboxExporterDown
• BlackboxSslExpiresCritical

The full pack of 44 tested rules + Alertmanager → Telegram: 790 ₽ (≈ $9, card payment in rubles)
Open in Telegram · Free Monitoring Lite kit · All runbook alerts