Русская версия

KubePodImagePullBackOff: what to do when it fires

Alert rule

alert: KubePodImagePullBackOff
expr: kube_pod_container_status_waiting_reason{reason=~"ImagePullBackOff|ErrImagePull"} == 1
for: 10m
severity: warning

Alert message: Pod {{ $labels.namespace }}/{{ $labels.pod }} cannot pull its image

What it means

A container has been unable to pull its image from the registry (ErrImagePull / ImagePullBackOff) for 10 minutes.

Common causes

A typo in the image name or tag, the image is not in the registry, no permissions (the imagePullSecret is missing or stale), the registry is unavailable or pull limits apply, no network on the node.

Diagnostics

kubectl -n <ns> describe pod <pod>              # error message in Events
kubectl -n <ns> get pod <pod> -o jsonpath='{.spec.containers[*].image}'
kubectl -n <ns> get secret                      # imagePullSecrets
ssh <node> 'crictl pull <image>'

How to fix

Fix the name/tag, recreate the registry access secret (kubectl create secret docker-registry ...) and reference it in imagePullSecrets, check that the registry is reachable from the node.

When it is OK to silence

Not until the application is running.

Related alerts

The full pack of 44 tested rules + Alertmanager → Telegram: 790 ₽ (≈ $9, card payment in rubles)
Open in Telegram · Free Monitoring Lite kit · All runbook alerts