Русская версия

KubePodCrashLooping: what to do when it fires

Alert rule

alert: KubePodCrashLooping
expr: kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"} == 1
for: 10m
severity: warning

Alert message: Pod {{ $labels.namespace }}/{{ $labels.pod }} is in CrashLoopBackOff

What it means

A container has been in CrashLoopBackOff for 10 minutes: the process starts, crashes, and Kubernetes restarts it with a growing delay.

Common causes

An application error at startup, a wrong config/secret, an unavailable database, out of memory (OOMKilled), a wrong command, a failing liveness probe.

Diagnostics

kubectl -n <ns> get pod <pod>
kubectl -n <ns> describe pod <pod>              # Events and Last State sections
kubectl -n <ns> logs <pod> -c <container> --previous
kubectl -n <ns> get events --sort-by=.lastTimestamp | tail -n 20

The --previous flag shows the log of the previous (crashed) run.

How to fix

Fix the cause shown in the log: the configuration, the secret, the dependency. If the liveness probe is failing, increase initialDelaySeconds / failureThreshold. After fixing: kubectl -n <ns> rollout restart deployment/<name>.

When it is OK to silence

For a known broken Pod in a test namespace. Not for production.

Related alerts

The full pack of 44 tested rules + Alertmanager → Telegram: 790 ₽ (≈ $9, card payment in rubles)
Open in Telegram · Free Monitoring Lite kit · All runbook alerts