KubeContainerOomKilled: what to do when it fires

Alert rule

alert: KubeContainerOomKilled
expr: kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
and on (namespace, pod, container)
increase(kube_pod_container_status_restarts_total[10m]) > 0
for: 0m
severity: warning

Alert message: Container {{ $labels.container }} was OOM-killed ({{ $labels.namespace }}/{{ $labels.pod }})

What it means

A container was restarted and the reason for its last termination was OOMKilled: the process exceeded its memory limit (resources.limits.memory). The alert is active while the restart counter has increased within 10 minutes.

Common causes

A limit that is too low, a memory leak, peak load, runtime settings (JVM heap, Node.js) above the container limit.

Diagnostics

kubectl -n <ns> describe pod <pod>              # Last State: Terminated, Reason: OOMKilled
kubectl -n <ns> top pod <pod> --containers
kubectl -n <ns> get pod <pod> -o jsonpath='{.spec.containers[*].resources}'
kubectl -n <ns> logs <pod> -c <container> --previous

How to fix

Increase the memory limit, bring the application settings in line with the limit, find the leak. Keep a sensible gap between requests and limits.

When it is OK to silence

For test and batch jobs where OOM is expected.

Related alerts

• KubeDeploymentReplicasMismatch
• KubePodImagePullBackOff
• KubeStatefulSetReplicasMismatch
• KubePodNotReady
• KubeDaemonSetNotReady

The full pack of 44 tested rules + Alertmanager → Telegram: 790 ₽ (≈ $9, card payment in rubles)
Open in Telegram · Free Monitoring Lite kit · All runbook alerts