KubeJobFailed: what to do when it fires

Alert rule

alert: KubeJobFailed
expr: kube_job_status_failed > 0
for: 1m
severity: warning

Alert message: Job {{ $labels.namespace }}/{{ $labels.job_name }} failed

What it means

A Job has Pods that finished with an error (kube_job_status_failed > 0). The task has not completed or is running with retries. For a CronJob the metric remains for as long as the Job object exists (limited by failedJobsHistoryLimit).

Common causes

An error in the task script, an unavailable database/API, insufficient permissions, a timeout (activeDeadlineSeconds), a shortage of resources.

Diagnostics

kubectl -n <ns> get jobs
kubectl -n <ns> describe job <name>
kubectl -n <ns> get pods -l job-name=<name>
kubectl -n <ns> logs -l job-name=<name> --tail=100

How to fix

Fix the cause using the logs, then delete the failed Job (kubectl -n <ns> delete job <name>) so the alert clears, or wait for the next successful CronJob run and the history cleanup.

When it is OK to silence

If the failure is known, understood and no rerun is needed (but then delete the Job).

Related alerts

• KubePvcAlmostFull
• KubeDaemonSetNotReady
• KubePvcCritical
• KubeStatefulSetReplicasMismatch
• KubeNodeNotReady

The full pack of 44 tested rules + Alertmanager → Telegram: 790 ₽ (≈ $9, card payment in rubles)
Open in Telegram · Free Monitoring Lite kit · All runbook alerts