This is the free runbook for a pack of 44 Prometheus alerting rules with promtool unit tests, a Kubernetes PrometheusRule and an Alertmanager example. The rules themselves cost 790 ₽ in Telegram · all kits · Русская версия
Every alert in rules/*.yml has a section below with the same name. The runbook_url link in an alert points to the section heading (the alert name in lower case).
Section format: what it means → common causes → diagnostics → how to fix → when it is OK to silence.
General rules:
rules/*.yml instead of ignoring it.Contents: Linux hosts · Docker · Kubernetes · Website checks
What it means. Prometheus has been unable to scrape node-exporter for 3 minutes (up == 0). This is not necessarily a server outage: only the exporter or the network path to it may be down.
Common causes.
- the server is powered off, hung or rebooting;
- a network problem or firewall rule between Prometheus and the host (port 9100);
- the node-exporter service is stopped or crashed;
- the wrong address in scrape_configs.
Diagnostics.
ping -c 3 <host>
curl -sS -m 5 http://<host>:9100/metrics | head
ssh <host> 'systemctl status node_exporter --no-pager'
ssh <host> 'journalctl -u node_exporter -n 50 --no-pager'
ssh <host> 'uptime; dmesg -T | tail -n 30'
The unit name depends on how it was installed (node_exporter, prometheus-node-exporter).
How to fix. If the server is unreachable, deal with the hosting provider or the hypervisor console. If only the exporter is down, start it: systemctl restart node_exporter, and check the firewall (ss -ltnp | grep 9100).
When it is OK to silence. During planned work or when decommissioning the host (better still, remove the target from the Prometheus configuration).
What it means. Less than 15% free space is left on the partition (space available to unprivileged processes is counted). There are no real consequences yet, but the headroom is shrinking.
Common causes. - log growth without rotation; - accumulated Docker images and layers, dumps, backups, package manager caches; - database growth; - a deleted file still held open by a running process (the space is not freed).
Diagnostics.
df -hT
du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -n 15
du -xh --max-depth=1 /var 2>/dev/null | sort -h | tail -n 10
journalctl --disk-usage
docker system df # if Docker is on the host
lsof +L1 | head # deleted but still open files
How to fix. Set up log rotation (logrotate, journalctl --vacuum-size=500M), delete what is not needed (apt clean / dnf clean all, docker image prune), move data to another disk or extend the volume. Restart processes that hold deleted files open.
When it is OK to silence. For partitions that fill up normally and are cleaned by the application itself (a cache, for example), it is better to exclude that mountpoint in the rule expression. A temporary silence is acceptable if a disk extension is already planned.
What it means. Less than 5% free space is left on the partition. Writes may start failing with errors at any moment.
Common causes. The same as for NodeDiskSpaceLow, but the headroom is used up: fast log growth, a runaway application writing to a file, a database overflow.
Diagnostics.
df -hT
du -xh --max-depth=2 / 2>/dev/null | sort -h | tail -n 20
ls -lhS /var/log | head
journalctl --disk-usage
docker system df
How to fix. Free space quickly: truncate the largest log (: > /var/log/<file> rather than rm if a process holds the file open), journalctl --vacuum-size=200M, docker system prune (read carefully what will be removed), delete old backups. Then find the source of the growth and extend the volume.
When it is OK to silence. Not until space has been freed. The exception is a read-only partition or an image that is not expected to have free space (for example a mounted iso/squashfs, if it is not filtered out).
What it means. Based on the linear trend over the last 6 hours (predict_linear), the partition will run out of space within the next 24 hours. The rule fires only if less than 40% is free, to avoid noise on large, mostly empty disks.
Common causes. - abnormal growth of logs or the database; - a running export, backup or data import; - a leak of temporary files.
Diagnostics.
df -h <mountpoint>
du -xh --max-depth=2 <mountpoint> 2>/dev/null | sort -h | tail -n 20
find <mountpoint> -xdev -type f -mmin -60 -size +100M -exec ls -lh {} +
Compare the node_filesystem_avail_bytes graph over the last 6–24 hours: is the growth steady or a sudden jump?
How to fix. Find and stop the source of the growth, add rotation, extend the disk. If the write is a one-off (an import), estimate whether the space will last until the operation finishes.
When it is OK to silence. If the growth is a known one-off (a migration, an import) and there is clearly enough space. In that case set the silence for a few hours, not days.
What it means. Less than 10% of inodes are free on the filesystem. Once they run out, no new file can be created even if there is disk space (the No space left on device error while df -h shows plenty of room).
Common causes. Millions of small files: PHP sessions, caches, the mail queue, temporary files, node_modules directories, old kernels in /usr/src.
Diagnostics.
df -i
for d in /var /tmp /home /usr; do echo "$d: $(find $d -xdev 2>/dev/null | wc -l)"; done
find /var -xdev -type d -size +1M 2>/dev/null | head # directories with huge file counts
du -x --inodes --max-depth=2 /var 2>/dev/null | sort -n | tail -n 15
The --inodes option of du is available in GNU coreutils.
How to fix. Delete the unneeded small files in the directory you found (find <dir> -type f -mtime +7 -delete, after checking the list without -delete) and set up cache/session cleanup. On ext4 the number of inodes is fixed when the filesystem is created and cannot be changed afterwards; you can only recreate the filesystem or move the data.
When it is OK to silence. For filesystems with a dynamic number of inodes (btrfs, xfs) that produce false positives, exclude them from the expression.
What it means. Less than 10% of RAM is available (based on MemAvailable, i.e. counting cache the kernel can give back). The system still works, but the headroom is small.
Common causes. A memory leak in an application, too many workers, growth of an application cache (Redis, JVM), a heavy task being run.
Diagnostics.
free -h
ps aux --sort=-%mem | head -n 10
smem -tk 2>/dev/null | tail -n 15 # if smem is installed
docker stats --no-stream # if Docker is present
dmesg -T | grep -i -E 'oom|killed process' | tail
How to fix. Find the consuming process, restart it if it leaks, reduce the number of workers or the cache size, add RAM. For a JVM, check -Xmx.
When it is OK to silence. On servers where memory is deliberately used for cache (databases with a large shared_buffers and the like), but then it is better to raise the threshold in the rule to suit your profile.
What it means. Less than 4% of memory is available: the kernel will soon start killing processes (OOM killer), and the system will go into swap and slow down.
Common causes. The same as in NodeMemoryLow, but more acute: a sharp leak, an unexpected load spike.
Diagnostics.
free -h
ps aux --sort=-%mem | head -n 10
dmesg -T | grep -i -E 'out of memory|killed process' | tail
journalctl -k --since "1 hour ago" | grep -i oom
vmstat 1 5
How to fix. Stop or restart the main consumer, take load off temporarily, add memory or swap as a temporary measure. Then look for the root cause.
When it is OK to silence. Not until the system has stabilised.
What it means. More than 80% of swap is in use (provided swap is configured). The system is pushing memory out to disk and runs slowly.
Common causes. A shortage of RAM; a single leaking process; a vm.swappiness that is too high combined with long-inactive pages (in that case used swap may be harmless, but 80% is a reason to take a look).
Diagnostics.
free -h
swapon --show
for f in /proc/*/status; do awk '/^Name|^VmSwap/{printf $2" "}END{print ""}' "$f"; done 2>/dev/null | sort -k2 -n -r | head
vmstat 1 5 # si/so columns: active swapping
sysctl vm.swappiness
How to fix. Free memory (see NodeMemoryLow). If processes have merely "lain" in swap with no active swapping (si/so near zero), you can reset swap: swapoff -a && swapon -a — only when RAM is enough for the entire swap contents.
When it is OK to silence. If si/so are zero and there is enough memory, but swap is full of long-inactive pages.
What it means. Average CPU load across all cores has been above 90% for 15 minutes.
Common causes. A traffic peak, a runaway process, a heavy backup/compression/build, a miner after a compromise, too few cores for the load.
Diagnostics.
top -b -n 1 | head -n 15
ps aux --sort=-%cpu | head -n 10
pidstat 1 5 # sysstat package
docker stats --no-stream
How to fix. Stop the culprit or limit it (nice, cpulimit, cgroup/Docker limits), optimise queries, add cores/nodes. Check an unknown process with high load for signs of compromise.
When it is OK to silence. For hosts meant for batch computation, where 100% CPU is normal: exclude them or raise the threshold.
What it means. The 5-minute load average has been more than 2 times the number of cores for 15 minutes. Load includes both processes waiting for CPU and processes in uninterruptible sleep (usually waiting for disk/NFS).
Common causes. CPU overload, a slow disk or network (NFS), processes in the D state, too many parallel tasks.
Diagnostics.
uptime
nproc
top -b -n 1 | head -n 15
ps -eo state,pid,cmd | awk '$1 ~ /D/' | head # processes in D state
iostat -x 1 3
How to fix. If %CPU is high, act as in NodeCpuHigh; if there are many D-state processes, look for a disk/NFS problem, as in NodeIowaitHigh.
When it is OK to silence. If it is a known batch workload outside working hours.
What it means. The CPU spends more than 30% of its time waiting for disk operations to complete. The disk (or network storage) is the bottleneck.
Common causes. Heavy database queries, a backup/copy, a degraded disk or RAID, overloaded cloud storage, active swapping, an unavailable NFS.
Diagnostics.
iostat -x 1 5 # sysstat package: watch %util and await
iotop -oPa # who is writing/reading
vmstat 1 5 # wa, si, so columns
dmesg -T | grep -i -E 'i/o error|ata|nvme|blk_update' | tail
smartctl -H /dev/sda # smartmontools package
How to fix. Stop or limit the culprit (ionice -c3), spread the load across disks, optimise queries/indexes, replace a failing disk, increase the IOPS of a cloud volume.
When it is OK to silence. During a scheduled backup or a RAID rebuild.
What it means. A network interface (other than lo and veth*) has had more than 1 receive or transmit error per second for 10 minutes. The errs counter holds driver/link-level errors, not ordinary packet drops.
Common causes. A bad cable or switch port, a duplex/speed mismatch, driver problems or a virtual network problem at the hosting provider, an overloaded interface.
Diagnostics.
ip -s link show <interface>
ethtool -S <interface> | grep -i -E 'err|drop|crc|fifo'
ethtool <interface> # speed and duplex
dmesg -T | grep -i <interface> | tail
How to fix. Replace the cable/port, align speed/duplex, update the driver or firmware, contact the hosting provider for virtual machines.
When it is OK to silence. For interfaces not used in production (for example service bridges): exclude them by the device label in the expression.
What it means. The system clock differs from the reference time by more than 0.05 seconds (kernel metric node_timex_offset_seconds) for more than 10 minutes.
Common causes. The time synchronisation service is not running, the NTP servers are unreachable (UDP/123 blocked), a virtual machine after a pause or migration.
Diagnostics.
timedatectl status
chronyc tracking && chronyc sources -v # if chrony is used
systemctl status chronyd systemd-timesyncd --no-pager
journalctl -u chronyd -u systemd-timesyncd -n 50 --no-pager
How to fix. Start and enable the synchronisation service (systemctl enable --now chronyd or systemd-timesyncd), check that the NTP servers are reachable, and use chronyc makestep to force an adjustment.
When it is OK to silence. On isolated machines without NTP where time accuracy does not matter (it is better then to disable the rule for those hosts).
What it means. The kernel reports that the clock is not synchronised (node_timex_sync_status == 0) for more than 15 minutes. The alert can also appear with a small offset, before NodeClockSkew fires.
Common causes. The same as for NodeClockSkew; also the time service has only just been started and has not yet reached synchronisation, or is not installed at all.
Diagnostics.
timedatectl status # the "System clock synchronized" line
chronyc tracking
journalctl -u chronyd -u systemd-timesyncd --since "1 hour ago" --no-pager
How to fix. Install and enable an NTP client, open outbound UDP 123 or specify reachable servers in the configuration (/etc/chrony.conf or /etc/systemd/timesyncd.conf).
When it is OK to silence. For machines deliberately running without synchronisation (lab setups).
What it means. Less than 10 minutes have passed since the host last booted. This is an informational alert: it exists so that you notice unexpected reboots that would otherwise go unnoticed (the host is visible again and NodeDown has no time to fire).
Common causes. A planned kernel update, a hardware or hypervisor failure, a kernel panic, an OOM with reset, a power outage, a watchdog.
Diagnostics.
uptime -s
last -x reboot shutdown | head
journalctl --list-boots | tail -n 5
journalctl -b -1 -n 100 --no-pager # end of the previous boot
systemctl --failed
How to fix. If the reboot was planned, nothing. If not, find the cause in the journal of the previous boot and make sure the services came back up (systemctl --failed).
When it is OK to silence. During scheduled updates that involve a reboot.
What it means. Container metrics are not being collected from the host: up == 0 for the cAdvisor job for 3 minutes. While this alert is active, the other Docker alerts for this host do not work.
Common causes. The cAdvisor container is stopped, port 8080 is unreachable, the host is down, the wrong address in scrape_configs.
Diagnostics.
docker ps -a --filter name=cadvisor
docker logs --tail 50 cadvisor
curl -sS -m 5 http://<host>:8080/metrics | head
ss -ltnp | grep 8080
How to fix. Start the container (docker start cadvisor / docker compose up -d), check the permissions of the mounts (/, /var/run, /sys, /var/lib/docker), fix the address or the firewall.
When it is OK to silence. During host maintenance or a cAdvisor upgrade.
What it means. The container was present in the metrics 5 minutes ago, and cAdvisor no longer sees it. The alert stays active for about 5 minutes after the container disappears and then clears on its own, so you need to react quickly.
Common causes. The container was stopped (docker stop, exit), removed during an update (docker compose up recreated it with a new ID), or crashed and was not restarted (no restart: always).
Diagnostics.
docker ps -a --filter name=<name>
docker inspect <name> --format '{{.State.Status}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}}'
docker logs --tail 100 <name>
journalctl -u docker -n 50 --no-pager
How to fix. Start the container (docker start <name> or docker compose up -d), use the logs to understand why it stopped, and if needed set the restart: unless-stopped policy.
When it is OK to silence. When a container is removed or renamed on purpose. For containers that by design live for a short time (jobs), exclude them by the name label.
What it means. In the last 15 minutes the container has been started again 3 or more times (judging by changes in container_start_time_seconds). The process crashes right after starting and Docker restarts it.
Common causes. An error in the configuration or environment variables, an unavailable dependency (database, network), out of memory (OOM), a wrong image or command, a missing file or volume.
Diagnostics.
docker ps -a --filter name=<name>
docker inspect <name> --format '{{.State.ExitCode}} {{.State.OOMKilled}} {{.RestartCount}}'
docker logs --tail 200 <name>
docker events --since 15m --filter container=<name>
docker stats --no-stream <name>
How to fix. Fix the cause using the last lines of the log before the crash. Exit code 137 usually means SIGKILL (including OOM), 1 means an application error. If the cause is memory, see DockerContainerOomKilled.
When it is OK to silence. When debugging on a test setup. Not in production.
What it means. The container is hitting its CPU limit: more than 25% of CFS scheduler periods were throttled over 15 minutes. The application slows down even though the host may have free CPU.
Common causes. A --cpus / cpu_quota / deploy.resources.limits.cpus value that is too low, increased load, a multithreaded application for which a fractional limit is too small.
Diagnostics.
docker stats --no-stream <name>
docker inspect <name> --format 'CPUs={{.HostConfig.NanoCpus}} quota={{.HostConfig.CpuQuota}} period={{.HostConfig.CpuPeriod}}'
docker top <name>
NanoCpus is in billionths of a core (1000000000 is one core); 0 means no limit.
How to fix. Raise the CPU limit, or optimise the application, or add replicas. If the limit exists only "for the sake of order", consider removing it and using cpu_shares.
When it is OK to silence. For background containers that are not hurt by throttling (low-priority jobs).
What it means. The container's memory working set (working_set) has exceeded 90% of its configured limit for 10 minutes. The rule does not fire for containers without a memory limit.
Common causes. The limit was chosen with too little headroom, a memory leak, growth of load or cache.
Diagnostics.
docker stats --no-stream <name>
docker inspect <name> --format '{{.HostConfig.Memory}}'
docker exec <name> sh -c 'cat /sys/fs/cgroup/memory.current /sys/fs/cgroup/memory.max 2>/dev/null'
docker logs --tail 100 <name>
The cgroup files depend on the version (v1/v2), so the command may return nothing.
How to fix. Increase the limit, find the leak (heap dump, profiling), limit the application cache (maxmemory in Redis, -Xmx for a JVM).
When it is OK to silence. If the application deliberately uses all the allocated memory for cache and you have verified this.
What it means. The kernel killed the container's process for exceeding its memory limit: the container_oom_events_total counter increased in the last 10 minutes.
Common causes. A limit that is too small, a memory leak, a load spike, wrong runtime settings (heap larger than the container limit).
Diagnostics.
docker inspect <name> --format 'OOMKilled={{.State.OOMKilled}} exit={{.State.ExitCode}} limit={{.HostConfig.Memory}}'
docker logs --tail 100 <name>
dmesg -T | grep -i -E 'oom|killed process' | tail
docker stats --no-stream
How to fix. Raise the memory limit, bring the application settings in line with the limit, fix the leak. Watch for restarts — DockerContainerRestartLoop.
When it is OK to silence. It must not be left unattended: an OOM means a lost process. Acceptable on test setups.
The commands below use <ns> for the namespace; add --context/-n to suit your cluster.
What it means. Prometheus has been unable to scrape kube-state-metrics for 5 minutes. All alerts about Pods, Deployments, StatefulSets, DaemonSets, Jobs, HPAs and nodes are blind during this period.
Common causes. The kube-state-metrics Pod crashed or is being rescheduled, a network policy blocks the scrape, the Service/ServiceMonitor name changed, the cluster ran out of resources.
Diagnostics.
kubectl -n <ns> get pods -l app.kubernetes.io/name=kube-state-metrics
kubectl -n <ns> describe pod -l app.kubernetes.io/name=kube-state-metrics
kubectl -n <ns> logs -l app.kubernetes.io/name=kube-state-metrics --tail=100
kubectl -n <ns> get servicemonitor,endpoints | grep -i state-metrics
How to fix. Fix the cause of the Pod failure (OOM, RBAC permissions, image), restore the ServiceMonitor/Service to a working state, raise the limits if memory is short.
When it is OK to silence. During a kube-prometheus-stack upgrade.
What it means. Prometheus has not received kubelet metrics from a node for 5 minutes. The node may have failed, or kubelet may have stopped, or the network to port 10250 may be broken.
Common causes. The node is down, kubelet crashed, the kubelet certificate expired, a network block, the node was removed from the cluster but is still in the targets.
Diagnostics.
kubectl get nodes -o wide
kubectl describe node <node>
ssh <node> 'systemctl status kubelet --no-pager; journalctl -u kubelet -n 100 --no-pager'
How to fix. Restart kubelet (systemctl restart kubelet), restore the node, check the certificates. If the node was removed on purpose, remove it from the scrape targets.
When it is OK to silence. When taking a node out of the cluster (drain).
What it means. A container has been in CrashLoopBackOff for 10 minutes: the process starts, crashes, and Kubernetes restarts it with a growing delay.
Common causes. An application error at startup, a wrong config/secret, an unavailable database, out of memory (OOMKilled), a wrong command, a failing liveness probe.
Diagnostics.
kubectl -n <ns> get pod <pod>
kubectl -n <ns> describe pod <pod> # Events and Last State sections
kubectl -n <ns> logs <pod> -c <container> --previous
kubectl -n <ns> get events --sort-by=.lastTimestamp | tail -n 20
The --previous flag shows the log of the previous (crashed) run.
How to fix. Fix the cause shown in the log: the configuration, the secret, the dependency. If the liveness probe is failing, increase initialDelaySeconds / failureThreshold. After fixing: kubectl -n <ns> rollout restart deployment/<name>.
When it is OK to silence. For a known broken Pod in a test namespace. Not for production.
What it means. A Pod has been in the Pending or Unknown phase for 15 minutes: Kubernetes cannot start it on a node or has lost contact with the node.
Common causes. Not enough resources (CPU/memory) on the nodes, nodeSelector/affinity/taints that cannot be satisfied, a PVC that does not bind, an unavailable node, namespace resource quotas.
Diagnostics.
kubectl -n <ns> describe pod <pod> # Events section: FailedScheduling and the reason
kubectl get nodes
kubectl top nodes
kubectl -n <ns> get pvc
kubectl -n <ns> describe resourcequota
How to fix. Add resources/nodes, relax the scheduling constraints, fix the PVC/StorageClass, bring the node back. If the requests were badly chosen, reduce them.
When it is OK to silence. If the Pod is waiting on purpose (for example for a GPU node that does not exist yet) and this is known.
What it means. A container has been unable to pull its image from the registry (ErrImagePull / ImagePullBackOff) for 10 minutes.
Common causes. A typo in the image name or tag, the image is not in the registry, no permissions (the imagePullSecret is missing or stale), the registry is unavailable or pull limits apply, no network on the node.
Diagnostics.
kubectl -n <ns> describe pod <pod> # error message in Events
kubectl -n <ns> get pod <pod> -o jsonpath='{.spec.containers[*].image}'
kubectl -n <ns> get secret # imagePullSecrets
ssh <node> 'crictl pull <image>'
How to fix. Fix the name/tag, recreate the registry access secret (kubectl create secret docker-registry ...) and reference it in imagePullSecrets, check that the registry is reachable from the node.
When it is OK to silence. Not until the application is running.
What it means. A container was restarted and the reason for its last termination was OOMKilled: the process exceeded its memory limit (resources.limits.memory). The alert is active while the restart counter has increased within 10 minutes.
Common causes. A limit that is too low, a memory leak, peak load, runtime settings (JVM heap, Node.js) above the container limit.
Diagnostics.
kubectl -n <ns> describe pod <pod> # Last State: Terminated, Reason: OOMKilled
kubectl -n <ns> top pod <pod> --containers
kubectl -n <ns> get pod <pod> -o jsonpath='{.spec.containers[*].resources}'
kubectl -n <ns> logs <pod> -c <container> --previous
How to fix. Increase the memory limit, bring the application settings in line with the limit, find the leak. Keep a sensible gap between requests and limits.
When it is OK to silence. For test and batch jobs where OOM is expected.
What it means. The number of available replicas of a Deployment has differed from the desired number for 15 minutes. Usually fewer are available; a surplus can occur during scaling.
Common causes. A failed rollout in progress (the new version does not start), a Pod failing its readiness probe, a shortage of resources, a removed node.
Diagnostics.
kubectl -n <ns> get deploy <name>
kubectl -n <ns> rollout status deployment/<name>
kubectl -n <ns> describe deploy <name>
kubectl -n <ns> get pods -l <selector-labels> -o wide
kubectl -n <ns> describe pod <problem-pod>
How to fix. Fix the cause of the unhealthy Pods (see KubePodCrashLooping, KubePodNotReady). If the new version is broken, roll back: kubectl -n <ns> rollout undo deployment/<name>.
When it is OK to silence. When replicas have been deliberately set to zero (replicas: 0) for maintenance.
What it means. The number of ready replicas of a StatefulSet has differed from the desired number for 15 minutes. A StatefulSet brings Pods up in order, so one problematic replica blocks the rest.
Common causes. A Pod is not ready (readiness probe), a volume does not bind, a shortage of resources, an unavailable node, a database cluster failing to form a quorum.
Diagnostics.
kubectl -n <ns> get statefulset <name>
kubectl -n <ns> get pods -l <selector-labels> -o wide
kubectl -n <ns> describe pod <name>-<index>
kubectl -n <ns> get pvc
kubectl -n <ns> logs <name>-<index> --previous
How to fix. Fix the cause in the problematic Pod; check the PVC and StorageClass; for databases, verify the replication state before deleting a Pod by hand. Do not delete StatefulSet Pods that hold data unless necessary.
When it is OK to silence. During planned scaling or a one-replica-at-a-time update.
What it means. A DaemonSet has had fewer ready Pods than it should (one per eligible node) for 15 minutes.
Common causes. A node is unavailable or NotReady, a taint on the node without a toleration, a shortage of resources on the node, a Pod crashing on a specific node (kernel/OS specifics).
Diagnostics.
kubectl -n <ns> get ds <name>
kubectl -n <ns> get pods -l <selector-labels> -o wide
kubectl -n <ns> describe ds <name>
kubectl -n <ns> logs <pod-on-problem-node> --previous
kubectl describe node <node> | grep -i -A5 taints
How to fix. Fix the cause on the specific nodes: bring the node back, add a toleration, free resources. If a node is being retired from the cluster, remove it with kubectl delete node.
When it is OK to silence. During node drain/upgrade.
What it means. A Job has Pods that finished with an error (kube_job_status_failed > 0). The task has not completed or is running with retries. For a CronJob the metric remains for as long as the Job object exists (limited by failedJobsHistoryLimit).
Common causes. An error in the task script, an unavailable database/API, insufficient permissions, a timeout (activeDeadlineSeconds), a shortage of resources.
Diagnostics.
kubectl -n <ns> get jobs
kubectl -n <ns> describe job <name>
kubectl -n <ns> get pods -l job-name=<name>
kubectl -n <ns> logs -l job-name=<name> --tail=100
How to fix. Fix the cause using the logs, then delete the failed Job (kubectl -n <ns> delete job <name>) so the alert clears, or wait for the next successful CronJob run and the history cleanup.
When it is OK to silence. If the failure is known, understood and no rerun is needed (but then delete the Job).
What it means. Less than 15% is free on a PersistentVolumeClaim volume. The metrics come from kubelet and are not available for all volume types.
Common causes. Database or log growth, no rotation, accumulated dumps and temporary files.
Diagnostics.
kubectl -n <ns> get pvc <name>
kubectl -n <ns> get pods -o wide | grep <pod-using-the-volume>
kubectl -n <ns> exec <pod> -- df -h <mount-path>
kubectl -n <ns> exec <pod> -- du -xh --max-depth=2 <mount-path>
kubectl get storageclass
How to fix. Clean up what is not needed or expand the PVC (if the StorageClass supports allowVolumeExpansion: true): kubectl -n <ns> patch pvc <name> -p '{"spec":{"resources":{"requests":{"storage":"<new-size>"}}}}'. The size cannot be reduced.
When it is OK to silence. If the growth is a one-off and an expansion is planned.
What it means. Less than 5% is free on a PVC volume. The application is about to be unable to write data.
Common causes. The same as for KubePvcAlmostFull, but the headroom is used up.
Diagnostics.
kubectl -n <ns> exec <pod> -- df -h <mount-path>
kubectl -n <ns> exec <pod> -- du -xh --max-depth=2 <mount-path>
kubectl -n <ns> get pvc <name> -o yaml | grep -i -E 'storageClassName|storage:'
kubectl get storageclass <name> -o jsonpath='{.allowVolumeExpansion}'
How to fix. Urgently expand the PVC or free space. For databases, do not delete data files by hand — use the database's own mechanisms (WAL/binlog cleanup, VACUUM and the like).
When it is OK to silence. Not until space has been freed.
What it means. A node has not reported the Ready state for 5 minutes. Kubernetes stops scheduling Pods on it and will begin evicting them after some time.
Common causes. The node is down or hung, kubelet is stopped, the network between the node and the API server, a shortage of resources on the node, a container runtime error.
Diagnostics.
kubectl get nodes -o wide
kubectl describe node <node> # Conditions and Events
kubectl get pods -A -o wide --field-selector spec.nodeName=<node>
ssh <node> 'systemctl status kubelet containerd --no-pager; journalctl -u kubelet -n 100 --no-pager'
ssh <node> 'df -h; free -h; uptime'
How to fix. Restart kubelet/the container runtime, free disk and memory, reboot or replace the node. If necessary, run kubectl drain <node> --ignore-daemonsets --delete-emptydir-data.
When it is OK to silence. During planned work on the node.
What it means. A node has reported one of the resource-shortage conditions: MemoryPressure, DiskPressure or PIDPressure (the name is in the condition label), for 5 minutes. Kubelet may start evicting Pods.
Common causes. A shortage of memory (many Pods without limits), a full disk (images, logs, emptyDir), too many processes.
Diagnostics.
kubectl describe node <node> # Conditions
kubectl top node <node>
ssh <node> 'df -h /var/lib/kubelet /var/lib/containerd; free -h'
ssh <node> 'crictl images | head; crictl ps | wc -l'
kubectl get pods -A --field-selector spec.nodeName=<node> | grep -i evicted
How to fix. For DiskPressure, clean up images and logs (crictl rmi --prune) and expand the disk. For MemoryPressure, set requests/limits and redistribute Pods. For PIDPressure, find the process that spawns threads.
When it is OK to silence. A short-lived peak with a known cause; not if it recurs.
What it means. An HPA has been running at its maximum number of replicas (maxReplicas) for 15 minutes. There is nowhere further to scale automatically, although the load may keep growing.
Common causes. A genuine traffic increase, a maxReplicas limit that is too small, an inefficient application, a poorly chosen scaling metric.
Diagnostics.
kubectl -n <ns> get hpa <name>
kubectl -n <ns> describe hpa <name> # Metrics, Conditions, Events
kubectl -n <ns> top pods
kubectl top nodes
How to fix. If the load is legitimate, raise maxReplicas (making sure the cluster has enough resources) or optimise the application. If the metric is wrong, adjust the target value.
When it is OK to silence. If running at the maximum is the normal mode during peak hours and you are fine with that.
What it means. The client certificates used to access kube-apiserver expire in less than 7 days (and not less than 1 day). The apiserver_client_certificate_expiration_seconds metric is not available in all distributions; the rule works only if it is present. The value is an estimate based on the lowest 1% of observed certificates.
Common causes. The certificates were not renewed automatically (kubeadm did not renew them, the certificates of kubeconfig users, service accounts or components expired).
Diagnostics.
kubeadm certs check-expiration # for kubeadm clusters, on the control plane
openssl x509 -noout -enddate -in /etc/kubernetes/pki/apiserver.crt
kubectl config view --raw -o jsonpath='{.users[*].user.client-certificate-data}' | base64 -d | openssl x509 -noout -enddate
How to fix. For kubeadm: kubeadm certs renew all on each control plane node, then restart the control plane components. In managed clusters (EKS/GKE/AKS and the like) the provider manages the certificates — see its documentation.
When it is OK to silence. If you already know about the expiry and the renewal is scheduled before the critical deadline.
What it means. The client certificates for accessing kube-apiserver expire in less than a day (or have already expired). After expiry, components and users may lose access to the cluster.
Common causes. The same as for KubeApiCertExpiresSoon: the renewal was missed.
Diagnostics.
kubeadm certs check-expiration
openssl x509 -noout -enddate -in /etc/kubernetes/pki/apiserver-kubelet-client.crt
ls -l /etc/kubernetes/pki /var/lib/kubelet/pki
How to fix. Renew the certificates immediately (kubeadm certs renew all and restart the control plane components), update administrators' kubeconfigs. Back up /etc/kubernetes before the operation.
When it is OK to silence. Not until renewed.
What it means. Prometheus has received no response from blackbox-exporter when probing a target (up == 0) for 3 minutes. Website checks are not being performed, so the other blackbox alerts are silent.
Common causes. The blackbox-exporter service is stopped, the wrong exporter address in relabel_configs (the replacement parameter), a firewall, an error in the module configuration.
Diagnostics.
systemctl status prometheus-blackbox-exporter --no-pager # or your unit name
curl -sS -m 10 'http://<exporter>:9115/probe?module=http_2xx&target=https://example.com' | head -n 20
journalctl -u prometheus-blackbox-exporter -n 50 --no-pager
For Docker/Kubernetes, use docker logs / kubectl logs of the corresponding Pod.
How to fix. Start the exporter, fix the address and port (9115 by default) in the Prometheus configuration, check the syntax of blackbox.yml.
When it is OK to silence. During an exporter upgrade.
What it means. The probe of a target has been failing (probe_success == 0) for 3 minutes in a row. The site or service is unavailable or responds differently from what the probe module expects.
Common causes. The service is down, DNS problems, a closed port or firewall, an expired/broken TLS certificate, a timeout, a wrong response code, the expected response text did not match.
Diagnostics.
curl -sS -v -m 10 -o /dev/null https://example.com
dig +short example.com
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null | openssl x509 -noout -dates
curl -sS -m 10 'http://<exporter>:9115/probe?module=http_2xx&target=https://example.com&debug=true'
The debug=true parameter makes blackbox-exporter return a detailed probe log.
How to fix. Based on the diagnostics: bring the service up, fix DNS or the certificate, open the port. If there are false positives due to a timeout, increase timeout in the blackbox module.
When it is OK to silence. During planned work or a site migration.
What it means. The average probe duration over 5 minutes has exceeded 2 seconds for 10 minutes, while the probe succeeds. The site responds, but slowly.
Common causes. Load on the server or database, slow external APIs, network or DNS problems, caching errors, large pages.
Diagnostics.
curl -sS -o /dev/null -w 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n' https://example.com
Then look at the state of the server itself: NodeCpuHigh, NodeIowaitHigh, and the application and database logs.
How to fix. Depending on which phase is slow in curl: DNS, connection, TLS or waiting for the application's response. Optimise queries, add a cache, add resources.
When it is OK to silence. If the 2 s threshold is too strict for this service, raise it in the rule. A silence is acceptable for the duration of a known load.
What it means. The final HTTP status of the response (after redirects) has been outside the 2xx range for 5 minutes. A zero status (no connection was made) is handled by the BlackboxProbeFailed alert.
Common causes. 5xx — an application or proxy error; 4xx — routes or authorisation changed, or the wrong URL is being probed; 3xx at the end of the chain — a redirect loop or a redirect to a page outside 2xx.
Diagnostics.
curl -sS -I -L -m 10 https://example.com
curl -sS -m 10 -o /dev/null -w '%{http_code} %{redirect_url}\n' https://example.com
Look at the web server and application logs for that time (journalctl -u nginx -n 100 --no-pager, docker logs --tail 100 <name>).
How to fix. Fix the application/configuration error. If the service deliberately returns a non-2xx status (for example 401 for a protected page), list the allowed codes in valid_status_codes of the blackbox module.
When it is OK to silence. If the address deliberately responds with non-2xx, it is better to fix the probe module than to silence the alert.
What it means. Less than 14 days but not less than 3 remain until the certificate expires (for 1 hour in a row). The value comes from probe_ssl_earliest_cert_expiry — the earliest expiry in the whole chain.
Common causes. Automatic renewal (ACME/certbot, cert-manager) did not work, the domain validation path changed, renewal is not configured at all, access to the DNS API was revoked.
Diagnostics.
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | openssl x509 -noout -subject -issuer -dates
certbot certificates # if certbot is used
systemctl list-timers | grep -i certbot
journalctl -u certbot -n 50 --no-pager
kubectl get certificate -A # if cert-manager is used
How to fix. Run the renewal manually (certbot renew, reissue via cert-manager), then reload the web server (systemctl reload nginx) so that it picks up the new file. Fix the reason the automation did not work.
When it is OK to silence. For certificates you are deliberately not renewing (the domain is being decommissioned).
What it means. Less than 3 days remain until the certificate expires, or it has already expired. Users will soon see a browser error, and API clients will start refusing the connection.
Common causes. The same as for BlackboxSslExpiresSoon, but the renewal was not done in time. It is also possible that the web server did not reload the renewed certificate.
Diagnostics.
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | openssl x509 -noout -enddate
ls -l /etc/letsencrypt/live/<domain>/ 2>/dev/null
nginx -t && systemctl status nginx --no-pager
If the file on disk is new but the server serves the old one, the server has not been reloaded.
How to fix. Renew or issue the certificate manually right away, reload the web server/load balancer, and verify the result with openssl s_client.
When it is OK to silence. Not until the certificate is renewed.