Русская версия

KubeHpaMaxedOut: what to do when it fires

Alert rule

alert: KubeHpaMaxedOut
expr: kube_horizontalpodautoscaler_status_current_replicas
  == kube_horizontalpodautoscaler_spec_max_replicas
for: 15m
severity: warning

Alert message: HPA {{ $labels.namespace }}/{{ $labels.horizontalpodautoscaler }} is at its maximum

What it means

An HPA has been running at its maximum number of replicas (maxReplicas) for 15 minutes. There is nowhere further to scale automatically, although the load may keep growing.

Common causes

A genuine traffic increase, a maxReplicas limit that is too small, an inefficient application, a poorly chosen scaling metric.

Diagnostics

kubectl -n <ns> get hpa <name>
kubectl -n <ns> describe hpa <name>              # Metrics, Conditions, Events
kubectl -n <ns> top pods
kubectl top nodes

How to fix

If the load is legitimate, raise maxReplicas (making sure the cluster has enough resources) or optimise the application. If the metric is wrong, adjust the target value.

When it is OK to silence

If running at the maximum is the normal mode during peak hours and you are fine with that.

Related alerts

The full pack of 44 tested rules + Alertmanager → Telegram: 790 ₽ (≈ $9, card payment in rubles)
Open in Telegram · Free Monitoring Lite kit · All runbook alerts