KubeStatefulSetReplicasMismatch: what to do when it fires

Alert rule

alert: KubeStatefulSetReplicasMismatch
expr: kube_statefulset_replicas
  != kube_statefulset_status_replicas_ready
for: 15m
severity: warning

Alert message: StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }}: not all replicas are ready

What it means

The number of ready replicas of a StatefulSet has differed from the desired number for 15 minutes. A StatefulSet brings Pods up in order, so one problematic replica blocks the rest.

Common causes

A Pod is not ready (readiness probe), a volume does not bind, a shortage of resources, an unavailable node, a database cluster failing to form a quorum.

Diagnostics

kubectl -n <ns> get statefulset <name>
kubectl -n <ns> get pods -l <selector-labels> -o wide
kubectl -n <ns> describe pod <name>-<index>
kubectl -n <ns> get pvc
kubectl -n <ns> logs <name>-<index> --previous

How to fix

Fix the cause in the problematic Pod; check the PVC and StorageClass; for databases, verify the replication state before deleting a Pod by hand. Do not delete StatefulSet Pods that hold data unless necessary.

When it is OK to silence

During planned scaling or a one-replica-at-a-time update.

Related alerts

The full pack of 44 tested rules + Alertmanager → Telegram: 790 ₽ (≈ $9, card payment in rubles)
Open in Telegram · Free Monitoring Lite kit · All runbook alerts