DockerContainerMemoryNearLimit: what to do when it fires

Alert rule

alert: DockerContainerMemoryNearLimit
expr: container_memory_working_set_bytes{name!=""}
  / (container_spec_memory_limit_bytes{name!=""} > 0) * 100 > 90
for: 10m
severity: warning

Alert message: Container {{ $labels.name }} is near its memory limit ({{ $labels.instance }})

What it means

The container's memory working set (working_set) has exceeded 90% of its configured limit for 10 minutes. The rule does not fire for containers without a memory limit.

Common causes

The limit was chosen with too little headroom, a memory leak, growth of load or cache.

Diagnostics

docker stats --no-stream <name>
docker inspect <name> --format '{{.HostConfig.Memory}}'
docker exec <name> sh -c 'cat /sys/fs/cgroup/memory.current /sys/fs/cgroup/memory.max 2>/dev/null'
docker logs --tail 100 <name>

The cgroup files depend on the version (v1/v2), so the command may return nothing.

How to fix

Increase the limit, find the leak (heap dump, profiling), limit the application cache (maxmemory in Redis, -Xmx for a JVM).

When it is OK to silence

If the application deliberately uses all the allocated memory for cache and you have verified this.

Related alerts

• DockerContainerOomKilled
• DockerContainerCpuThrottled
• DockerContainerRestartLoop
• DockerContainerGone
• DockerCadvisorDown

The full pack of 44 tested rules + Alertmanager → Telegram: 790 ₽ (≈ $9, card payment in rubles)
Open in Telegram · Free Monitoring Lite kit · All runbook alerts