NodeNetworkErrors: what to do when it fires

Alert rule

alert: NodeNetworkErrors
expr: sum by (instance, device) (
  rate(node_network_receive_errs_total{device!~"lo|veth.*"}[5m])
  + rate(node_network_transmit_errs_total{device!~"lo|veth.*"}[5m])
) > 1
for: 10m
severity: warning

Alert message: Errors on network interface {{ $labels.device }} ({{ $labels.instance }})

What it means

A network interface (other than lo and veth*) has had more than 1 receive or transmit error per second for 10 minutes. The errs counter holds driver/link-level errors, not ordinary packet drops.

Common causes

A bad cable or switch port, a duplex/speed mismatch, driver problems or a virtual network problem at the hosting provider, an overloaded interface.

Diagnostics

ip -s link show <interface>
ethtool -S <interface> | grep -i -E 'err|drop|crc|fifo'
ethtool <interface>                    # speed and duplex
dmesg -T | grep -i <interface> | tail

How to fix

Replace the cable/port, align speed/duplex, update the driver or firmware, contact the hosting provider for virtual machines.

When it is OK to silence

For interfaces not used in production (for example service bridges): exclude them by the device label in the expression.

Related alerts

• NodeClockSkew
• NodeIowaitHigh
• NodeClockNotSynced
• NodeLoadHigh
• NodeRebooted

The full pack of 44 tested rules + Alertmanager → Telegram: 790 ₽ (≈ $9, card payment in rubles)
Open in Telegram · Free Monitoring Lite kit · All runbook alerts