Solution

Triage

1. Check node conditions:

   kubectl describe node node-7.internal | grep -A 10 Conditions
   

2. List evicted pods:

   kubectl get pods --all-namespaces --field-selector status.phase=Failed,status.reason=Evicted
   

3. Check resource allocation on the node:

   kubectl describe node node-7.internal | grep -A 20 "Allocated resources"
   

4. Identify top memory consumers:

   kubectl top pods --all-namespaces --sort-by=memory | head -20
   

Root Cause

The kubelet on node-7 is triggering evictions because actual memory usage crosses the eviction threshold (default: memory.available < 100Mi). Multiple pods are deployed without memory requests or limits, meaning:

1. The scheduler treats them as BestEffort QoS and cannot account for their real usage when making placement decisions.
2. Pods get scheduled to the node because the scheduler sees "available" capacity based on requests, not actual usage.
3. When real usage exceeds node capacity, the kubelet evicts pods starting with BestEffort, then Burstable pods exceeding their requests.
4. Evicted pods are rescheduled and may land on the same node, repeating the cycle.

Fix

Immediate (stop the eviction storm):

1. Cordon the node to prevent new scheduling:

   kubectl cordon node-7.internal
   

2. Identify and delete evicted pod shells (they consume etcd storage):

   kubectl get pods --all-namespaces --field-selector status.phase=Failed,status.reason=Evicted -o json | kubectl delete -f -
   

3. Manually reschedule heavy workloads to other nodes if needed.

Permanent fix:

1. Add memory requests and limits to all deployments:

   resources:
     requests:
       memory: "256Mi"
     limits:
       memory: "512Mi"
   

2. Create a LimitRange in each namespace to enforce defaults:

   apiVersion: v1
   kind: LimitRange
   metadata:
     name: default-mem-limit
   spec:
     limits:
     - default:
         memory: 512Mi
       defaultRequest:
         memory: 256Mi
       type: Container
   

3. Set ResourceQuotas per namespace to prevent overcommitment.
4. Uncordon the node once resource requests are in place:

   kubectl uncordon node-7.internal
   

Rollback / Safety

• Do not uncordon until workloads have proper resource specs.
• Monitor node conditions after uncordoning to confirm stability.
• Review cluster-wide resource utilization to determine if more nodes are needed.

Common Traps

• Deleting evicted pods without fixing the cause. They will just be recreated and evicted again.
• Setting requests too low. If requests do not reflect actual usage, overcommitment continues.
• Ignoring disk pressure. The same eviction mechanism triggers for ephemeral storage and inodes.
• Not checking for memory leaks. A single pod with a leak can push the entire node into pressure.
• Relying on limits without requests. If only limits are set, Kubernetes sets requests equal to limits, which may waste capacity. Set both intentionally.