Solution

Triage

1. Review the error output from the failed drain command to identify blocking pods.
2. List all pods on the node to categorize DaemonSet vs. non-DaemonSet pods:

   kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=node-5.internal
   

3. Identify which DaemonSets are running on the node:

   kubectl get daemonsets --all-namespaces
   

4. Check if any DaemonSet pods use local storage:

   kubectl get pods -n kube-system fluentd-ds-xr4t2 -o jsonpath='{.spec.volumes[*].name}'
   

Root Cause

kubectl drain by default refuses to delete pods managed by DaemonSets. This is by design: DaemonSet pods are tied to nodes, and evicting them would just cause the DaemonSet controller to immediately recreate them on the same node. The drain command requires explicit acknowledgment via --ignore-daemonsets to skip these pods.

The engineer ran kubectl drain node-5.internal without the flag, causing the command to fail before evicting any pods.

Fix

1. Run the drain with the correct flags:

   kubectl drain node-5.internal --ignore-daemonsets --delete-emptydir-data
   

2. If Fluentd uses a buffer directory, allow time for graceful shutdown by setting an appropriate --timeout:

   kubectl drain node-5.internal --ignore-daemonsets --delete-emptydir-data --timeout=300s
   

3. Verify all non-DaemonSet pods have been evicted:

   kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=node-5.internal
   

   Only DaemonSet pods should remain.
4. Once the node is fully drained, proceed with decommissioning (remove from cloud provider, delete the Node object):

   kubectl delete node node-5.internal
   

   This triggers DaemonSet pod cleanup.

Rollback / Safety

• If the drain needs to be aborted, uncordon the node:

  kubectl uncordon node-5.internal
  

• Verify that evicted workloads are running healthy on other nodes before deleting the node.
• If Fluentd has a persistent buffer, ensure log delivery is caught up before removing the node.

Common Traps

• Forgetting --ignore-daemonsets in automation. Any drain script or CI pipeline must include this flag; otherwise it will fail on every node.
• Confusing --ignore-daemonsets with deleting DaemonSet pods. The flag tells drain to skip them, not to evict them. They remain running.
• Using --force instead of --ignore-daemonsets. --force handles pods not managed by any controller (bare pods). It does not address DaemonSet pods. You typically need both flags for a clean drain.
• Not accounting for --delete-emptydir-data. If pods use emptyDir volumes, drain will refuse without this flag. Fluentd commonly uses emptyDir for its buffer.
• Assuming DaemonSet pods vanish after drain. They persist until the Node object is deleted or the DaemonSet is modified.