K8S Node Lifecycle¶
21 cards — 🟢 3 easy | 🟡 4 medium | 🔴 3 hard
🟢 Easy (3)¶
1. What is the lifecycle of a Kubernetes node?
Show answer
Provision -> register (kubelet joins cluster) -> schedule workloads -> cordon (stop new pods) -> drain (move existing pods) -> decommission or upgrade -> re-register -> repeat. Nodes are treated as ephemeral — the Kubernetes model assumes they can be replaced.Remember: Node lifecycle: join→Ready→(cordon)→drain→remove. Monitor: `kubectl get nodes`.
2. What is the difference between cordoning and draining a node?
Show answer
Cordoning marks a node as unschedulable — no new pods will be placed on it, but existing pods continue running. Draining goes further: it cordons the node AND evicts all existing pods, moving them to other nodes. Getting drain right means zero-downtime maintenance.Remember: Cordon=unschedulable, pods stay. "Caution tape." `kubectl uncordon` removes it.
3. What is a PodDisruptionBudget (PDB) and why does it create tension with node drains?
Show answer
A PDB specifies the minimum number (or percentage) of pods that must remain available during voluntary disruptions like node drains. It prevents drain from breaking applications by ensuring enough replicas stay running. However, PDBs are also what cause drains to get stuck — if a PDB cannot be satisfied (e.g., minAvailable equals replicas and no room to reschedule), the drain blocks indefinitely.Remember: Drain=cordon+evict. `--ignore-daemonsets --delete-emptydir-data` for stubborn pods.
Gotcha: DaemonSet pods can't be drained — that's why `--ignore-daemonsets` exists.
🟡 Medium (4)¶
1. How do taints and tolerations control pod scheduling on nodes?
Show answer
Taints are applied to nodes to repel pods that don't explicitly tolerate them (e.g., key=gpu:NoSchedule). Tolerations are set on pods to allow scheduling on tainted nodes. Three taint effects: NoSchedule (prevent scheduling), PreferNoSchedule (soft preference), NoExecute (evict existing pods). This mechanism is used to dedicate nodes for specific workloads like GPU jobs or system components.Remember: Effects: NoSchedule(hard), PreferNoSchedule(soft), NoExecute(evict+block). Increasing severity.
2. What node conditions does Kubernetes monitor for health, and what happens when a node goes NotReady?
Show answer
Kubernetes monitors conditions like MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable, and Ready. When a node's Ready condition becomes False or Unknown (kubelet stops reporting), the node controller waits for a grace period, then marks pods as Unknown status and begins evicting them to healthy nodes. This is the automated response to node failures.Remember: Node lifecycle: join→Ready→(cordon)→drain→remove. Monitor: `kubectl get nodes`.
3. What is the difference between manual node scaling and cluster autoscaling?
Show answer
Manual scaling requires an operator to add or remove nodes. Cluster Autoscaler automatically adds nodes when pods are pending due to insufficient resources and removes underutilized nodes. The autoscaler respects PDBs during scale-down and checks that all pods on a node can be rescheduled elsewhere before removing it.Remember: Node lifecycle: join→Ready→(cordon)→drain→remove. Monitor: `kubectl get nodes`.
4. What is the standard approach for upgrading Kubernetes node versions?
Show answer
Cordon the node, drain it (respecting PDBs), upgrade the kubelet and container runtime, then uncordon and allow pods to schedule back. In managed Kubernetes (EKS, GKE, AKS), this is often handled by rolling replacement of node groups — new nodes with the updated version are added while old nodes are cordoned, drained, and terminated.Remember: Node lifecycle: join→Ready→(cordon)→drain→remove. Monitor: `kubectl get nodes`.
🔴 Hard (3)¶
1. What causes a node drain to get stuck and how do you troubleshoot it?
Show answer
Common causes: PDB cannot be satisfied (minAvailable equals replicas with no room to reschedule), pods with local storage (emptyDir) that can't be rescheduled, pods without a controller (bare pods not managed by a Deployment/ReplicaSet), or pods with long terminationGracePeriodSeconds. Troubleshoot by checking which pods remain, examining their PDBs, and using --ignore-daemonsets --delete-emptydir-data --force flags if safe.Remember: Drain=cordon+evict. `--ignore-daemonsets --delete-emptydir-data` for stubborn pods.
Gotcha: DaemonSet pods can't be drained — that's why `--ignore-daemonsets` exists.
2. What are common causes of kubelet failures and how do they manifest?
Show answer
Kubelet failures cause the node to go NotReady. Common causes: kubelet process crash (check systemctl status kubelet and journalctl -u kubelet), certificate expiration (kubelet can't authenticate to API server), disk pressure triggering evictions, container runtime failure (containerd/CRI-O down), or network partition isolating the node from the control plane. Each manifests differently in node conditions.Remember: Node lifecycle: join→Ready→(cordon)→drain→remove. Monitor: `kubectl get nodes`.
3. What combination of mechanisms ensures zero-downtime during node maintenance?
Show answer
PDBs ensure minimum pod availability during drain. Pod anti-affinity spreads replicas across nodes so no single node holds all instances. PreStop hooks give pods time to finish in-flight requests before termination. The Cluster Autoscaler or surge capacity ensures replacement nodes exist before draining. Together: PDB + anti-affinity + graceful shutdown + capacity headroom = zero-downtime maintenance.Remember: Node lifecycle: join→Ready→(cordon)→drain→remove. Monitor: `kubectl get nodes`.