Kubernetes Debugging Footguns¶

Mistakes that turn a 5-minute diagnosis into a 2-hour rabbit hole or make the problem worse.

1. Deleting a pod instead of investigating it¶

The pod is CrashLooping. You run kubectl delete pod to "fix" it. The deployment creates a new pod. The new pod also CrashLoops. But now the old pod's logs are gone, the old container's filesystem is gone, and you have no evidence to debug.

Why people do it: Restarting things is the first instinct. It works for laptops. Kubernetes makes deletion feel free because the controller recreates the pod.

Fix: Before deleting anything, capture the evidence: kubectl logs <pod> --previous, kubectl describe pod <pod>, kubectl get events. If you need a fresh pod, scale the deployment up by one rather than deleting the broken one.

Debug clue: kubectl debug (since k8s 1.25 stable) lets you attach an ephemeral debug container to a running or crashed pod without restarting it: kubectl debug -it <pod> --image=busybox --target=<container>. This shares the process namespace so you can inspect /proc, run strace, and check files — all without losing the pod's state.

2. Ignoring the Events section in `kubectl describe`¶

You run kubectl describe pod and read the top: status, container states, resource requests. You stop scrolling. The Events section at the bottom has the actual answer -- FailedScheduling, FailedMount, ImagePullBackOff with a specific registry error.

Why people do it: describe output is long. The Events section is at the bottom and easy to miss. People jump to kubectl logs which only shows application output, not infrastructure failures.

Fix: Read Events first. They tell you what Kubernetes tried and what failed. kubectl get events -n <namespace> --sort-by='.lastTimestamp' gives you the timeline. Events expire after 1 hour by default -- check quickly.

3. Debugging in the wrong namespace¶

You run kubectl get pods and see nothing. The application "isn't running." It is running -- in a different namespace. You spend 20 minutes checking RBAC, node health, and the deployment before realizing your context is set to default and the app is in production.

Why people do it: kubectl uses whatever namespace is in your kubeconfig context. Most people never change it from default. The -n flag is easy to forget.

Fix: Always specify namespace explicitly: kubectl get pods -n <namespace>. Better: set your context namespace to match your work: kubectl config set-context --current --namespace=production. Use a prompt plugin that shows the current context and namespace.

4. Running `kubectl exec` into the wrong container in a multi-container pod¶

Your pod has an app container and a sidecar (envoy, fluentd, istio-proxy). You run kubectl exec -it <pod> -- sh. You land in the sidecar. You do not realize this. You look at the filesystem, see unexpected binaries, try to find the application config, get confused. You debug the wrong container for 30 minutes.

Why people do it: kubectl exec defaults to the first container in the pod spec. People do not check which container they are in.

Fix: Always specify the container: kubectl exec -it <pod> -c <container-name> -- sh. If you are unsure which containers exist: kubectl get pod <pod> -o jsonpath='{.spec.containers[*].name}'.

5. Assuming ImagePullBackOff means the image does not exist¶

The pod shows ImagePullBackOff. You check the image name for typos. It looks correct. The image exists in the registry. You spend 45 minutes verifying image tags. The actual problem is the node cannot authenticate to the registry -- the imagePullSecret is missing, expired, or in the wrong namespace.

Why people do it: "ImagePull" implies the image is the problem. The actual error message in Events tells you whether it is a 401 (auth), 404 (not found), or network error, but people do not read that far.

Fix: Run kubectl describe pod <pod> and read the full event message. A 401 or "unauthorized" means auth. Check imagePullSecrets: kubectl get pod <pod> -o jsonpath='{.spec.imagePullSecrets}'. Verify the secret exists and is not expired.

Default trap: When no imagePullSecrets are specified, Kubernetes uses the node's container runtime credentials (e.g., ~/.docker/config.json on the node). This works on initial setup but silently breaks when nodes are replaced by autoscaling — new nodes don't have the cached credentials. Always use explicit imagePullSecrets in the pod spec or a mutating webhook to inject them.

6. Port-forwarding to debug a service and concluding "it works"¶

You run kubectl port-forward svc/myapp 8080:80 and curl localhost. It works. You close the ticket. But port-forward bypasses the service mesh, network policies, ingress rules, and DNS resolution. The actual traffic path is broken -- port-forward just created a direct tunnel that proves nothing about the real path.

Why people do it: Port-forward is convenient. It confirms the application is responding. It feels like a definitive test.

Fix: Port-forward confirms the pod is healthy. It does not validate the network path. After confirming the app works, test the real path: kubectl exec <another-pod> -- curl http://myapp.namespace.svc.cluster.local. Test from outside the cluster via ingress. Test each hop separately.

7. Applying resource changes with `kubectl apply` during an active incident¶

The pod is OOMKilled. You increase the memory limit in the deployment YAML and kubectl apply. The rolling update starts. New pods fail to schedule because the node does not have enough memory for the increased limit. Now you have zero running pods instead of crash-looping pods.

Why people do it: Increasing resources feels like an obvious fix. kubectl apply is the muscle-memory command. Nobody checks whether the cluster has capacity for the new request.

Fix: Before increasing resource limits, check node capacity: kubectl describe node | grep -A5 "Allocated resources". If tight, consider scaling down replicas first or evicting less critical workloads. Apply the change to one pod first (canary) before rolling across the deployment.

8. Not checking previous container logs¶

The pod restarted. kubectl logs <pod> shows the current container's startup messages. The crash that caused the restart is in the previous container's logs. You see healthy startup output and conclude "it looks fine."

Why people do it: kubectl logs shows the current container by default. If the pod has restarted, you are looking at the new container, not the one that crashed.

Fix: Always check previous logs after a restart: kubectl logs <pod> --previous. Compare with current logs. If restartCount in kubectl get pods is incrementing, previous logs are where the crash evidence lives.

9. Scaling to zero replicas to "stop the bleeding"¶

Your service is throwing errors. You scale to zero: kubectl scale deployment myapp --replicas=0. The errors stop. But now every dependent service starts failing because they cannot reach yours. The cascading failure is worse than the original errors.

Why people do it: Stopping the source of errors feels like damage control. It is the Kubernetes equivalent of unplugging the server.

Fix: If the service is serving bad responses, fix the response -- do not remove the service. Consider rolling back to the last known-good image: kubectl rollout undo deployment/myapp. If you must drain traffic, use a service mesh fault injection or weight traffic to healthy pods, not a hard zero.

10. Forgetting that resource limits cause throttling, not just OOMKill¶

Your application is slow. CPU usage looks low in monitoring. You check memory -- no OOMKills. Everything "looks fine." But the pod has a CPU limit of 500m, and the application is being throttled. Kubernetes throttles CPU silently -- there is no event, no log, no alert by default.

Why people do it: CPU throttling is invisible in standard monitoring. kubectl top pod shows usage relative to request, not limit. The application just appears slow.

Fix: Check CPU limits vs actual usage: kubectl top pod <pod> and compare with the limit in the pod spec. Check throttling metrics from cAdvisor: container_cpu_cfs_throttled_periods_total. Consider removing CPU limits entirely (keep requests) -- many production clusters run better this way.

Under the hood: CPU throttling happens at the CFS (Completely Fair Scheduler) level. The kernel enforces limits over 100ms windows: if your limit is 500m (50ms of CPU per 100ms), a burst that uses 50ms in the first 10ms is throttled for the remaining 90ms. This means latency-sensitive applications can see 90ms stalls even when average CPU is low. The metric container_cpu_cfs_throttled_seconds_total in Prometheus quantifies the total time lost to throttling.