Skip to content

Container Runtime Debugging Cheat Sheet

Remember: The debug hierarchy matches the abstraction layers: kubectl (K8s level) > crictl (container runtime level) > nsenter (Linux namespace level) > /proc (kernel level). Start at the top — most issues are visible with kubectl describe and kubectl logs. Only dive deeper when the higher layers do not show the problem. Each layer down gives more detail but requires node access.

Debug Hierarchy

kubectl (high level)
  └── crictl (container runtime)
       └── nsenter (namespace level)
            └── /proc (kernel level)

kubectl debug

# Ephemeral debug container (K8s 1.23+)
kubectl debug -it <pod> --image=busybox --target=<container>

# Copy pod with debug image
kubectl debug <pod> --copy-to=debug-pod --image=ubuntu -- bash

# Debug node directly
kubectl debug node/<node-name> -it --image=ubuntu
# → gives you a shell with node's filesystem at /host

# Debug with specific tools image
kubectl debug -it <pod> --image=nicolaka/netshoot --target=app

crictl (Container Runtime Interface)

# List containers
crictl ps
crictl ps -a  # Include stopped

# List pods
crictl pods

# Inspect container
crictl inspect <container-id>

# Container logs
crictl logs <container-id>
crictl logs --tail=50 -f <container-id>

# Exec into container
crictl exec -it <container-id> sh

# Pull image
crictl pull <image>

# Container stats
crictl stats

# Pod stats
crictl statsp

Name origin: crictl = "Container Runtime Interface CTL" — it speaks the CRI gRPC protocol directly, so it works with containerd, CRI-O, or any CRI-compliant runtime. Unlike docker CLI (which only works with Docker), crictl is runtime-agnostic. On modern Kubernetes clusters (1.24+), Docker is no longer the default runtime, so crictl is the right tool.

nsenter — Enter Container Namespaces

# Find container PID
PID=$(crictl inspect <container-id> | jq '.info.pid')

# Enter all namespaces of a container
nsenter -t $PID -m -u -i -n -p -- bash

# Enter just the network namespace (useful for tcpdump)
nsenter -t $PID -n -- tcpdump -i eth0 -nn port 80

# Enter just the mount namespace (see container filesystem)
nsenter -t $PID -m -- ls /app

# Namespace flags:
# -m  mount     (filesystem)
# -u  UTS       (hostname)
# -i  IPC       (shared memory)
# -n  network   (interfaces, routes)
# -p  PID       (process tree)

/proc Inspection

# Find container's PID from node
PID=$(crictl inspect <container-id> | jq '.info.pid')

# Process info
cat /proc/$PID/status       # Process status, memory
cat /proc/$PID/cmdline      # Command line (null-separated)
ls -la /proc/$PID/fd        # Open file descriptors
cat /proc/$PID/limits       # Resource limits (ulimits)

# Network info
cat /proc/$PID/net/tcp      # TCP connections
cat /proc/$PID/net/tcp6     # TCP6 connections
cat /proc/$PID/net/sockstat # Socket statistics

# Filesystem info
cat /proc/$PID/mounts       # Mount points
cat /proc/$PID/mountinfo    # Detailed mount info

# Memory
cat /proc/$PID/smaps_rollup # Memory summary
cat /proc/$PID/oom_score    # OOM kill priority (higher = more likely)

# cgroup info
cat /proc/$PID/cgroup       # cgroup membership

Under the hood: nsenter works by entering one or more Linux namespaces of a target process. Containers are just processes with isolated namespaces (mount, network, PID, etc.). By entering only the network namespace (-n), you can run tcpdump from the host with the host's tools but see the container's network stack. This is the key technique for debugging distroless containers that have no shell or tools.

strace — System Call Tracing

# Trace a running process
strace -p $PID

# Trace with timestamps and follow forks
strace -p $PID -f -tt

# Trace specific syscalls
strace -p $PID -e trace=network    # Network calls only
strace -p $PID -e trace=file       # File operations
strace -p $PID -e trace=open,read  # Specific syscalls

# Trace with summary
strace -p $PID -c                  # Syscall count/time summary

# Trace from container (if strace not installed)
nsenter -t $PID -p -m -- strace -p 1

Network Debugging from Node

# tcpdump in container's network namespace
nsenter -t $PID -n -- tcpdump -i eth0 -nn -c 100

# Check container's DNS resolution
nsenter -t $PID -n -- nslookup kubernetes.default

# Check container's routes
nsenter -t $PID -n -- ip route

# Check container's iptables
nsenter -t $PID -n -- iptables -t nat -L -n

# ss from container's namespace
nsenter -t $PID -n -- ss -tlnp

Distroless Container Debugging

Distroless images have no shell. Use these approaches:

# 1. Ephemeral debug container (best option)
kubectl debug -it <pod> --image=busybox --target=<container>
# Shares PID namespace — can see container processes

# 2. Copy pod with debug image
kubectl debug <pod> --copy-to=debug-pod \
  --set-image=app=ubuntu --share-processes

# 3. From node via nsenter
nsenter -t $PID -m -n -p -- /bin/sh
# May fail if no shell in container
# Use node's tools with container's namespaces instead:
nsenter -t $PID -n -- tcpdump -i eth0

Common Debug Patterns

Problem Tool Command
App not responding strace strace -p $PID -e trace=network
High memory /proc cat /proc/$PID/smaps_rollup
Too many connections nsenter nsenter -t $PID -n -- ss -s
DNS failures nsenter nsenter -t $PID -n -- cat /etc/resolv.conf
File permission errors nsenter nsenter -t $PID -m -- ls -la /app/data
Container won't start crictl crictl logs <id>
OOMKilled /proc cat /proc/$PID/oom_score + check cgroup limits