Solution

Triage

1. Get pod status and restart count:

   kubectl get pod auth-service-8c4f7d6b2-qp5r3 -n prod
   

2. Describe the pod to see container state and exit code:

   kubectl describe pod auth-service-8c4f7d6b2-qp5r3 -n prod
   

3. Attempt to read logs (will be empty):

   kubectl logs auth-service-8c4f7d6b2-qp5r3 -n prod --previous
   

4. Check the exit code in the terminated state -- 127 means "command not found."

Root Cause

Exit code 127 indicates the shell could not find the command specified as the container's entrypoint. The Dockerfile uses a multi-stage build. The final stage copies the compiled binary from the build stage, but the recent Dockerfile change modified the build output path without updating the COPY instruction in the final stage. As a result, the binary is missing from the final image.

Since the entrypoint binary does not exist, the container exits immediately before any application code runs, producing zero log output.

Example of the bug in the Dockerfile:

# Build stage
FROM golang:1.21 AS builder
WORKDIR /app
COPY . .
RUN go build -o /app/bin/auth-service ./cmd/auth  # Output path changed

# Final stage
FROM gcr.io/distroless/static
COPY --from=builder /app/auth-service /auth-service  # OLD path, file does not exist
ENTRYPOINT ["/auth-service"]

Fix

1. Immediate: Roll back the deployment to the previous working image:

   kubectl rollout undo deployment auth-service -n prod
   

2. Fix the Dockerfile: Correct the COPY path in the final stage:

   COPY --from=builder /app/bin/auth-service /auth-service
   

3. Build, test locally, and push the corrected image:

   docker build -t registry.internal/auth-service:v2.4.1-fixed .
   docker run --rm registry.internal/auth-service:v2.4.1-fixed
   docker push registry.internal/auth-service:v2.4.1-fixed
   

4. Update the deployment with the fixed image.

Rollback / Safety

• kubectl rollout undo is safe and instant -- it creates a new ReplicaSet with the previous pod template.
• Verify the rollback succeeded: kubectl rollout status deployment auth-service -n prod.
• Add a CI step that runs docker run <image> --version or a smoke test to catch missing binaries before deploy.

Common Traps

• Assuming empty logs means a Kubernetes issue. Empty logs almost always mean the process never started. Look at the exit code.
• Confusing exit code 126 (permission denied) with 127 (not found). Both produce empty logs but have different causes.
• Not testing distroless/scratch images locally. These images have no shell, making in-cluster debugging very difficult.
• Using kubectl exec on a CrashLoopBackOff pod. The container restarts too fast. Use kubectl debug with an ephemeral container or override the entrypoint to sleep.
• Overlooking dynamic linker errors. If using a compiled binary in a minimal image, missing libc or other shared libraries also cause exit code 127. Check with ldd in the build stage.