Skip to content

Symptoms: Deployment Stuck, ImagePull Auth Failure, Fix Is Vault Secret Rotation

Domains: kubernetes_ops | security | devops_tooling Level: L2 Estimated time: 30-45 min

Initial Alert

Slack CI/CD notification at 09:45 UTC:

:x: Deployment FAILED — order-service v2.14.0
Namespace: prod
Reason: ProgressDeadlineExceeded
Replicas: 0/3 ready after 600s timeout

Followed by:

CRITICAL: order-service — 0 ready replicas
WARNING: order-service — ImagePullBackOff on 3 pods

Observable Symptoms

  • kubectl get pods -n prod -l app=order-service shows all 3 new pods in ImagePullBackOff.
  • kubectl describe pod shows: Failed to pull image "registry.internal:5000/order-service:v2.14.0": unauthorized: authentication required.
  • The old deployment (v2.13.0) is still running on the previous ReplicaSet with 3 healthy pods.
  • Other services deployed from the same registry in the past hour succeeded.
  • The image order-service:v2.14.0 exists in the registry (verified via curl to the registry API with manual credentials).
  • The imagePullSecrets in the pod spec references regcred-order-service.

The Misleading Signal

ImagePullBackOff with "authentication required" looks like a Kubernetes imagePullSecrets misconfiguration. The engineer's instinct is to check the Secret, verify the credentials, recreate the docker-registry Secret, or check if the ServiceAccount has the right imagePullSecrets attached. The fact that other services deployed successfully from the same registry makes this look like an order-service-specific Kubernetes configuration problem.