Skip to content

Pattern: latest Tag in Production

ID: FP-033 Family: Configuration Landmine Frequency: Very Common Blast Radius: Single Service to Multi-Service Detection Difficulty: Subtle

The Shape

Using a mutable image tag (latest, stable, main) in a production deployment means the deployed image can change without any deployment action. When a pod restarts for any reason (OOMKill, node maintenance, rolling restart), the kubelet pulls the current image behind the tag, which may differ from what was originally deployed. The deployment spec hasn't changed; no deploy was triggered; but the running code has changed silently.

How You'll See It

In Kubernetes

image: myapp:latest with imagePullPolicy: Always. Developer pushes a new image to latest for testing. A production pod is evicted (node pressure). On restart, it pulls the test image. Production now runs test code on one pod. No deployment was triggered. No alert fires.

# Detect mixed versions:
kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[0].imageID}{"\n"}{end}'

In Linux/Infrastructure

Docker Compose with image: redis:latest. Redis team publishes Redis 8.0 as latest. A server reboots. Docker pulls Redis 8.0. Production Redis is now running a major version upgrade without any approval, testing, or migration plan.

In CI/CD

Base image FROM python:latest in Dockerfile. Python 3.12 is published. Next build uses Python 3.12. Code that worked on Python 3.11 breaks silently. The code didn't change; the build environment did.

The Tell

kubectl get pods -o json | jq '.items[].status.containerStatuses[].imageID' shows different SHA256 digests across pods in the same deployment. The tag referenced in the deployment spec resolves to different images on different dates. A pod restart triggered unexpected behavior change without any deploy.

Common Misdiagnosis

Looks Like But Actually How to Tell the Difference
Application bug introduced in recent deploy Silent image tag update No deploy happened; image digest changed behind the mutable tag
Flaky behavior Mixed-version fleet Some pods behave correctly; others don't; image digests differ across pods
Dependency issue Image update Container behavior changed; application code unchanged

The Fix (Generic)

  1. Immediate: Pin the deployment to a specific image digest: kubectl set image deployment/myapp container=myapp@sha256:<digest>.
  2. Short-term: Update CI/CD pipeline to tag images with an immutable identifier (git SHA, semantic version + build number); enforce imagePullPolicy: IfNotPresent for pinned tags.
  3. Long-term: Use OPA/Kyverno admission webhook to reject latest or untagged images in production namespaces; require image digest pinning in production Helm values.

Real-World Examples

  • Example 1: nginx:latest in a production Ingress controller DaemonSet. NGINX published 1.27 as latest. During a routine node drain for maintenance, NGINX 1.27 was pulled. Breaking change in header handling caused 503s for 10% of requests until identified.
  • Example 2: Three-service application all using latest. All three images were updated on the same day for a staging test. One production pod restarted (OOMKill). Mixed versions caused an API contract mismatch between services.

War Story

"Why is production returning 400 for some users?" No recent deploy. I checked pod image digests and found two different SHA256s for the same deployment. Someone had pushed to :latest for a "quick test" and forgotten to untag it. Two pods had restarted (OOMKills from the week before) and pulled the new image. We were running a partially-tested experimental version in production on 20% of pods. We immediately pinned all deployments to digests as an emergency measure, then made image digest pinning a required CI check.

Cross-References