Progressive Hints

Hint 1 (after 5 min)

Compare the image digest running in the pods with what was pushed by CI. The pods are running sha256:3e7a9f2b...c841. The CI pipeline pushed sha256:9c4d8e1a...f273. These are different digests, even though both are tagged latest. The running pods have an older version.

Hint 2 (after 10 min)

The app's app_build_info metric reports version 2.3.1 with commit b91e4c7a built on October 8. The CI pipeline pushed version 2.5.0 with commit e3b7d9f1 on November 19. ArgoCD shows "Synced" with "0 updated, 3 unchanged" — it compared the manifest (which says tag: latest) with what is deployed (which also says tag: latest) and concluded nothing changed. ArgoCD does not check image digests by default.

Hint 3 (after 15 min)

This is a notification service that sends emails and SMS messages. The team pushed v2.5.0 (which includes a new SMS provider via feature flag sms_provider_v2), but the deployed pods are still running v2.3.1 from October. The trap is the latest tag: Kubernetes sees the same tag and uses imagePullPolicy: IfNotPresent (the default for named tags). Since the image is already cached on the nodes, Kubernetes does not pull the new digest. ArgoCD sees the manifests are unchanged (tag is still latest) and reports "Synced." Meanwhile, SMS failures are elevated (2,104 failures) because the legacy SMS gateway is degraded, and the new provider (v2.5.0) was never actually deployed.