Skip to content

Solution: Lab Runtime 07 -- GitOps Sync and Drift

SPOILER WARNING: Try to solve it yourself first.

Hint Ladder

Hint 1: Configuration drift means the live cluster state doesn't match what's declared in Git/Helm. What was changed manually?

Hint 2: Check the deployment's current replica count and environment variables. Compare to the Helm values file.

Hint 3: The break script manually scaled the deployment and injected a rogue environment variable -- both bypassing Helm. helm get values still shows the old values.

Hint 4: Re-apply the declared state with a Helm upgrade. This is what ArgoCD sync does: helm upgrade grokdevops devops/helm/grokdevops -n grokdevops -f devops/helm/values-dev.yaml.

Minimal Solution

helm upgrade grokdevops devops/helm/grokdevops -n grokdevops -f devops/helm/values-dev.yaml
kubectl rollout status deployment/grokdevops -n grokdevops --timeout=120s

Explain

Symptom: Deployment has unexpected replica count or environment variables not defined in Helm values.

Evidence: kubectl get deploy grokdevops -n grokdevops -o yaml shows values that don't match helm get values grokdevops -n grokdevops.

Root cause: Someone (or a script) used kubectl scale or kubectl set env to change the deployment directly, bypassing Helm. Helm's stored state is now out of sync with the live state. In a GitOps setup, ArgoCD would detect this as "OutOfSync."

Key insight: GitOps works because the desired state lives in Git and a reconciliation loop (ArgoCD/Flux) continuously compares live state to desired state. Manual changes create drift that the reconciler will revert on the next sync cycle.

Prevent

  • Use ArgoCD/Flux to auto-sync and prevent drift
  • Restrict kubectl write access in production (RBAC)
  • Add alerts for configuration drift (ArgoCD's OutOfSync notification)
  • Never use kubectl edit, kubectl scale, or kubectl set in production -- always go through Helm/Git

See Also