Skip to content

Portal | Level: L2: Operations | Topics: GitOps | Domain: DevOps & Tooling

Lab Runtime 07 — GitOps Sync and Drift

Objective

Demonstrate GitOps sync and drift reconciliation using ArgoCD concepts. This lab introduces configuration drift by manually modifying a running deployment outside of Helm/Git, then reconciles the drift by re-applying the declared Helm state (simulating an ArgoCD sync).

Prerequisites

  • make deploy-all has been run and the grokdevops app is deployed and healthy
  • Helm 3 installed
  • kubectl configured for the target cluster
  • ArgoCD installed (optional -- the lab simulates ArgoCD sync behavior via Helm)

Steps

  1. Break: Run ./break.sh to introduce drift by manually scaling the deployment and injecting a rogue environment variable, bypassing Helm/Git.
  2. Observe: Compare the live state to the declared Helm values. In a real GitOps setup, ArgoCD would flag this as "OutOfSync".
  3. Fix: Run ./fix.sh to reconcile drift by performing a Helm upgrade with the declared values (simulating an ArgoCD sync).
  4. Verify: Run ./verify.sh to confirm replica count and environment variables match the declared state.
  5. Teardown: Run ./teardown.sh to ensure clean Helm state.

Expected Observations

  • kubectl get deployment grokdevops -n grokdevops shows a replica count that differs from the Helm-declared value (e.g., manually scaled to 5 when Helm says 2).
  • kubectl get deployment grokdevops -n grokdevops -o jsonpath='{.spec.template.spec.containers[0].env}' shows a rogue environment variable not present in the Helm values file.
  • helm get values grokdevops -n grokdevops does not include the manual changes, confirming drift between desired state (Helm/Git) and actual state (cluster).

Wrong Turns

  1. Accepting drift as normal — Drift means the cluster no longer matches the declared source of truth. This leads to unreproducible deployments, mystery behavior, and broken disaster recovery.
  2. Using kubectl to "fix" the drift — Applying more manual kubectl patches creates additional drift. The correct approach is to reconcile via Helm (or ArgoCD sync) so the declared state wins.
  3. Blaming ArgoCD or the GitOps tool — ArgoCD correctly detects and reports drift. The problem is the manual change, not the tool that flags it. The fix is to re-sync from the declared state.

Minimal Explanation

Helm stores the rendered manifests of each release revision in a Kubernetes Secret. When you use kubectl to directly modify a resource (e.g., scaling replicas or adding env vars), those changes exist only in the live cluster state (etcd) and are not recorded in Helm's stored manifests or in Git. This creates divergence: the desired state in Git says one thing, the cluster says another. GitOps controllers like ArgoCD continuously compare the two and flag differences as "OutOfSync." The fix is to re-apply the declared state via helm upgrade (or ArgoCD sync), which overwrites the manual changes with what Git declares.

Transfer Pattern

  • Emergency hotfixes in production: An engineer uses kubectl to patch a production issue at 2 AM. The fix works but is never committed to Git, so the next deploy reverts it.
  • Team members using kubectl directly: In organizations without strict GitOps enforcement, developers kubectl apply changes that bypass the release pipeline, creating invisible drift.

See Also

  • training/library/guides/gitops-example.md
  • training/interview-scenarios/07-config-drift-detected.md

Solution (spoilers)

See training/library/solutions/labs/lab-runtime-07.md for hints and explanation.

Teardown

./teardown.sh

Or reset the entire environment:

make undeploy-all

Wiki Navigation

Prerequisites

  • Kubernetes Exercises (Quest Ladder) (CLI) (Exercise Set, L1)

Related Content