Skip to content

Portal | Level: L2: Operations | Topics: GitOps | Domain: DevOps & Tooling

GitOps & ArgoCD Drills

Remember: GitOps has two sync strategies: automated (ArgoCD applies changes immediately when Git changes) and manual (human clicks sync). In production, start with manual + auto-prune disabled. Enable auto-sync only after you trust the pipeline. Mnemonic: the Git repo is the single source of truth — if it is not in Git, it should not be in the cluster.

Gotcha: ArgoCD's selfHeal: true reverts manual kubectl changes back to what Git says. This is the point — but it catches teams off-guard when they kubectl edit a deployment for a hotfix and it immediately reverts. For emergency changes, either commit to Git first or temporarily disable auto-sync on that app.

Debug clue: If ArgoCD shows OutOfSync but Sync does nothing, the diff is likely in a field that Kubernetes mutates (like metadata.resourceVersion or status). Check the diff in the ArgoCD UI — you may need to add the field to ignoreDifferences in the Application spec.

Drill 1: Create an ArgoCD Application

Difficulty: Easy

Q: Write an ArgoCD Application manifest that deploys from https://github.com/org/app.git, path k8s/production, to the production namespace with automated sync and self-heal.

Answer 
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/org/app.git
    targetRevision: main
    path: k8s/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
Key fields: - `prune: true` — deletes resources removed from Git - `selfHeal: true` — reverts manual kubectl changes - `CreateNamespace=true` — creates namespace if missing

Drill 2: Debug OutOfSync

Difficulty: Easy

Q: An app shows OutOfSync in ArgoCD. What commands do you run to understand why?

Answer 
# 1. Get app status
argocd app get my-app

# 2. View the exact diff
argocd app diff my-app

# 3. Hard refresh (re-read Git, clear cache)
argocd app get my-app --hard-refresh

# 4. Check sync status details
argocd app get my-app -o json | jq '.status.conditions'

# 5. Check resource-level sync status
argocd app resources my-app
Common causes: - Server-side defaults (fields added by K8s that differ from Git) - HPA modifying replicas - Mutating webhooks adding/modifying fields - Finalizers or other controller-set fields

Drill 3: Fix HPA vs ArgoCD Conflict

Difficulty: Medium

Q: ArgoCD keeps reverting spec.replicas to 3 (Git value) while HPA tries to scale to 8. How do you fix this?

Answer 
# Add to the Application spec:
spec:
  ignoreDifferences:
  - group: apps
    kind: Deployment
    jsonPointers:
    - /spec/replicas

# Immediate fix via CLI:
argocd app set my-app --ignore-difference \
  '{"group":"apps","kind":"Deployment","jsonPointers":["/spec/replicas"]}'
Best practice: Remove `spec.replicas` from the Deployment manifest in Git entirely when using HPA. Let HPA be the sole owner of replica count.

Drill 4: App-of-Apps Pattern

Difficulty: Medium

Q: You have 20 microservices. Set up an App-of-Apps so adding a new service is just adding one file to Git.

Answer 
repo/
├── apps/           # App-of-Apps root
│   ├── svc-a.yaml  # Application manifest for service A
│   ├── svc-b.yaml  # Application manifest for service B
│   └── svc-c.yaml  # ...
└── services/       # Actual K8s manifests
    ├── svc-a/
    ├── svc-b/
    └── svc-c/

# Parent Application (App-of-Apps)
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app-of-apps
  namespace: argocd
spec:
  source:
    repoURL: https://github.com/org/repo.git
    path: apps
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

# apps/svc-a.yaml — child Application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: svc-a
  namespace: argocd
spec:
  source:
    repoURL: https://github.com/org/repo.git
    path: services/svc-a
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated: { prune: true, selfHeal: true }
To add a new service: create `apps/svc-d.yaml` and `services/svc-d/`. ArgoCD auto-discovers it.

Drill 5: ApplicationSet

Difficulty: Medium

Q: Replace the App-of-Apps with an ApplicationSet that auto-generates an Application for each subdirectory under services/.

Answer 
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: services
  namespace: argocd
spec:
  generators:
  - git:
      repoURL: https://github.com/org/repo.git
      revision: main
      directories:
      - path: services/*
  template:
    metadata:
      name: '{{path.basename}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/org/repo.git
        targetRevision: main
        path: '{{path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{path.basename}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
        - CreateNamespace=true
Now adding `services/new-svc/` auto-creates an ArgoCD Application for it.

Drill 6: Sync Waves

Difficulty: Medium

Q: You need to deploy a database before the app, and a migration Job between them. Set up sync waves.

Answer 
# Wave 0: Database (deployed first)
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  annotations:
    argocd.argoproj.io/sync-wave: "0"
---
# Wave 1: Migration Job (runs after DB is healthy)
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
  annotations:
    argocd.argoproj.io/sync-wave: "1"
    argocd.argoproj.io/hook: Sync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
---
# Wave 2: Application (deployed after migration)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  annotations:
    argocd.argoproj.io/sync-wave: "2"
ArgoCD applies resources in wave order. Within a wave, it waits for health before proceeding to the next wave.

Drill 7: Rollback

Difficulty: Easy

Q: A bad deployment went out via ArgoCD. How do you rollback?

Answer 
# 1. View sync history
argocd app history my-app

# 2. Rollback to a specific revision
argocd app rollback my-app <history-id>
**But the GitOps way is:** 
# Revert the commit in Git
git revert HEAD
git push

# ArgoCD will auto-sync to the reverted state
Why Git revert is better: - Maintains Git as the source of truth - Creates an audit trail - ArgoCD's self-heal won't fight the rollback - The `argocd app rollback` is a temporary override that self-heal will revert

Drill 8: Image Updater

Difficulty: Medium

Q: Configure ArgoCD Image Updater to automatically update the web container's image tag when a new semver tag is pushed to the registry.

Answer 
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
  annotations:
    argocd-image-updater.argoproj.io/image-list: web=registry.example.com/my-app
    argocd-image-updater.argoproj.io/web.update-strategy: semver
    argocd-image-updater.argoproj.io/web.allow-tags: regexp:^v[0-9]+\.[0-9]+\.[0-9]+$
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/git-branch: main
spec:
  # ... normal Application spec
Update strategies: - `semver` — latest semantic version - `latest` — most recently built - `name` — alphabetically latest tag - `digest` — detect digest changes for same tag

Drill 9: Multi-Cluster Deployment

Difficulty: Hard

Q: You have 3 clusters (dev, staging, prod). Set up an ApplicationSet that deploys the same app to all clusters with environment-specific overrides.

Answer 
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: my-app
  namespace: argocd
spec:
  generators:
  - list:
      elements:
      - cluster: dev
        url: https://dev-cluster:6443
        replicas: "1"
        values:
          environment: dev
      - cluster: staging
        url: https://staging-cluster:6443
        replicas: "2"
        values:
          environment: staging
      - cluster: prod
        url: https://prod-cluster:6443
        replicas: "5"
        values:
          environment: prod
  template:
    metadata:
      name: 'my-app-{{cluster}}'
    spec:
      source:
        repoURL: https://github.com/org/repo.git
        path: k8s/overlays/{{values.environment}}
      destination:
        server: '{{url}}'
        namespace: my-app
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
Use Kustomize overlays or Helm values per environment in the repo.

Drill 10: Disaster Recovery

Difficulty: Hard

Q: Your ArgoCD instance is completely lost. How do you recover?

Answer 
# 1. Reinstall ArgoCD
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# 2. Reconnect to Git repos (if using declarative setup)
# Apply your ArgoCD configuration from Git:
kubectl apply -f argocd-config/repositories.yaml
kubectl apply -f argocd-config/projects.yaml

# 3. Re-apply all Application manifests
kubectl apply -f argocd-apps/

# 4. ArgoCD will auto-sync everything from Git
# All state is in Git — ArgoCD is stateless!

# 5. Verify
argocd app list
argocd app get my-app
This is a key GitOps benefit: ArgoCD is reconstructable from Git. You don't need to back up ArgoCD itself — back up your Git repo and Application manifests. What you DO need to back up: - ArgoCD secrets (repo credentials, cluster credentials) - `argocd-cm` and `argocd-rbac-cm` ConfigMaps (if not in Git)

Wiki Navigation

Related Content