Portal | Level: L2: Operations | Topics: ArgoCD & GitOps | Domain: DevOps & Tooling

ArgoCD & GitOps — Primer

Why This Matters

GitOps shifts the deployment model: Git is the single source of truth for cluster state, and an agent continuously reconciles the live cluster toward whatever is committed. This eliminates the "works on my laptop, drift in prod" problem because nobody runs kubectl apply by hand — every change flows through a pull request and an audit trail.

Name origin: "Argo" comes from the ship Argo in Greek mythology — the vessel that carried Jason and the Argonauts. The Argo project family (Argo CD, Argo Workflows, Argo Rollouts, Argo Events) was created at Applatix (later Intuit) and donated to the CNCF, graduating in 2022.

ArgoCD is the dominant GitOps controller for Kubernetes. Understanding it deeply means you can build self-healing, auditable, multi-cluster delivery pipelines rather than fragile imperative scripts. When an engineer pushes a bad commit, ArgoCD either auto-reverts (if you configured self-heal) or holds a clear diff of what drifted — either way, the on-call has a precise starting point.

The shift from push-based CI/CD (pipeline calls kubectl) to pull-based (controller watches Git) is architectural. It changes who has cluster credentials (only ArgoCD, not every CI runner), how you audit changes (every deploy is a commit), and how you recover (revert the commit, not roll back in CI). Ops people who understand both models can make deliberate tradeoffs rather than just using whatever the team inherited.

Core Concepts

1. GitOps: Pull vs Push

Push-based (classical CI/CD): A pipeline builds an image, then calls kubectl apply or helm upgrade. The pipeline needs cluster credentials. Drift accumulates if anyone applies outside the pipeline.

Pull-based (GitOps): A controller running inside the cluster watches a Git repository. When the repo changes, the controller applies the diff. Credentials never leave the cluster. The Git repo is the source of truth.

Push model:
  Developer → Git → CI pipeline → kubectl apply → Cluster

Pull model:
  Developer → Git ← ArgoCD controller → Cluster
                       (reconcile loop)

The pull model catches out-of-band changes (drift) and can auto-heal them. The push model cannot.

2. ArgoCD Architecture

ArgoCD runs as a set of controllers in the argocd namespace:

Component	Role
argocd-server	API server + UI, handles auth, RBAC
argocd-repo-server	Clones repos, renders manifests (Helm, Kustomize, plain YAML)
argocd-application-controller	The reconciliation brain — diffs desired vs live, syncs
argocd-dex-server	OIDC/SSO provider integration
argocd-notifications-controller	Slack/PagerDuty/email alerts on sync events
argocd-applicationset-controller	Generates Applications from templates

# Check component health
kubectl -n argocd get pods
kubectl -n argocd logs deploy/argocd-application-controller -f

# ArgoCD version
argocd version

3. The Application Resource

An Application is the core CRD. It declares: what repo/path to watch, what cluster/namespace to deploy into, and sync policy.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io  # cascade-delete on app deletion
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/gitops-repo
    targetRevision: HEAD
    path: apps/my-app/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: my-app
  syncPolicy:
    automated:
      prune: true       # delete resources removed from Git
      selfHeal: true    # revert manual kubectl changes
    syncOptions:
      - CreateNamespace=true
      - PrunePropagationPolicy=foreground
      - ApplyOutOfSyncOnly=true

Key fields: - targetRevision: branch, tag, or commit SHA. Use a tag for production (not HEAD). - prune: true: resources deleted from Git get deleted from the cluster. Default false — do NOT enable without understanding the blast radius. - selfHeal: true: any manual kubectl apply that diverges from Git gets reverted within ~3 minutes.

Gotcha: Enabling prune: true with selfHeal: true on a shared namespace is the most common ArgoCD foot-gun. If another team or controller creates resources in that namespace and ArgoCD does not manage them, ArgoCD will delete them because they are not in Git. Use PrunePropagationPolicy=foreground and carefully scope which resources ArgoCD manages.

4. AppProject — RBAC & Policy

AppProject scopes what repos, clusters, and namespaces an Application can use. It is the multi-tenancy boundary.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-payments
  namespace: argocd
spec:
  description: "Payments team GitOps project"
  sourceRepos:
    - https://github.com/myorg/payments-gitops
    - https://github.com/myorg/shared-charts
  destinations:
    - namespace: payments-*
      server: https://kubernetes.default.svc
    - namespace: payments-*
      server: https://prod-cluster.example.com
  clusterResourceWhitelist:
    - group: ""
      kind: Namespace
  namespaceResourceBlacklist:
    - group: ""
      kind: ResourceQuota   # team cannot create ResourceQuotas
  roles:
    - name: team-deployer
      policies:
        - p, proj:team-payments:team-deployer, applications, sync, team-payments/*, allow
        - p, proj:team-payments:team-deployer, applications, get, team-payments/*, allow
      groups:
        - payments-engineers

5. Sync Policies, Waves, and Hooks

Sync waves control ordering within a single sync operation. Lower wave numbers sync first. ArgoCD waits for all resources in wave N to be healthy before starting wave N+1.

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"   # runs first

Common wave pattern for a full-stack app: - Wave -2: Namespace, RBAC - Wave -1: ConfigMaps, Secrets - Wave 0: Database migration Job - Wave 1: Application Deployment - Wave 2: Ingress, HPA

Sync hooks execute at specific phases: PreSync, Sync, PostSync, SyncFail.

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: myapp:v1.2.3
          command: ["python", "manage.py", "migrate"]
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: url

hook-delete-policy options: - BeforeHookCreation: delete old hook resource before creating the new one (default sane choice) - HookSucceeded: delete after success (clean, but makes debugging harder) - HookFailed: delete after failure (almost never what you want)

6. App of Apps Pattern

The App of Apps pattern uses one root Application that renders a directory of Application manifests. This is how you bootstrap an entire cluster from a single commit.

gitops-repo/
├── root-app.yaml           ← applied once by hand to bootstrap
├── apps/
│   ├── monitoring.yaml     ← Application for Prometheus stack
│   ├── ingress.yaml        ← Application for nginx-ingress
│   ├── cert-manager.yaml   ← Application for cert-manager
│   └── my-service.yaml     ← Application for your service

# root-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/gitops-repo
    targetRevision: main
    path: apps
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Bootstrap procedure:

kubectl apply -f root-app.yaml
# ArgoCD syncs root-app, which creates all child Applications
# Each child Application then syncs its own workloads

7. ApplicationSet — Templated Multi-Cluster Deployments

ApplicationSet generates multiple Application resources from a single template. It supports several generators.

List generator — explicit list of targets:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: my-app-all-clusters
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - cluster: prod-us-east
            url: https://prod-us-east.example.com
            env: prod
          - cluster: prod-eu-west
            url: https://prod-eu-west.example.com
            env: prod
          - cluster: staging
            url: https://staging.example.com
            env: staging
  template:
    metadata:
      name: "my-app-{{cluster}}"
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/gitops-repo
        targetRevision: main
        path: "apps/my-app/overlays/{{env}}"
      destination:
        server: "{{url}}"
        namespace: my-app
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Git generator — discovers targets from directory structure:

generators:
  - git:
      repoURL: https://github.com/myorg/gitops-repo
      revision: main
      directories:
        - path: "clusters/*/apps"

Cluster generator — targets all registered clusters matching a label:

generators:
  - clusters:
      selector:
        matchLabels:
          environment: production

8. Drift Detection and Health Checks

ArgoCD computes diff by comparing the Git manifest (desired) against the live Kubernetes resource (actual). It marks an Application OutOfSync if any managed field differs.

# See what's drifted
argocd app diff my-app

# Get detailed sync status
argocd app get my-app

# Force sync (override auto-sync)
argocd app sync my-app --force

# Sync only specific resources
argocd app sync my-app --resource apps:Deployment:my-app

Health checks are built in for common resources: - Deployment: Healthy when availableReplicas == desiredReplicas - StatefulSet: Healthy when readyReplicas == replicas - Job: Healthy when succeeded >= 1 - PVC: Healthy when phase == Bound

Custom health check via Lua (in ArgoCD ConfigMap):

# argocd-cm ConfigMap
data:
  resource.customizations.health.batch_CronJob: |
    hs = {}
    if obj.status ~= nil then
      if obj.status.lastScheduleTime ~= nil then
        hs.status = "Healthy"
        hs.message = "Last scheduled: " .. obj.status.lastScheduleTime
        return hs
      end
    end
    hs.status = "Progressing"
    hs.message = "Waiting for first schedule"
    return hs

9. Multi-Cluster Patterns

Register external clusters:

# Login with admin credentials first
argocd login argocd.example.com --sso

# Add a cluster
argocd cluster add prod-us-east --name prod-us-east

# List clusters
argocd cluster list

# Check cluster connection
argocd cluster get prod-us-east

ArgoCD stores cluster credentials as Secrets in the argocd namespace:

kubectl -n argocd get secret -l argocd.argoproj.io/secret-type=cluster

Hub-spoke model: One ArgoCD instance (hub) manages multiple clusters (spokes). All Applications reference the spoke cluster's API server URL.

Instance-per-cluster: Each cluster runs its own ArgoCD. A gitops-meta repo drives each ArgoCD instance's own ApplicationSets. More isolation, more operational overhead.

10. RBAC

ArgoCD RBAC uses a Casbin policy stored in argocd-rbac-cm:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.default: role:readonly
  policy.csv: |
    # Grant team-alpha read+sync access to their apps
    p, role:team-alpha, applications, get, team-alpha/*, allow
    p, role:team-alpha, applications, sync, team-alpha/*, allow
    p, role:team-alpha, applications, action/*, team-alpha/*, allow
    p, role:team-alpha, repositories, get, *, allow

    # Grant release-managers ability to sync prod apps
    p, role:release-manager, applications, sync, prod/*, allow
    p, role:release-manager, applications, get, */*, allow

    # Bind SSO groups to roles
    g, myorg:team-alpha, role:team-alpha
    g, myorg:release-managers, role:release-manager
  scopes: "[groups]"

11. FluxCD vs ArgoCD

Feature	ArgoCD	FluxCD
UI	Rich built-in UI	No built-in UI (use Weave GitOps)
CRD model	Application, AppProject, ApplicationSet	Kustomization, HelmRelease, GitRepository
Multi-tenancy	AppProject scoping	Namespace-per-tenant, RBAC
Helm native	Via source plugin	HelmRelease CRD, native charts
Image automation	Via image-updater	ImagePolicy, ImageUpdateAutomation built-in
Notification	argocd-notifications	notification-controller built-in
Multi-cluster	Hub-spoke, cluster secrets	Kubeconfig secrets per cluster
Community	CNCF graduated, wider adoption	CNCF graduated, GitLab-heavy shops

Use ArgoCD if you want a UI and centralized control. Use Flux if you prefer a more Kubernetes-native, controller-per-concern architecture.

Interview tip: When asked "ArgoCD vs Flux," the strongest answer acknowledges that both are CNCF graduated and production-ready. The real differentiator is operational model: ArgoCD is a centralized hub (one UI, one RBAC, multi-cluster), while Flux is decentralized (one instance per cluster, no built-in UI). Choose based on team topology, not features.

12. Image Updater

ArgoCD Image Updater watches container registries and updates the image tag in Git (or in-cluster) when new images are published.

# Annotation on the Application
metadata:
  annotations:
    argocd-image-updater.argoproj.io/image-list: myapp=ghcr.io/myorg/myapp
    argocd-image-updater.argoproj.io/myapp.update-strategy: semver
    argocd-image-updater.argoproj.io/myapp.allow-tags: regexp:^v[0-9]+\.[0-9]+\.[0-9]+$
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/git-branch: main

Update strategies: - semver: latest semver matching constraint - latest: highest lexicographically sorted tag - digest: track latest digest for a fixed tag (mutable tag tracking) - name: alphabetically newest tag

Quick Reference

# Install ArgoCD
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Get initial admin password
argocd admin initial-password -n argocd

# CLI login
argocd login argocd.example.com --sso
argocd login argocd.example.com --username admin --password $(argocd admin initial-password -n argocd | head -1)

# App operations
argocd app list
argocd app get my-app
argocd app diff my-app
argocd app sync my-app
argocd app sync my-app --dry-run
argocd app sync my-app --prune
argocd app history my-app
argocd app rollback my-app 3

# Force immediate reconciliation (bypass 3-min poll)
argocd app get my-app --refresh

# Create app from CLI
argocd app create my-app \
  --repo https://github.com/myorg/gitops \
  --path apps/my-app \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace my-app \
  --sync-policy automated \
  --auto-prune \
  --self-heal

# Delete app (and its resources if finalizer is set)
argocd app delete my-app

# Project operations
argocd proj list
argocd proj get team-payments
argocd proj create team-payments

# Cluster operations
argocd cluster list
argocd cluster add prod-us-east
argocd cluster rm prod-us-east

# Repository operations
argocd repo list
argocd repo add https://github.com/myorg/gitops --ssh-private-key-path ~/.ssh/id_rsa
argocd repo add https://charts.bitnami.com/bitnami --type helm --name bitnami

# Check sync status across all apps
argocd app list -o wide | grep -v Synced

# Port-forward UI locally
kubectl port-forward svc/argocd-server -n argocd 8080:443

---

Wiki Navigation

Prerequisites

• Kubernetes Ops (Production) (Topic Pack, L2)
• Git for DevOps (Topic Pack, L0)

Next Steps

• Progressive Delivery (Topic Pack, L2)

Pages that link here

• Anti-Primer: Argocd Gitops
• ArgoCD & GitOps
• Comparison: GitOps CD
• Git for DevOps
• Kubernetes Ops (Production)
• Production Readiness Review: Answer Key
• Production Readiness Review: Study Plans
• Progressive Delivery
• Thinking Out Loud: ArgoCD & GitOps