Kustomize - Street-Level Ops¶

Quick Diagnosis Commands¶

# --- Basic Build and Preview ---

# Preview rendered output (dry run — does not apply)
kubectl kustomize overlays/production/
kustomize build overlays/production/   # standalone kustomize CLI (same output)

# Apply kustomization to cluster
kubectl apply -k overlays/production/

# Diff current cluster state vs kustomize output
kubectl diff -k overlays/production/

# Apply and show what changed
kubectl apply -k overlays/production/ --dry-run=client

# --- Debugging Build Failures ---

# Verbose output to trace what's being included
kubectl kustomize overlays/production/ 2>&1 | head -100

# Check kustomization.yaml syntax
kustomize build overlays/production/ > /dev/null  # any error shows here

# List all resources that will be applied
kubectl kustomize overlays/production/ | grep -E "^kind:|^  name:"

# List images being referenced
kubectl kustomize overlays/production/ | grep "image:"

# --- Inspect the Kustomization Structure ---

# Show the kustomization.yaml being used
cat overlays/production/kustomization.yaml

# List resources included
kustomize build overlays/production/ | yq eval '.kind + "/" + .metadata.name' -

# Check what version of kustomize kubectl is using
kubectl version --client | grep -i kustomize
kustomize version

Common Scenarios¶

Scenario 1: Patch Not Being Applied¶

You added a strategic merge patch to the overlay but the rendered output doesn't show your changes.

# Step 1: Confirm the patch file is in kustomization.yaml
cat overlays/production/kustomization.yaml | grep -A10 patches

# Step 2: Build and check the output
kubectl kustomize overlays/production/ | grep -A30 "kind: Deployment"

# Step 3: Check patch file syntax — must match resource name exactly
cat overlays/production/increase-replicas.yaml
# Common mistake: wrong name/namespace — patch silently skips if no match
# (this is the #1 kustomize debugging issue — no error, no warning, just silence)

# Step 4: Validate that the base resource exists with the right name
kustomize build base/ | grep "name: myapp"
# Patch target name must match exactly

# Step 5: For JSON patches, check the path syntax
cat overlays/production/json-patch.yaml
# op: replace, path: /spec/replicas, value: 5
# Path must exist in the base document

# Example correct strategic merge patch:
cat > overlays/production/increase-replicas.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp           # MUST match base resource name exactly
  namespace: production # include namespace if base sets it
spec:
  replicas: 5
EOF
# Then in kustomization.yaml:
# patches:
#   - path: increase-replicas.yaml

Scenario 2: Image Tag Not Updating in Overlay¶

The images transformer in kustomization.yaml should be overriding the image tag, but pods are still using the old image.

# Step 1: Check the images section in kustomization.yaml
cat overlays/production/kustomization.yaml
# Should look like:
# images:
#   - name: myapp
#     newTag: v1.2.3
# OR
#   - name: docker.io/myorg/myapp
#     newTag: v1.2.3

# Step 2: Check what image name is in the base deployment
kustomize build base/ | grep "image:"
# The 'name' in images transformer must match the image name in the base EXACTLY
# If base has: image: docker.io/myorg/myapp:v1.0.0
# Then name must be: docker.io/myorg/myapp (not just myapp)

# Step 3: Build the overlay and check the image
kustomize build overlays/production/ | grep "image:"
# Verify the tag changed

# Step 4: Correct form with full registry path
# images:
#   - name: docker.io/myorg/myapp
#     newName: ghcr.io/myorg/myapp   # optional: change registry
#     newTag: v1.2.3
#     digest: sha256:abc123           # optional: pin by digest

Scenario 3: ConfigMap Not Regenerating After Content Change¶

You're using configMapGenerator but the ConfigMap name hash isn't changing after editing the source file.

# Step 1: Check your kustomization.yaml configMapGenerator
cat overlays/production/kustomization.yaml
# configMapGenerator:
#   - name: myapp-config
#     files:
#       - configs/app.properties
#     options:
#       disableNameSuffixHash: false   # must be false (default) for auto-rotation

# Step 2: Build and check the generated name
kustomize build overlays/production/ | grep "name: myapp-config"
# Should have a hash suffix like: myapp-config-g74bgt5d9m

# Step 3: If hash changed but pods aren't updating
# Kustomize updates the ConfigMap name, and patches references in Deployments
# But you need to ensure the Deployment references it via name (not hardcoded)
kustomize build overlays/production/ | grep -A20 "kind: Deployment" | grep "configMapRef\|configMap"

# Step 4: Confirm the deployment was also updated to reference the new ConfigMap name
kustomize build overlays/production/ | yq eval 'select(.kind == "Deployment") | .spec.template.spec.volumes'

# Step 5: If disableNameSuffixHash is true, force a rollout manually
kubectl rollout restart deployment/myapp -n production

Scenario 4: Namespace Not Applied to Resources¶

Resources are being applied to the wrong namespace or no namespace.

# Step 1: Add namespace to kustomization.yaml
cat overlays/production/kustomization.yaml
# namespace: production   <-- add this at top level

# Step 2: Build and verify namespace is set
kustomize build overlays/production/ | grep "namespace:"

# Step 3: Check if base resources have hardcoded namespaces that override
kustomize build base/ | grep "namespace:"
# Kustomize namespace field in kustomization.yaml overrides base namespaces

# Step 4: For cluster-scoped resources (ClusterRole, etc.) namespace is ignored
# that's expected behavior

# Step 5: Add namespace prefix to avoid conflicts
# In kustomization.yaml:
# namePrefix: prod-
# nameSuffix: -v2

Key Patterns¶

Directory Layout (Base + Overlays)¶

k8s/
├── base/
│   ├── kustomization.yaml    # lists resources
│   ├── deployment.yaml
│   ├── service.yaml
│   └── configmap.yaml
└── overlays/
    ├── development/
    │   ├── kustomization.yaml  # patches for dev
    │   └── replicas.yaml
    ├── staging/
    │   └── kustomization.yaml
    └── production/
        ├── kustomization.yaml
        ├── increase-replicas.yaml
        └── resource-limits.yaml

# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
  - configmap.yaml

# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: production
bases:
  - ../../base
patches:
  - path: increase-replicas.yaml
  - path: resource-limits.yaml
images:
  - name: myapp
    newTag: v1.2.3
configMapGenerator:
  - name: myapp-env
    envs:
      - env.properties

Strategic Merge Patch vs JSON Patch¶

# Strategic Merge Patch — good for adding/replacing fields
# File: overlays/production/resource-limits.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  template:
    spec:
      containers:
        - name: myapp       # must match container name in base
          resources:
            requests:
              cpu: "500m"
              memory: "256Mi"
            limits:
              cpu: "1"
              memory: "512Mi"

# JSON Patch — good for precise array element operations
# File: overlays/production/json-patch.yaml
- op: replace
  path: /spec/replicas
  value: 5
- op: add
  path: /spec/template/spec/containers/0/env/-
  value:
    name: ENV
    value: production

# Reference the patch type in kustomization.yaml
patches:
  - path: resource-limits.yaml
    target:
      kind: Deployment
      name: myapp
  - patch: |-     # inline patch
      - op: replace
        path: /spec/replicas
        value: 5
    target:
      kind: Deployment
      name: myapp

SecretGenerator (Don't Commit Secrets to Git)¶

# kustomization.yaml — generate from env file (gitignored)
secretGenerator:
  - name: myapp-secrets
    envs:
      - secrets.env    # gitignored file: KEY=value
    options:
      disableNameSuffixHash: false

# secrets.env (gitignored)
DB_PASSWORD=supersecret
API_KEY=abc123

# Build shows the base64-encoded secret (still sensitive!)
kustomize build overlays/production/ | grep -A5 "kind: Secret"

# Better: use External Secrets Operator instead of committing even encoded secrets

Kustomize with Helm (Helm Inflation)¶

# kustomization.yaml — inflate a Helm chart as a base
helmCharts:
  - name: prometheus
    repo: https://prometheus-community.github.io/helm-charts
    version: 25.3.1
    namespace: monitoring
    valuesFile: prometheus-values.yaml

# Then apply kustomize patches on top of the helm output
patches:
  - path: prometheus-retention-patch.yaml

Gotcha: bases is deprecated in kustomize v2.1+ — use resources instead. Both work today, but bases emits a deprecation warning that breaks strict CI pipelines. Replace bases: [../../base] with resources: [../../base] in your kustomization.yaml files.

Debug clue: When kubectl diff -k overlays/production/ shows no diff but kubectl apply -k changes resources anyway, the likely cause is server-side fields that kustomize does not manage (e.g., annotations added by admission webhooks). Use --server-side apply to avoid fighting over unmanaged fields.