Skip to content

Secrets Management - Street-Level Ops

Real-world patterns and gotchas from production secrets management.

Quick Diagnosis Commands

# Check if a secret exists and its age
kubectl get secret <name> -n <ns> -o jsonpath='{.metadata.creationTimestamp}'

# Decode a specific key
kubectl get secret <name> -n <ns> -o jsonpath='{.data.<key>}' | base64 -d

# List all secrets in a namespace (names only, no values)
kubectl get secrets -n <ns>

# Check Sealed Secrets controller status
kubectl logs -n kube-system deploy/sealed-secrets-controller --tail=50

# Check ESO sync status
kubectl get externalsecret -n <ns>

# Vault: check auth health
kubectl exec -n vault vault-0 -- vault status

Gotcha: Pods Don't Pick Up Secret Changes

Kubernetes does NOT restart pods when a Secret changes. The pod still has the old value.

Workarounds: 

# Option 1: Rolling restart
kubectl rollout restart deployment/<name> -n <ns>

# Option 2: Use Reloader (auto-restarts on secret change)
# https://github.com/stakater/Reloader
helm install reloader stakater/reloader -n kube-system

# Annotate your deployment
metadata:
  annotations:
    reloader.stakater.com/auto: "true"

# Option 3: Volume-mounted secrets auto-update (but app must re-read)
# Kubelet refreshes mounted secrets every ~60s

Gotcha: Sealed Secrets Key Backup

If you lose the Sealed Secrets controller key, all your SealedSecrets are permanently unrecoverable.

# BACK UP THE KEY
kubectl get secret -n kube-system sealed-secrets-key -o yaml > sealed-secrets-key-backup.yaml

# Store this backup in a secure location (NOT in the same Git repo)
# Vault, AWS Secrets Manager, or a physical safe

Under the hood: Kubernetes Secrets are stored in etcd as base64-encoded plaintext by default. Anyone with kubectl get secret RBAC permission can read them. Enable etcd encryption at rest (--encryption-provider-config) to encrypt Secrets in etcd with AES-CBC or AES-GCM. Without this, a compromised etcd backup exposes every Secret in the cluster.

Gotcha: ESO RefreshInterval vs Pod Restart

ESO updates the K8s Secret on refreshInterval, but pods don't see it unless: 1. The secret is mounted as a volume (kubelet refreshes ~60s) 2. You use Reloader to trigger a rollout 3. Your app watches the file for changes

Best practice: Set refreshInterval: 1h and use Reloader for automatic pod restarts.

Pattern: Secret Rotation Checklist

  1. Generate new credential in external store (Vault/AWS SM)
  2. Verify ESO syncs the new value: kubectl get externalsecret -n <ns>
  3. Rolling restart the deployment
  4. Verify app works with new credential
  5. Revoke old credential in external store
  6. Check no other services used the old credential

Pattern: Namespace Isolation for Secrets

# SecretStore per namespace (recommended over ClusterSecretStore)
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: team-secrets
  namespace: team-a
spec:
  provider:
    vault:
      server: https://vault.example.com
      path: secret/data/team-a  # Scoped path

This ensures Team A can only access their own secrets in Vault.

Pattern: Multi-Tenant Secret Access

# Vault policy per team
vault policy write team-a - <<EOF
path "secret/data/team-a/*" {
  capabilities = ["read", "list"]
}
path "secret/data/shared/*" {
  capabilities = ["read"]
}
EOF

Gotcha: Vault auto-unseal (via AWS KMS, Azure Key Vault, or Transit) generates recovery keys instead of unseal keys. Recovery keys CANNOT unseal Vault if the auto-unseal provider is unavailable — they are authorization-only. If your KMS key is deleted, Vault is permanently sealed. Always back up both the recovery keys AND ensure KMS key deletion protection is enabled.

Emergency: Secret Leaked

# 1. ROTATE IMMEDIATELY (don't investigate first)
# Rotate the credential in the external store

# 2. Check who accessed it
kubectl get events -n <ns> --field-selector reason=Get,involvedObject.name=<secret-name>

# 3. Audit Git history
git log --all -p -- '*.yaml' | grep -i password  # Check for committed secrets

# 4. If in Git, clean history (git-filter-repo is the recommended tool, faster than BFG)
pip install git-filter-repo
git filter-repo --path-glob '*.secret.yaml' --invert-paths

# 5. Add prevention
git secrets --install
git secrets --register-aws

Anti-Pattern: Environment Variables for Secrets

# AVOID: secrets visible in kubectl describe, process listing, crash dumps
env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-creds
        key: password

# PREFER: volume-mounted secrets
volumes:
  - name: secrets
    secret:
      secretName: db-creds
containers:
  - name: app
    volumeMounts:
      - name: secrets
        mountPath: /etc/secrets
        readOnly: true
    # App reads from /etc/secrets/password

Volume-mounted secrets: auto-update, not in kubectl describe, not in /proc/*/environ.

War story: A team stored database passwords as environment variables. During an incident, an engineer ran kubectl describe pod and pasted the output into a public Slack channel. The password was visible in the Environment section. After rotating the credential, they switched to volume-mounted secrets and added kubectl describe output to their secret-scanning tool's detection patterns.

Quick Reference