Skip to content

Portal | Level: L2: Operations | Topics: Secrets Management | Domain: Security

Secrets Management Drills

Remember: Secrets management layers: Storage (where secrets live — Vault, AWS Secrets Manager, SOPS), Delivery (how they reach the app — env vars, volume mounts, CSI driver), Rotation (how and when they change — auto-rotate, short TTL). All three layers must be covered. Most breaches exploit the delivery layer — secrets logged, leaked via env vars in crash dumps, or exposed in kubectl describe.

Gotcha: Environment variables containing secrets show up in kubectl describe pod, docker inspect, /proc/PID/environ, and crash dump logs. Volume-mounted secrets (files) do not appear in any of these. For sensitive values, prefer volume mounts over env vars.

Default trap: kubectl create secret generic stores values as base64 in etcd. Without etcd encryption at rest enabled (--encryption-provider-config), anyone with etcd access reads all secrets in plaintext. Many clusters skip this step during setup.

Drill 1: Decode a Kubernetes Secret

Difficulty: Easy

Q: A Secret named db-creds exists in namespace production. How do you view the decoded value of the password key?

Answer 
kubectl get secret db-creds -n production -o jsonpath='{.data.password}' | base64 -d
Remember: Kubernetes Secrets are base64-encoded, not encrypted. Anyone with RBAC access to read Secrets can decode them.

Drill 2: Seal a Secret

Difficulty: Easy

Q: Using Sealed Secrets, create an encrypted secret for DB_PASSWORD=hunter2 that's safe to commit to Git.

Answer 
# Create a regular secret (dry-run) and pipe to kubeseal
kubectl create secret generic db-creds \
  --from-literal=DB_PASSWORD=hunter2 \
  --dry-run=client -o yaml | \
  kubeseal --format yaml > sealed-db-creds.yaml

# The output is safe to commit
git add sealed-db-creds.yaml
git commit -m "Add sealed DB credentials"
The SealedSecret can only be decrypted by the Sealed Secrets controller in the target cluster.

Drill 3: External Secrets Operator

Difficulty: Medium

Q: Write an ExternalSecret that fetches prod/api-key from AWS Secrets Manager and creates a K8s Secret named api-credentials with the key API_KEY.

Answer 
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-store
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: eso-sa
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: api-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-store
  target:
    name: api-credentials
    creationPolicy: Owner
  data:
  - secretKey: API_KEY
    remoteRef:
      key: prod/api-key
ESO polls every `refreshInterval` and updates the K8s Secret automatically.

Drill 4: SOPS Encryption

Difficulty: Medium

Q: You have a secret.yaml file with plaintext secrets. Encrypt it using SOPS with an AWS KMS key and configure .sops.yaml for the project.

Answer 
# .sops.yaml — project config
creation_rules:
  - path_regex: .*\.secret\.yaml$
    kms: arn:aws:kms:us-east-1:123456789:key/abc-def-123
  - path_regex: .*\.secret\.env$
    kms: arn:aws:kms:us-east-1:123456789:key/abc-def-123

# Encrypt
sops --encrypt secret.yaml > secret.secret.yaml

# Decrypt (for viewing)
sops --decrypt secret.secret.yaml

# Edit encrypted file in-place
sops secret.secret.yaml

# With SOPS, only values are encrypted, keys remain readable
# This makes diffs meaningful in Git

Drill 5: Vault Kubernetes Auth

Difficulty: Hard

Q: Configure Vault's Kubernetes auth method so that pods with ServiceAccount my-app in namespace production can read secrets at secret/data/production/*.

Answer 
# 1. Enable Kubernetes auth in Vault
vault auth enable kubernetes

# 2. Configure it to talk to the K8s API
vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc:443"

# 3. Create a policy
vault policy write my-app-policy - <<EOF
path "secret/data/production/*" {
  capabilities = ["read"]
}
EOF

# 4. Create a role binding the SA to the policy
vault write auth/kubernetes/role/my-app \
  bound_service_account_names=my-app \
  bound_service_account_namespaces=production \
  policies=my-app-policy \
  ttl=1h
Pods with SA `my-app` in `production` namespace can now authenticate to Vault and read `secret/data/production/*`.

Drill 6: Vault Agent Injection

Difficulty: Medium

Q: Add Vault Agent annotations to a Deployment so the database password from secret/data/db-creds is mounted at /vault/secrets/db-password inside the pod.

Answer 
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "my-app"
        vault.hashicorp.com/agent-inject-secret-db-password: "secret/data/db-creds"
        vault.hashicorp.com/agent-inject-template-db-password: |
          {{- with secret "secret/data/db-creds" -}}
          {{ .Data.data.password }}
          {{- end -}}
The Vault Agent sidecar: 1. Authenticates to Vault using the pod's ServiceAccount 2. Fetches the secret 3. Renders it to `/vault/secrets/db-password` 4. Keeps it updated if the secret changes

Drill 7: Secret Rotation

Difficulty: Hard

Q: You need to rotate a database password used by 5 microservices with zero downtime. Describe the process.

Answer 
1. Generate new password

2. Configure DB to accept BOTH old and new passwords
   - Most DBs: create a new user or ALTER USER with new password
   - Some DBs support multiple valid passwords

3. Update the secret source (Vault / AWS SM / K8s Secret)
   - Update with new password

4. Rolling restart all consuming services
   kubectl rollout restart deployment svc-1 svc-2 svc-3 svc-4 svc-5 -n prod

5. Verify all services are using the new password
   - Check logs for auth errors
   - Monitor error rates

6. Revoke old password
   - ALTER USER app_user PASSWORD 'new_password'
   - (or delete the old user if using dual-user strategy)

7. Verify no service is still using old password
For automated rotation: - Vault Dynamic Secrets: Vault generates short-lived credentials per pod - ESO + AWS Secrets Manager: Auto-rotation with Lambda - Sealed Secrets: Re-seal and commit, ArgoCD syncs

Drill 8: etcd Encryption at Rest

Difficulty: Hard

Q: Kubernetes Secrets are stored in etcd as base64 — not encrypted. How do you enable encryption at rest?

Answer 
# /etc/kubernetes/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: $(head -c 32 /dev/urandom | base64)
  - identity: {}  # Fallback for reading old unencrypted secrets

# Add to kube-apiserver manifest:
# --encryption-provider-config=/etc/kubernetes/encryption-config.yaml

# Re-encrypt all existing secrets:
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
After enabling: - New secrets are encrypted with AES-CBC - Old secrets remain unencrypted until replaced - The `identity: {}` provider allows reading old unencrypted data For cloud clusters: Use KMS provider instead of local keys (AWS KMS, GCP KMS, Azure Key Vault).

Drill 9: Audit Secret Access

Difficulty: Medium

Q: How do you audit who accessed a specific Kubernetes Secret?

Answer 
# Enable audit logging in kube-apiserver
# /etc/kubernetes/audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: RequestResponse
  resources:
  - group: ""
    resources: ["secrets"]
  verbs: ["get", "list", "watch"]

# Add to kube-apiserver:
# --audit-policy-file=/etc/kubernetes/audit-policy.yaml
# --audit-log-path=/var/log/kubernetes/audit.log

# Search audit logs
grep "db-creds" /var/log/kubernetes/audit.log | jq '{
  user: .user.username,
  verb: .verb,
  resource: .objectRef.name,
  namespace: .objectRef.namespace,
  time: .requestReceivedTimestamp
}'
Also check RBAC: 
# Who can read secrets in production?
kubectl auth can-i get secrets -n production --as=system:serviceaccount:default:my-app
kubectl auth can-i list secrets -n production --list

Drill 10: Pre-commit Secret Detection

Difficulty: Easy

Q: Set up a pre-commit hook to prevent accidentally committing secrets to Git.

Answer 
# .pre-commit-config.yaml
repos:
- repo: https://github.com/Yelp/detect-secrets
  rev: v1.4.0
  hooks:
  - id: detect-secrets
    args: ['--baseline', '.secrets.baseline']

# Or using git-secrets (AWS-focused):
- repo: https://github.com/awslabs/git-secrets
  rev: master
  hooks:
  - id: git-secrets

# Initialize detect-secrets baseline
detect-secrets scan > .secrets.baseline

# Install the hook
pre-commit install

# Test it
echo "AWS_SECRET_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE" > test.txt
git add test.txt && git commit -m "test"
# → Should be blocked by the hook
Also add to CI pipeline: 
# GitHub Actions
- name: Detect secrets
  uses: Yelp/detect-secrets-action@master

Wiki Navigation

Prerequisites

Related Content