Skip to content

Secrets Management Without Tears

  • lesson
  • env-vars
  • files
  • hashicorp-vault
  • k8s-secrets
  • external-secrets
  • rotation ---# Secrets Management Without Tears

Topics: env vars, files, HashiCorp Vault, K8s secrets, external-secrets, rotation Level: L1–L2 (Foundations → Operations) Time: 45–60 minutes Prerequisites: None

The Mission

You're reviewing a PR and find this:

DB_PASSWORD = "production-p@ssw0rd-2026"

Hardcoded in the source code. Committed to Git. In every clone, every fork, every backup, forever. The password is compromised the moment it's pushed. Even if you delete it in the next commit, git log remembers.

This lesson builds up from the worst practices to the best, level by level. Each level solves the previous level's problem.

Level 0: Hardcoded in Code (Never Do This)

# BAD — committed to Git, visible to everyone, lives forever
DATABASE_URL = "postgresql://admin:s3cr3t@db.example.com:5432/myapp"

Why it's terrible: source code is shared, versioned, cloned, and often public. Secrets in code are the #1 finding in security audits and the #1 cause of credential leaks on GitHub.

War Story: Uber's 2016 breach started with AWS credentials committed to a private GitHub repository. Attackers found them, accessed an S3 bucket with 57 million user records, and Uber paid $100K in hush money. Fine: $148 million.

Level 1: Environment Variables

export DATABASE_URL="postgresql://admin:s3cr3t@db.example.com:5432/myapp"

import os
DATABASE_URL = os.environ["DATABASE_URL"]

Better than hardcoded — the secret isn't in source code. But environment variables are visible in /proc/<pid>/environ, docker inspect, ps eww, and any process that reads the environment.

# Anyone with access to the container can see all env vars
docker inspect mycontainer | jq '.[0].Config.Env'
# → ["DATABASE_URL=postgresql://admin:s3cr3t@..."]

# Or from inside
cat /proc/1/environ | tr '\0' '\n'

Level 2: Files with Restricted Permissions

# Secret in a file, readable only by the app user
echo "s3cr3t" > /etc/myapp/db_password
chmod 600 /etc/myapp/db_password
chown appuser:appuser /etc/myapp/db_password

with open("/etc/myapp/db_password") as f:
    DB_PASSWORD = f.read().strip()

Files with 600 permissions are safer than env vars — they're not in docker inspect or /proc/environ. But they're still plaintext on disk, and anyone with root access can read them.

In Docker, mount the secret as a file:

docker run -v /secrets/db_password:/run/secrets/db_password:ro myapp

Level 3: Kubernetes Secrets

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  password: czNjcjN0  # base64-encoded (NOT encrypted)

# Mount as file in pod
containers:
  - name: myapp
    volumeMounts:
      - name: db-creds
        mountPath: /run/secrets
        readOnly: true
volumes:
  - name: db-creds
    secret:
      secretName: db-credentials

Gotcha: Kubernetes Secrets are base64-encoded, NOT encrypted. Anyone with kubectl get secret -o yaml can decode them. They're stored in etcd, which may or may not be encrypted at rest (depends on your cluster configuration). Enable etcd encryption: --encryption-provider-config on the API server.

Quick Check: Levels 0-3

Q: Your K8s Secret has password: czNjcjN0. Is this encrypted?

No. It's base64-encoded — echo czNjcjN0 | base64 -d = s3cr3t. Anyone with kubectl get secret -o yaml can decode it.

Q: Environment variables — are they safer than hardcoded passwords?

Marginally. They're not in Git, but they're visible via docker inspect, /proc/PID/environ, and ps eww. File mounts with chmod 600 are safer.

Level 4: HashiCorp Vault (Dynamic Secrets)

Vault is a dedicated secrets management system. The key innovation: dynamic secrets. Instead of storing a static password, Vault creates temporary credentials on demand:

# App requests database credentials from Vault
vault read database/creds/my-role
# → username: v-app-my-role-abc123
# → password: randomized-password-xyz
# → lease_duration: 1h

# After 1 hour, the credentials expire automatically
# No manual rotation needed

The app gets fresh credentials every hour. If credentials are compromised, they expire on their own. No one knows the "real" database password — because there isn't one.

Level 5: External Secrets Operator (Kubernetes + Vault/AWS/GCP)

Bridge between Vault (or AWS Secrets Manager, GCP Secret Manager) and Kubernetes Secrets:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: db-credentials
  data:
    - secretKey: password
      remoteRef:
        key: secret/data/myapp/database
        property: password

The operator syncs secrets from Vault/AWS into Kubernetes Secrets automatically. The secret source of truth is Vault, not Kubernetes. Rotation happens in Vault and propagates to K8s.

Secret Rotation: The Part Everyone Skips

A secret that never rotates is a ticking time bomb. If it's compromised, the attacker has access forever. Rotation limits the damage window.

Secret type Rotation frequency How to rotate
Database passwords 30-90 days Vault dynamic secrets (automatic)
API keys 90 days Generate new key, update references, revoke old
TLS certificates 90 days (moving toward 47) cert-manager (automatic)
SSH keys 6-12 months SSH certificates with short TTLs (automatic)
Encryption keys 1 year Key versioning (new key encrypts, old key decrypts)

Flashcard Check

Q1: Environment variables are visible where?

/proc/<pid>/environ, docker inspect, ps eww. Anyone with container or process access can read them.

Q2: Kubernetes Secrets are encrypted, right?

No. Base64-encoded, not encrypted. Enable etcd encryption separately. Anyone with kubectl get secret -o yaml can decode them.

Q3: What are dynamic secrets?

Temporary credentials generated on demand (e.g., by Vault). They expire automatically after a TTL. No static password to rotate or leak.

Q4: External Secrets Operator — what does it do?

Syncs secrets from Vault/AWS/GCP into Kubernetes Secrets automatically. Source of truth stays in the secrets manager, K8s secrets are kept in sync.

Cheat Sheet

The Ladder

Level Where secrets live Risk
0: In code Git history Permanent compromise
1: Env vars Process environment Visible via inspect/proc
2: Files (600) Filesystem Root can read
3: K8s Secrets etcd (base64) Not encrypted by default
4: Vault Encrypted, audited, dynamic Vault itself becomes critical
5: External Secrets Vault → K8s (synced) Best of both worlds

Takeaways

  1. Never commit secrets to Git. Even private repos. Even "just for now." Git history is forever.

  2. Environment variables are not secure. They're visible via docker inspect and /proc. Use file mounts with restricted permissions.

  3. Kubernetes Secrets are base64, not encrypted. Enable etcd encryption. Use RBAC to restrict who can read them.

  4. Dynamic secrets are the gold standard. Vault generates temporary credentials that expire automatically. No rotation needed.

  5. Rotate everything on a schedule. A never-rotated secret is a permanent access token for anyone who finds it.

Exercises

  1. Prove environment variables are visible. Run a container with a "secret" env var: docker run -d --name secret-test -e MY_SECRET=hunter2 nginx. Use docker inspect secret-test | grep MY_SECRET to see the secret in plaintext. Then exec in and check /proc/1/environ: docker exec secret-test cat /proc/1/environ | tr '\0' '\n' | grep MY_SECRET. Clean up with docker rm -f secret-test. This demonstrates why env vars are not secure for sensitive data.

  2. Mount a secret as a file with restricted permissions. Create a secret file: echo "my-db-password" > /tmp/test-secret && chmod 600 /tmp/test-secret. Mount it into a container: docker run -d --name file-secret -v /tmp/test-secret:/run/secrets/db_password:ro nginx. Verify it's readable inside: docker exec file-secret cat /run/secrets/db_password. Verify it does NOT appear in docker inspect file-secret | grep my-db-password (the Env section won't contain it). Clean up with docker rm -f file-secret && rm /tmp/test-secret.

  3. Decode a Kubernetes Secret. Create a K8s secret (or use kubectl locally if available): kubectl create secret generic test-creds --from-literal=password=s3cr3t. Retrieve it: kubectl get secret test-creds -o jsonpath='{.data.password}'. Decode the base64 value: echo "<value>" | base64 -d. Confirm this shows the original plaintext. This proves K8s Secrets are encoded, not encrypted. Clean up with kubectl delete secret test-creds.

  4. Scan a repo for hardcoded secrets. Clone any public repo (or use a local one). Install a secret scanning tool like trufflehog (pip install trufflehog3) or gitleaks. Run it against the repo and review the findings. If no real secrets are found, create a test file with a fake AWS key pattern (AKIA...) and confirm the scanner detects it. Delete the test file.

  5. Implement secret rotation with short-lived tokens. Write a shell script that generates a random password (openssl rand -base64 32), writes it to a file, and schedules itself to run again in 60 seconds (using a loop with sleep). Start a second script that reads the file every 10 seconds and prints the current value. Observe the password changing. This simulates the dynamic secrets model where credentials rotate automatically.

  • Permission Denied — when RBAC blocks secret access
  • The Container Escape — when Docker socket access exposes secrets
  • The Git Disaster Recovery Guide — when secrets are committed to Git