Secrets Management¶
40 cards — 🟢 8 easy | 🟡 11 medium | 🔴 8 hard
🟢 Easy (8)¶
1. Why are Kubernetes Secrets not truly secure by default?
Show answer
Kubernetes Secrets are only base64-encoded, not encrypted. Anyone with kubectl get secret access can decode them. They also lack encryption at rest in etcd unless explicitly enabled.Remember: "Vault = secrets API, not a file." Apps request secrets at runtime instead of reading config files.
Fun fact: HashiCorp Vault can generate dynamic credentials — each app gets unique, short-lived database passwords.
2. How do Sealed Secrets work at a high level?
Show answer
Developers encrypt secrets with a cluster-specific public key using kubeseal. The encrypted SealedSecret YAML is safe to commit to Git. The Sealed Secrets controller in the cluster decrypts it into a regular Kubernetes Secret.Gotcha: Seal/unseal is Vault's cold start problem. Vault starts sealed and needs N-of-M key shares to unseal.
Remember: "Sealed = encrypted, Unsealed = ready. Auto-unseal delegates to a KMS."
3. Why should you never store secrets in a Git repository, even a private one?
Show answer
Secrets in Git persist in the repository history forever, even after deletion. Anyone with repo access (including future contributors and compromised accounts) can recover them.Remember: "KV v2 = versioned secrets." You can retrieve previous versions and soft-delete secrets.
Example: vault kv put secret/myapp password=s3cret && vault kv get -version=1 secret/myapp
4. What does it mean to seal and unseal HashiCorp Vault?
Show answer
Vault starts in a sealed state where it cannot read its encrypted storage. Unsealing requires a threshold of key shares (Shamir's Secret Sharing) — typically 3 of 5 shares. Once unsealed, the master key decrypts the encryption key in memory. Auto-unseal via cloud KMS (AWS, GCP, Azure) eliminates manual unseal but requires the KMS to be available. If Vault restarts, it re-seals and must be unsealed again.5. What are three best practices for handling secrets in Kubernetes beyond the defaults?
Show answer
1) Enable encryption at rest in etcd (EncryptionConfiguration with aescbc or secretbox provider).2) Use RBAC to restrict who can read secrets (most pods should not have kubectl get secret access).
3) Mount secrets as files, not environment variables — env vars appear in process listings (ps aux), crash dumps, and child processes. Files are readable only by the container's filesystem permissions.
6. What qualifies as a "secret" in infrastructure management?
Show answer
Passwords, API keys, tokens, certificates, SSH private keys, and database credentials. Anything that grants access to a system or service if exposed.Remember: "SOPS = Secrets OPerationS." It encrypts values in YAML/JSON files, leaving keys readable for git diffs.
Example: sops -e secrets.yaml > secrets.enc.yaml — encrypts values with KMS or PGP.
7. Name three places where secrets should NEVER be stored.
Show answer
Git repositories (even private ones), Dockerfiles or container images, and plaintext config files on servers. Also avoid log files and environment variable dumps in error pages.Remember: "Never store secrets in environment variables visibly." Use /proc/PID/environ exposure risk.
Gotcha: docker inspect shows env vars in plaintext. Use Docker secrets or volume-mounted files instead.
8. What are three appropriate places to store secrets?
Show answer
Dedicated secrets managers (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager), environment variables for runtime injection, and encrypted config files (Ansible Vault, SOPS).Remember: "Rotate early, rotate often." Short-lived credentials limit blast radius. If a secret leaks, damage is time-bounded.
Example: AWS STS temporary credentials expire in 1-12 hours by default.
🟡 Medium (11)¶
1. What are the four key concepts in HashiCorp Vault?
Show answer
Secret engine (stores/generates secrets: kv, database, pki, transit), Auth method (how clients authenticate: kubernetes, OIDC, AppRole), Policy (what secrets a client can access), Lease (TTL on dynamic secrets, auto-revoked when expired).Remember: "Dynamic secrets = generated on demand, auto-expired." No shared passwords, no rotation burden.
Example: vault read database/creds/my-role returns a unique username/password with a 1-hour TTL.
2. How does the External Secrets Operator (ESO) work?
Show answer
ESO syncs secrets from external stores (Vault, AWS Secrets Manager, GCP Secret Manager) into Kubernetes Secrets. An ExternalSecret resource defines which remote secret to fetch, and ESO auto-refreshes it based on a configurable refreshInterval.3. How does SOPS encrypt secrets differently from Sealed Secrets?
Show answer
SOPS encrypts specific values within YAML/JSON files while leaving keys readable, using age, PGP, or cloud KMS. It works at the file level in CI/CD pipelines, while Sealed Secrets is Kubernetes-native and encrypts entire secret data fields.4. Why does changing a secret in Vault or AWS Secrets Manager not automatically update running pods?
Show answer
The pods were started with the old secret value. Updating the external store does not restart pods. You need ESO's refreshInterval to update the Kubernetes Secret, plus a mechanism (like a rollout restart or Reloader) to restart the pods.5. How does AWS Secrets Manager differ from AWS Parameter Store for secrets?
Show answer
Secrets Manager: built-in automatic rotation, cross-account sharing, higher cost (~$0.40/secret/month), native RDS/Redshift integration.Parameter Store: free tier (standard), no built-in rotation (requires custom Lambda), simpler API, supports both secrets and config. Use Secrets Manager for database credentials needing rotation. Use Parameter Store for application config and secrets where you manage rotation yourself.
6. Why is automated secret rotation critical and what are the key patterns?
Show answer
Static secrets that never rotate accumulate risk — any leak or ex-employee compromise persists indefinitely. Rotation patterns:1) Dual-secret: create new secret, update consumers, delete old (requires brief overlap).
2) Dynamic: Vault generates per-consumer credentials with TTL (no rotation needed, secrets expire).
3) Grace period: new and old secrets both valid during transition. Automate via Secrets Manager rotation Lambdas or Vault's built-in rotation.
7. How do Vault dynamic database secrets work end-to-end?
Show answer
1) Configure the database secrets engine with a connection string and allowed roles.2) Define a role with a creation statement (SQL template for creating users) and TTL.
3) Application requests credentials: vault read database/creds/myapp-role.
4) Vault creates a temporary database user with the role's permissions and returns credentials.
5) When the TTL expires, Vault automatically revokes the user. Each pod gets unique, short-lived credentials — a leak affects only one consumer for a limited time.
8. How can you detect secrets that were accidentally committed to a Git repository?
Show answer
Use trufflehog (trufflehog git file://./my-repo --only-verified) to scan the entire git history for verified secrets. You can also use "git log --all -p | grep -iE 'password|secret|api_key|token'" for a quick manual check, though this produces more false positives.9. What is SOPS, and how does it protect secrets in config files?
Show answer
SOPS (Secrets OPerationS) encrypts config files in place while keeping the structure (keys) visible and only encrypting the values. This lets you version config files in git safely. Encrypt with "sops --encrypt --in-place config.yaml" and decrypt with "sops --decrypt config.yaml". SOPS supports AWS KMS, GCP KMS, Azure Key Vault, and PGP as encryption backends.10. How do you check when a TLS certificate expires, and why is automating certificate renewal important?
Show answer
Check with: openssl s_client -connect example.com:443 2>/dev/null | openssl x509 -noout -dates. Automating renewal (e.g., with certbot renew) is critical because expired certificates cause outages — services become unreachable when clients reject the expired cert.11. What is the recommended SSH key type to generate, and how do you deploy it to a server?
Show answer
Generate with "ssh-keygen -t ed25519 -C user@company.com" (ed25519 is shorter, faster, and more secure than RSA). Deploy to a server with "ssh-copy-id -i ~/.ssh/id_ed25519.pub user@server". Use an SSH agent (ssh-add) to avoid typing the passphrase repeatedly.🔴 Hard (8)¶
1. What are dynamic secrets in Vault and why are they more secure than static secrets?
Show answer
Dynamic secrets are generated on demand with a TTL and automatically revoked when the lease expires. For example, Vault can create a temporary database credential per pod that expires in 1 hour, so there is no long-lived password to leak.2. How does Vault's Kubernetes auth method work?
Show answer
A pod authenticates to Vault using its ServiceAccount JWT token. Vault validates the token against the Kubernetes API, checks the ServiceAccount name and namespace against a configured role, and returns a Vault token scoped to the role's policy.3. What are the three scope levels in Sealed Secrets and when would you use each?
Show answer
Strict (default): sealed secret is bound to a specific name and namespace. Namespace-wide: can be renamed within the same namespace. Cluster-wide: can be used anywhere. Use strict for maximum security, namespace-wide for flexibility within a team, cluster-wide only for truly shared secrets.4. How do Vault access policies work and what is the principle of least privilege?
Show answer
Vault policies are path-based ACL rules granting capabilities (create, read, update, delete, list) on secret paths.Example: path "secret/data/myapp/*" { capabilities = ["read"] } grants read-only access to myapp secrets. Assign policies to auth method roles (e.g., Kubernetes ServiceAccount "myapp-sa" gets the "myapp-read" policy). Never grant root policy to applications. Use deny by default — only grant the minimum paths each consumer needs.
5. What is a break-glass procedure for secrets management and why is it necessary?
Show answer
Break-glass provides emergency access when normal secret retrieval fails (Vault down, KMS outage, ESO broken). Store emergency credentials in a sealed envelope or hardware security module with strict access logging. Requirements: dual-person authorization, automatic audit trail, time-limited access, and mandatory post-incident rotation of any secrets accessed via break-glass. Test the procedure quarterly.6. Why is the most common attack vector leaked credentials rather than sophisticated exploits, and what does this imply?
Show answer
Because credentials in git repos, logs, or error pages are trivially discoverable with automated scanning tools. This means guarding secrets through proper storage, rotation, and leak detection (CI secret scanning) provides more security ROI than defending against zero-days. Prevention is about process and tooling, not exotic defenses.7. How do you verify the full TLS certificate chain for a domain, and what does an incomplete chain cause?
Show answer
Use "openssl s_client -connect example.com:443 -showcerts" to see every certificate in the chain from server cert to root CA. An incomplete chain (missing intermediate certificate) causes some clients to reject the connection with a certificate validation error even though the server cert itself is valid — different clients have different trust store behaviors.8. Why is a regular patching schedule better than reactive patching, and what role does vulnerability scanning play?