Skip to content

Portal | Level: L2: Operations | Topics: Secrets Management | Domain: Security

Secrets Management - Primer

Why This Matters

Secrets (API keys, database passwords, TLS certs, tokens) are the most dangerous thing in your infrastructure. A leaked secret can compromise an entire organization in minutes. Kubernetes Secrets are base64-encoded, not encrypted. Storing them in Git is a ticking time bomb. You need a real secrets management strategy.

The Problem with Kubernetes Secrets

# This is NOT encrypted - just base64
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
data:
  password: cGFzc3dvcmQxMjM=  # echo -n 'password123' | base64

Anyone with kubectl get secret access can decode this. And if you commit this YAML to Git, the secret is in your repo history forever.

Gotcha: Base64 is encoding, not encryption. echo cGFzc3dvcmQxMjM= | base64 -d instantly reveals the password. Many engineers mistakenly believe Kubernetes Secrets provide security — they do not. They are a convenience for storing non-public data separately from ConfigMaps, but without etcd encryption at rest, the Secret values sit in plaintext in etcd's data directory on the control plane node.

Solutions Landscape

Tool How it works Best for
Sealed Secrets Encrypt in Git, decrypt in cluster Teams already using GitOps
SOPS Encrypt files with KMS/PGP/age Existing CI/CD pipelines
External Secrets Operator Sync from external store to K8s Cloud-native, Vault users
HashiCorp Vault Central secret store + dynamic secrets Enterprise, multi-platform
Vault Agent Injector Sidecar injects secrets into pods Vault users on K8s

Sealed Secrets

Bitnami Sealed Secrets encrypts secrets with a cluster-specific public key. Only the controller in the cluster can decrypt them.

How It Works

[Developer] --kubeseal--> [SealedSecret YAML] --git push--> [Git]
                                                                |
[Sealed Secrets Controller] <--- watches --- [K8s API]
        |
        v
[Decrypted K8s Secret]

Usage

# Install controller
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets -n kube-system

# Create a regular secret, then seal it
kubectl create secret generic db-creds \
  --from-literal=password=supersecret \
  --dry-run=client -o yaml | kubeseal --format yaml > sealed-secret.yaml

# The sealed secret is safe to commit to Git
cat sealed-secret.yaml

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-creds
spec:
  encryptedData:
    password: AgB7...encrypted...data...

Scopes

Scope What it means
strict (default) Bound to name + namespace
namespace-wide Can be renamed within the namespace
cluster-wide Can be used anywhere in the cluster 
# Seal with namespace-wide scope
kubeseal --scope namespace-wide --format yaml < secret.yaml > sealed.yaml

SOPS (Secrets OPerationS)

Mozilla SOPS encrypts specific values in YAML/JSON files, leaving keys readable.

Name origin: SOPS stands for Secrets OPerationS. It was created by Julien Vehent at Mozilla in 2015 for managing secrets in the Firefox infrastructure. Its killer feature is partial encryption — keys remain readable so you can git diff and review changes without decrypting, while values stay encrypted. This makes code review of secret changes practical.

How It Works

# Configure SOPS to use age (simple key management)
age-keygen -o key.txt
export SOPS_AGE_KEY_FILE=key.txt

# Create .sops.yaml config
cat > .sops.yaml << 'SOPSEOF'
creation_rules:
  - path_regex: .*secrets.*\.yaml$
    age: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
SOPSEOF

# Encrypt a file
sops -e secrets.yaml > secrets.enc.yaml

# Decrypt (needs private key)
sops -d secrets.enc.yaml

With Cloud KMS

# AWS KMS
sops --encrypt --kms 'arn:aws:kms:us-east-1:123456:key/abc-123' secrets.yaml > secrets.enc.yaml

# GCP KMS
sops --encrypt --gcp-kms 'projects/myproject/locations/global/keyRings/myring/cryptoKeys/mykey' secrets.yaml

# Azure Key Vault
sops --encrypt --azure-kv 'https://myvault.vault.azure.net/keys/mykey/abc123' secrets.yaml

ArgoCD + SOPS

ArgoCD can decrypt SOPS-encrypted files using the ksops plugin or helm-secrets.

External Secrets Operator (ESO)

ESO syncs secrets from external stores (Vault, AWS Secrets Manager, GCP Secret Manager) into Kubernetes Secrets.

How It Works

[AWS Secrets Manager] <--- reads --- [ESO Controller] --creates--> [K8s Secret]
[Vault]                                                             (auto-refreshed)
[GCP Secret Manager]

Setup

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace

SecretStore + ExternalSecret

# Connect to AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets
  namespace: grokdevops
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
---
# Sync a specific secret
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: grokdevops
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
  target:
    name: db-credentials
  data:
    - secretKey: password
      remoteRef:
        key: prod/grokdevops/db
        property: password

ClusterSecretStore

Share a secret store across all namespaces:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault
spec:
  provider:
    vault:
      server: https://vault.example.com
      path: secret
      auth:
        kubernetes:
          mountPath: kubernetes
          role: external-secrets

HashiCorp Vault

The most feature-rich option. Provides static secrets, dynamic secrets (auto-rotating database credentials), encryption as a service, and PKI.

Interview tip: When asked about Vault's key advantage, focus on dynamic secrets. Unlike static secrets that exist until manually rotated, Vault can generate short-lived database credentials on demand (e.g., a PostgreSQL user with a 1-hour TTL). When the lease expires, Vault automatically revokes the credentials. This means a leaked credential is only useful for minutes, not months.

Key Concepts

Concept What it means
Secret engine Backend that stores/generates secrets (kv, database, pki, transit)
Auth method How clients authenticate (kubernetes, OIDC, AppRole, token)
Policy What secrets a client can access
Lease TTL on dynamic secrets (auto-revoked when expired)

Kubernetes Auth

# Enable Kubernetes auth in Vault
vault auth enable kubernetes

vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc"

# Create a policy
vault policy write grokdevops - <<EOF
path "secret/data/grokdevops/*" {
  capabilities = ["read"]
}
EOF

# Create a role bound to a K8s ServiceAccount
vault write auth/kubernetes/role/grokdevops \
  bound_service_account_names=grokdevops \
  bound_service_account_namespaces=grokdevops \
  policies=grokdevops \
  ttl=1h

Vault Agent Injector

Automatically injects secrets into pods via init/sidecar containers:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grokdevops
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "grokdevops"
        vault.hashicorp.com/agent-inject-secret-db-password: "secret/data/grokdevops/db"
        vault.hashicorp.com/agent-inject-template-db-password: |
          {{- with secret "secret/data/grokdevops/db" -}}
          {{ .Data.data.password }}
          {{- end -}}
    spec:
      serviceAccountName: grokdevops
      containers:
        - name: app
          # Secret available at /vault/secrets/db-password

Decision Guide

Small team, GitOps?          -> Sealed Secrets
CI/CD pipeline encryption?   -> SOPS + age/KMS
Cloud-native, multiple stores? -> External Secrets Operator
Enterprise, dynamic secrets?  -> Vault + ESO or Agent Injector
Multiple platforms (K8s + VMs)? -> Vault

Common Pitfalls

  1. Committing plaintext secrets to Git — Use git-secrets or pre-commit hooks to scan for secrets before commit
  2. Sealed Secrets key rotation — Back up the controller key. If you lose it, all SealedSecrets are unrecoverable
  3. Secret rotation — Changing the secret in Vault/AWS doesn't restart pods. Use ESO's refreshInterval + a restart mechanism
  4. RBAC on K8s Secrets — Even with ESO/Vault, the resulting K8s Secret is still base64. Lock down get secret RBAC
  5. etcd encryption — Enable encryption at rest for etcd to protect K8s Secrets on disk

Wiki Navigation

Prerequisites

Next Steps

Related Content