Skip to content

Secrets Management Cheat Sheet

Gotcha: Kubernetes Secrets are base64-encoded, NOT encrypted. Anyone with kubectl get secret access can decode them instantly: echo <value> | base64 -d. Base64 is an encoding (like URL encoding), not encryption. You need etcd encryption at rest AND RBAC restrictions on Secret access to actually protect secrets in a cluster.

The Problem

DON'T: Secret in Git → anyone with repo access sees it
DO:    Encrypted/referenced in Git → decrypted only at deploy time

Option Comparison

Tool Approach Encryption GitOps Friendly
Sealed Secrets Encrypt in Git, decrypt in cluster Asymmetric (RSA) Yes
SOPS Encrypt files in Git KMS / PGP / age Yes
External Secrets (ESO) Reference external store N/A (fetched at runtime) Yes
Vault + Agent Inject at runtime Transit / Shamir Partial

Sealed Secrets

# Install
helm install sealed-secrets sealed-secrets/sealed-secrets -n kube-system

# Encrypt a secret
kubectl create secret generic db-creds \
  --from-literal=password=hunter2 \
  --dry-run=client -o yaml | \
  kubeseal --format yaml > sealed-db-creds.yaml

# The SealedSecret is safe to commit to Git
git add sealed-db-creds.yaml && git commit -m "Add sealed DB creds"

# Fetch the public cert (for offline sealing)
kubeseal --fetch-cert > pub-cert.pem
kubeseal --cert pub-cert.pem < secret.yaml > sealed-secret.yaml

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-creds
spec:
  encryptedData:
    password: AgBy8h...  # Only decryptable by this cluster

Under the hood: Sealed Secrets uses asymmetric encryption (RSA). The controller generates a key pair — the public key encrypts (anyone can seal), the private key decrypts (only the controller in the cluster can unseal). If you lose the controller's private key (e.g., cluster rebuild without backup), all existing SealedSecrets become permanently undecryptable. Back up the controller's key: kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > master-key.yaml.

SOPS (Secrets OPerationS)

# Encrypt with AWS KMS
sops --encrypt --kms arn:aws:kms:us-east-1:123:key/abc secret.yaml > secret.enc.yaml

# Encrypt with age
sops --encrypt --age age1... secret.yaml > secret.enc.yaml

# Decrypt
sops --decrypt secret.enc.yaml

# Edit in place
sops secret.enc.yaml

# .sops.yaml — project-level config
creation_rules:
  - path_regex: .*\.secret\.yaml$
    kms: arn:aws:kms:us-east-1:123:key/abc

External Secrets Operator (ESO)

# SecretStore — connect to external provider
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: eso-sa
---
# ExternalSecret — what to fetch
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-creds
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
  target:
    name: db-creds           # K8s Secret created automatically
  data:
  - secretKey: password       # Key in the K8s Secret
    remoteRef:
      key: prod/db-password   # Key in AWS Secrets Manager

Vault Integration

# Enable KV secrets engine
vault secrets enable -path=secret kv-v2

# Write a secret
vault kv put secret/db-creds password=hunter2

# Read
vault kv get secret/db-creds

# Kubernetes auth
vault auth enable kubernetes
vault write auth/kubernetes/config \
  kubernetes_host=https://kubernetes.default.svc

# Vault Agent sidecar injection
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "my-app"
        vault.hashicorp.com/agent-inject-secret-db: "secret/data/db-creds"
        vault.hashicorp.com/agent-inject-template-db: |
          {{- with secret "secret/data/db-creds" -}}
          {{ .Data.data.password }}
          {{- end -}}

etcd Encryption at Rest

# /etc/kubernetes/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources: ["secrets"]
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: <base64-encoded-key>
  - identity: {}  # Fallback: read unencrypted

Name origin: SOPS = "Secrets OPerationS." Created by Mozilla in 2015 for managing secrets in Git. Unlike Sealed Secrets (cluster-specific), SOPS encrypts values but leaves keys in plaintext — so you can git diff encrypted files and see which fields changed without decrypting. It supports multiple key backends: AWS KMS, GCP KMS, Azure Key Vault, age, and PGP.

Secret Rotation Checklist

  1. Generate new credential
  2. Add new credential alongside old one
  3. Update application to use new credential
  4. Verify application works with new credential
  5. Remove old credential
  6. Verify no service uses old credential

Quick Decision Tree

Need secrets in Git?
├── Yes, encrypt at rest → Sealed Secrets or SOPS
└── No, reference only
    ├── Cloud-native secrets store → ESO
    └── Need advanced policies (rotation, dynamic) → Vault