Skip to content

Portal | Level: L2: Operations | Topics: Secrets Management | Domain: Security

Scenario: Vault Tokens Expired Across All Services

The Prompt

"All our microservices suddenly started failing with 'permission denied' errors when trying to read secrets from Vault. The Vault server itself is healthy. What happened?"

Initial Report

Alert: "5 services in production returning 500 errors. All log: 'vault: permission denied'. Vault UI is accessible and admin login works."

Constraints

  • Time pressure: 5 services are down simultaneously.
  • Vault is a shared service: Changes affect all teams.
  • Don't rotate all secrets unless necessary — that's a bigger operation.

Observable Evidence

  • Application logs: vault: permission denied
  • Vault audit logs show: auth/kubernetes/login returning 403
  • Vault is healthy: vault status shows unsealed, HA active
  • Kubernetes auth is enabled in Vault
  • All pods have the correct ServiceAccount annotations

Expected Investigation Path

# 1. Check Vault health
vault status

# 2. Test Kubernetes auth manually
kubectl exec -it test-pod -- sh
# Inside pod:
JWT=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl -s --request POST \
  --data "{\"jwt\":\"$JWT\",\"role\":\"my-app\"}" \
  http://vault:8200/v1/auth/kubernetes/login

# 3. Check the Kubernetes auth config in Vault
vault read auth/kubernetes/config
# Look at: kubernetes_host, token_reviewer_jwt

# 4. THE PROBLEM: token_reviewer_jwt has expired
# This is the JWT Vault uses to validate K8s ServiceAccount tokens

# 5. Check K8s API server connectivity from Vault
vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc:443"
# (re-configures with current service account token)

# 6. Verify fix
vault write auth/kubernetes/login role=my-app jwt=$JWT

# 7. Restart affected pods to get fresh Vault tokens
kubectl rollout restart deployment -n production

Root Cause

Vault's Kubernetes auth method stores a JWT for talking to the Kubernetes API (to validate pod tokens). This JWT expired (ServiceAccount tokens have a TTL). When Vault can't validate incoming tokens against the K8s API, all auth attempts fail.

What a Strong Answer Includes

  • Quick triage: Vault is healthy, the issue is authentication not authorization
  • Understanding the Kubernetes auth flow (pod JWT → Vault → K8s API verification)
  • Identifying that the reviewer token can expire
  • Fix: reconfigure the Kubernetes auth with a fresh token
  • Prevention: use a long-lived or auto-rotating reviewer token, or use Vault's built-in K8s service account
  • Not panicking: existing cached/in-memory secrets may still work, only new token requests fail

Wiki Navigation

Related Content