Skip to content

Runbook: Credential Rotation (Exposed Secret)

Field Value
Domain Security
Alert Secret detected in git/logs/public location, or rotation triggered by policy
Severity P1 (if exposed), P2 (routine rotation)
Est. Resolution Time 30-60 minutes
Escalation Timeout 20 minutes — page if not resolved (if actively exploited)
Last Tested 2026-03-19
Prerequisites Access to credential vault/system, ability to update Kubernetes secrets, application restart permissions, IAM/cloud console access

Quick Assessment (30 seconds)

# Run this first — it tells you the scope of the problem
# Identify which credential was exposed and where it was found
git log --all --oneline -20
# Then check if the secret is still visible in recent commits:
git log -p --all -- <FILE_WHERE_SECRET_WAS_FOUND> | head -50
If output shows: the secret in a recent commit that was never pushed → You may be able to rewrite history before it goes public; escalate immediately to Step 1 anyway If output shows: the secret in a pushed commit or a public location → Assume it is compromised — proceed to Step 1 immediately, do not investigate scope first

Step 1: REVOKE IMMEDIATELY — Before Everything Else

Why: Every second the credential is active is a second an attacker can use it. Revoke first; investigate scope after. This is the single most important rule in credential exposure response. 

# AWS — disable or delete the access key:
aws iam update-access-key --access-key-id <ACCESS_KEY_ID> --status Inactive
# Or delete it entirely:
aws iam delete-access-key --access-key-id <ACCESS_KEY_ID>

# GCP — disable a service account key:
gcloud iam service-accounts keys disable <KEY_ID> \
  --iam-account=<SERVICE_ACCOUNT_EMAIL>

# GitHub — revoke a personal access token or OAuth token:
# Go to GitHub → Settings → Developer settings → Personal access tokens → Delete

# Database password — reset it immediately:
# MySQL/Postgres: ALTER USER '<USERNAME>'@'<HOST>' IDENTIFIED BY '<NEW_PASSWORD>';

# API key (generic) — use the issuing service's admin UI or API to revoke/rotate
echo "REVOKE THE CREDENTIAL NOW — use the issuing system's console"
Expected output: 
AWS: no output on success (or confirmation message).
GCP: "Updated key [<KEY_ID>]."
After revocation, any system using the old credential should start failing with auth errors —
this is expected and confirms the revocation worked.
If this fails: If you do not have permission to revoke the credential yourself, call the security on-call immediately while you escalate — do not wait.

Step 2: Check Audit Logs for Unauthorized Use

Why: After revoking, determine if the credential was already used by an attacker. Scope defines the blast radius — you need this to assess impact and decide whether a broader incident response is needed. 

# AWS CloudTrail — look for unexpected API calls using this credential:
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=Username,AttributeValue=<IAM_USERNAME_OR_ROLE> \
  --start-time $(date -d '48 hours ago' --iso-8601=seconds) \
  --output json | jq '.Events[] | {Username, EventName, EventTime, SourceIPAddress}'

# GCP audit logs:
gcloud logging read \
  'resource.type="service_account" AND protoPayload.authenticationInfo.principalEmail="<SERVICE_ACCOUNT_EMAIL>"' \
  --limit=50 \
  --format="json" | jq '.[] | {time: .timestamp, method: .protoPayload.methodName, ip: .httpRequest.remoteIp}'

# Kubernetes audit logs (if the secret was a kube credential):
# Check with your cluster provider — EKS: CloudTrail; GKE: Cloud Audit Logs; self-managed: /var/log/kube-audit.log

# GitHub — check audit log for token usage:
# Go to Organization → Settings → Audit log → filter by actor or token
Expected output: 
If no unauthorized use: all events will show expected IPs, usernames, and API calls.
If unauthorized use detected: you will see unexpected source IPs, unusual API calls
  (ListBuckets, DescribeInstances, CreateUser, etc.), or actions at unusual times.
If this fails: If audit logs are not accessible, escalate to the security team — they have broader log access and this data is critical for breach scope assessment.

Step 3: Generate a New Credential

Why: Applications that depended on the old credential need a replacement. Create this before updating application secrets to minimize downtime. 

# AWS — create new access key:
aws iam create-access-key --user-name <IAM_USERNAME>
# Note the AccessKeyId and SecretAccessKey — you cannot retrieve the secret again later

# GCP — create new service account key:
gcloud iam service-accounts keys create <OUTPUT_KEY_FILE>.json \
  --iam-account=<SERVICE_ACCOUNT_EMAIL>

# Database password — generate a strong random password:
openssl rand -base64 32

# API key — use the issuing service's admin UI to generate a new key
echo "Save the new credential securely — you will need it in the next step"
Expected output: 
AWS: JSON with "AccessKeyId" and "SecretAccessKey" fields.
GCP: key file written to <OUTPUT_KEY_FILE>.json.
openssl: a 32-character base64-encoded random string.
If this fails: If you cannot create a new credential (permission denied), escalate to the security team or cloud admin. Do not use a shared credential as a temporary workaround.

Step 4: Update the Kubernetes Secret

Why: Applications read credentials from Kubernetes secrets at startup or periodically. The secret must be updated so the application can authenticate after you restart it. 

# Update an existing Kubernetes generic secret:
kubectl create secret generic <SECRET_NAME> \
  --from-literal=<KEY_NAME>=<NEW_CREDENTIAL_VALUE> \
  -n <NAMESPACE> \
  --dry-run=client -o yaml | kubectl apply -f -

# If the secret is a Docker pull secret:
kubectl create secret docker-registry <SECRET_NAME> \
  --docker-server=<REGISTRY_HOST> \
  --docker-username=<USERNAME> \
  --docker-password=<NEW_PASSWORD> \
  -n <NAMESPACE> \
  --dry-run=client -o yaml | kubectl apply -f -

# Verify the secret was updated:
kubectl get secret <SECRET_NAME> -n <NAMESPACE> -o jsonpath='{.metadata.resourceVersion}'
# The resourceVersion should have changed from before the update
Expected output: 
"secret/<SECRET_NAME> configured"  — if updated
"secret/<SECRET_NAME> created"     — if newly created
If this fails: If you get a "forbidden" error, you need secrets write permission in that namespace. Ask the platform team to apply the change or grant you temporary elevated permissions.

Step 5: Restart Affected Deployments

Why: Most applications only read secrets at startup — updating the Kubernetes secret does not cause running pods to pick up the new value. You must restart them. 

# Rolling restart (no downtime if replicas > 1):
kubectl rollout restart deployment/<DEPLOY_NAME> -n <NAMESPACE>

# Monitor the restart:
kubectl rollout status deployment/<DEPLOY_NAME> -n <NAMESPACE>

# Check logs to confirm the new credential is being used successfully:
kubectl logs deployment/<DEPLOY_NAME> -n <NAMESPACE> --since=5m | grep -i "auth\|connect\|login\|error"
Expected output: 
"deployment.apps/<DEPLOY_NAME> restarted"
"deployment "<DEPLOY_NAME>" successfully rolled out"
Logs should show successful authentication with no credential errors.
If this fails: If pods crash after restart, the new credential may be wrong (typo, wrong format). Check the credential value in the secret: kubectl get secret <SECRET_NAME> -n <NAMESPACE> -o jsonpath='{.data.<KEY_NAME>}' | base64 -d

Step 6: Remove the Old Credential from All Cached Locations

Why: Revoking the credential and updating Kubernetes is not enough if the credential is still cached in CI/CD variables, .env files, or git history — it must be purged from all locations. 

# Check CI/CD system environment variables (update manually in the UI):
# GitHub: Settings → Secrets → update or rotate the relevant secret
# GitLab: Settings → CI/CD → Variables → update
# Jenkins: Manage Jenkins → Credentials → rotate

# Check for .env files with the old credential (local only — never commit these):
find . -name ".env*" -not -path "*/.git/*" | xargs grep -l "<PARTIAL_SECRET_VALUE>" 2>/dev/null

# Scan git history for the exposed value (use a partial, non-sensitive fragment):
git log -p --all | grep -i "<PARTIAL_SECRET_VALUE_SAFE_TO_PRINT>"
Expected output: 
CI/CD variables updated — confirm by triggering a test pipeline run.
find output: any .env files containing the old credential (update manually, never commit).
git log: if the secret appears in history, you need to rewrite history (see below — requires coordination).
If this fails: If the secret is in git history and the repo is public or the commit has been pushed, treat the credential as permanently compromised regardless of revocation status — escalate to security team for full incident response.

Step 7: Verify the New Credential Works End-to-End

Why: All the previous steps are worthless if the new credential doesn't actually work — verify before closing the incident. 

# Test the new credential directly (example: AWS):
AWS_ACCESS_KEY_ID=<NEW_ACCESS_KEY_ID> \
AWS_SECRET_ACCESS_KEY=<NEW_SECRET_ACCESS_KEY> \
aws sts get-caller-identity

# Check application health after restart:
kubectl get pods -n <NAMESPACE> -l app=<APP_LABEL>
# All pods should be Running with READY containers

# Check application logs for authentication success:
kubectl logs deployment/<DEPLOY_NAME> -n <NAMESPACE> --since=10m | grep -i "error\|auth\|connect" | head -20
Expected output: 
aws sts get-caller-identity: JSON with "Account", "UserId", and "Arn" fields — confirms new key works.
All pods: Running with READY status.
Logs: no authentication errors.
If this fails: If the new credential fails authentication, the key creation may have had an error. Return to Step 3 and regenerate. Do not close the incident until authentication is confirmed working.

Verification

# Confirm the issue is resolved
kubectl get pods -n <NAMESPACE> -l app=<APP_LABEL>
aws sts get-caller-identity  # or equivalent for your credential type
Success looks like: All pods running, no auth errors in logs, new credential authenticates successfully, old credential is revoked (verify: aws iam get-access-key-last-used --access-key-id <OLD_KEY_ID> should show no recent use after revocation). If still broken: Escalate — see below.

Escalation

Condition Who to Page What to Say
Not resolved in 20 min (active exploit) Security on-call "P1 Security incident: credential exposed and actively exploited — need immediate incident response"
Evidence of unauthorized access Security on-call "Security incident: audit logs show unauthorized use of exposed credential — potential breach"
Cannot revoke credential Security on-call + Cloud Admin "Unable to revoke exposed credential for — need emergency access"
Scope expanding Security on-call "Credential exposure may have led to lateral movement — scope expanding beyond original service"

Post-Incident

  • Update monitoring if alert was noisy or missing
  • File postmortem if P1/P2
  • Update this runbook if steps were wrong or incomplete
  • Rewrite git history if credential is in commit history (coordinate with team first — force push required)
  • Review how the credential was exposed and add pre-commit hooks or scanning to prevent recurrence
  • Add secret scanning to CI pipeline (GitHub: Secret Scanning; GitLab: Secret Detection; or tools like gitleaks, trufflehog)
  • Audit all other credentials managed the same way — if one leaked, others may be at risk

Common Mistakes

  1. Investigating before revoking: Every second you spend investigating before revoking is a second the attacker can act. Revoke immediately — you can investigate with the logs after.
  2. Not checking all places the old secret was used: Applications often cache credentials in multiple places — Kubernetes secrets, CI/CD variables, .env files, Helm values, and running pods. Missing one location means the old credential is still in play.
  3. Forgetting CI/CD environment variables: Kubernetes secrets and application restarts are not enough if the CI/CD system still has the old credential as an environment variable for future builds.
  4. Not scanning git history: If the secret was committed even once, it's in git history forever unless you rewrite it. git rm does not remove it from history.
  5. Treating routine rotation the same as exposure: For P1 exposures, follow all steps including audit log review. For P2 routine rotation, Steps 2 is optional but Steps 1, 3-7 are still required.

Cross-References

  • Topic Pack: training/library/topic-packs/security-fundamentals/ (deep background on secrets management)
  • Related Runbook: unauthorized-access.md — if audit logs show the credential was used by an attacker
  • Related Runbook: cve-response.md — if the exposure was via a vulnerable dependency
  • Related Runbook: ../kubernetes/secret_rotation.md — Kubernetes-specific secret rotation steps

Wiki Navigation

Related Content