Skip to content

Portal | Level: L2: Operations | Topics: Secrets Management, Git | Domain: Security

Scenario: Secret Leaked to Git

The Prompt

"A developer just pinged you - they accidentally committed a database password to a public GitHub repository. The commit was pushed 2 hours ago. What's your response?"

Initial Report

Slack message: "I pushed a commit with the DB password in a YAML file. It's been on GitHub for 2 hours. What do I do?"

Constraints

  • Time pressure: Credential may already be harvested by bots scanning GitHub.
  • Blast radius: The password is for the production database.
  • Coordination: Need to rotate without causing downtime.

Expected Response Path

# IMMEDIATE (first 5 minutes):
1. Rotate the database password NOW
   - Don't investigate first. Assume compromised.
   - Generate new password
   - Update in Vault/AWS Secrets Manager/wherever it's stored
   - Update K8s Secret
   - Rolling restart all pods using the credential

2. Revoke old credential
   - Change the DB password so old one no longer works
   - Verify app is working with new password

# WITHIN 1 HOUR:
3. Clean Git history
   - git filter-repo or BFG Repo Cleaner to remove from all commits
   - Force push cleaned history
   - All contributors must re-clone

4. Audit
   - Check database access logs for unauthorized access in the 2-hour window
   - Check if the password was used by any unexpected IP
   - Check if any data was accessed or exfiltrated

# WITHIN 1 DAY:
5. Prevent recurrence
   - Install pre-commit hooks (git-secrets, detect-secrets)
   - Add CI pipeline scanning for secrets
   - Review: why was the secret in a file at all? Use ESO/Vault instead.
   - Team training on secrets hygiene

What a Strong Answer Includes

  • Rotate first, investigate second — This is the key insight. Don't waste time on Git cleanup while the credential may be actively exploited.
  • Understanding that deleting the commit is NOT enough (forks, cached copies, bot harvesters)
  • Specific tools for Git history cleaning (git filter-repo, BFG)
  • Long-term prevention (pre-commit hooks, secrets management tooling)
  • Audit trail checking
  • Blameless approach: "How do we prevent this?" not "Who did this?"

Wiki Navigation

Related Content