Skip to content

The Secrets in the Repo

Category: The Close Call Domains: security, git Read time: ~5 min

Setting the Scene

I'd just gotten a new MacBook Pro. Fresh setup, migration assistant skipped because I wanted a "clean start." I spent Monday morning installing Homebrew, Docker, VS Code, and cloning our repos. By noon I was back to writing code. What I didn't realize was that our pre-commit hooks were installed via a setup script I forgot to run. The .pre-commit-config.yaml was there, but pre-commit install had never executed on this machine.

Our team had a shared AWS credentials approach: each developer had IAM user keys for the dev account, stored in ~/.aws/credentials. But I was also working on a deployment automation project that needed production cross-account access, so I had a second set of keys — production IAM keys with AdministratorAccess — in an .env file in the project root.

What Happened

I was refactoring the deployment script and decided to add a configuration module. I created a new file, config.py, that loaded environment variables. To test it locally, I'd been sourcing the .env file. The .gitignore had .env listed, so I wasn't worried.

Except I'd also created a file called env.py that hardcoded fallback values for local development. Including the AWS access key and secret key. I told myself I'd replace them with placeholders before committing. I forgot.

At 2:23 PM, I ran git add . then git commit -m "add config module" then git push origin main. No pre-commit hook fired. No warning. The keys were in the commit, in a public GitHub repository, in plain text.

At 2:27 PM — four minutes later — I got an email from GitHub: "We found a secret in your repository." GitHub's secret scanning had detected the AWS key pattern (AKIA...) and flagged it. Simultaneously, our AWS organization had a CloudWatch Events rule that triggered on CreateAccessKey and GetCallerIdentity calls from unknown IPs. Within 8 minutes, our security team had an automated runbook that rotated the compromised key and attached a deny-all policy to the IAM user.

I checked CloudTrail. Between 2:23 PM and 2:31 PM, there were 14 unauthorized API calls from three different IP addresses in three countries using my key. They were trying DescribeInstances, ListBuckets, and CreateUser. All denied because the automated rotation beat them.

The Moment of Truth

GitHub's secret scanning caught the key in 4 minutes. Our automated rotation killed it in 8 minutes. The attackers got zero access. But if either of those systems hadn't existed — if we'd relied solely on the pre-commit hook that wasn't installed — those production keys would have been live on a public repo with no alerting. The median time for leaked AWS keys to be exploited is under 5 minutes. We were inside that window.

The Aftermath

We moved to AWS IAM Identity Center (SSO) with temporary credentials. No more long-lived IAM keys, period. We added detect-aws-credentials and detect-private-key to a server-side pre-receive hook on GitHub Enterprise, so it doesn't matter if the developer's local hooks aren't installed. We also added trufflehog to our CI pipeline as a backstop. And I ran pre-commit install on my new laptop.

The Lessons

  1. Defense in depth is mandatory: A single control will fail. The pre-commit hook failed. GitHub scanning caught it. Automated rotation contained it. Any one layer alone would not have been enough.
  2. Don't rely on a single control: .gitignore protects .env. Pre-commit hooks catch secrets. GitHub scanning detects patterns. Server-side hooks block pushes. You need all of them, because each one has a failure mode the others cover.
  3. Secret scanning is essential: GitHub's built-in secret scanning is free for public repos. Enable it. Also enable push protection, which blocks the push before it even hits the repo. Four minutes was fast enough this time. Next time it might not be.

What I'd Do Differently

I'd eliminate long-lived credentials entirely from day one. Use IAM Identity Center, OIDC federation for CI/CD, and aws sso login for developer access. If the credentials don't exist as strings, they can't be committed. The best secret management is having no secrets to manage.

The Quote

"git add . is the most dangerous command in computing. It means 'I trust every file in this directory with my career.'"

Cross-References