Thinking Out Loud: Git

A senior SRE's internal monologue while working through a real Git crisis. This isn't a tutorial — it's a window into how experienced engineers actually think.

The Situation

A developer force-pushed to main and overwrote 3 days of commits from the team. Six developers' work is gone from the remote. I need to recover the commits and restore the branch. The developer who force-pushed is panicking.

The Monologue

Force push to main. Deep breath. Git almost never truly loses data — it just makes it hard to find. The commits exist somewhere. Let me figure out the recovery path.

First, let me check the damage. What does main look like now?

git log --oneline -10 origin/main

The last commit on main is from Friday. Today is Monday. Three days of work are missing from the remote. Let me check my local main — I pulled yesterday, so I might have the commits.

git log --oneline -10 main

My local main has commits from Saturday and Sunday. Good — I have some of the missing work. But other developers' commits from today (Monday) might not be on my machine.

Let me check the reflog on the remote. If it's a platform like GitHub, they keep the reflog for a while.

git reflog show origin/main | head -20

Hmm, the local reflog for origin/main shows my last fetch. That might have the old HEAD. Let me check.

Mental Model: The Reflog Is Your Safety Net

Git's reflog records every position that HEAD (or a branch tip) has been at. Even after a force push, the old commits still exist in the repo until garbage collection runs (usually 30-90 days). The reflog is the map to find them. On GitHub, you can also use the Events API to find the old branch tip SHA.

Let me find the commit SHA that main SHOULD be pointing to. I need the SHA from before the force push.

# Check if other developers have the correct main
# Ask the team or check GitHub's audit log
gh api repos/our-org/our-repo/events --paginate | jq '.[] | select(.type=="PushEvent") | {actor: .actor.login, before: .payload.before, head: .payload.head, ref: .payload.ref, created: .created_at}' | head -20

Found it. The PushEvent from the force push shows before: abc123def456. That was the correct HEAD of main before the force push. Let me verify this SHA has the expected commits.

git log --oneline abc123def456 | head -10

Yes — there are the Monday commits I was missing. This SHA is the correct state of main. Now let me restore it.

Before I push, let me make sure I'm not going to overwrite the force-pusher's work. They might have had a commit they needed.

git log --oneline $(git merge-base abc123def456 origin/main)..origin/main

The force-pusher's branch has commits that are from an old feature branch. They merged their feature branch to main locally (wrong — it was stale) and force-pushed. Their feature commits are already in the correct history at abc123def456 (they were merged via PR last week). So restoring to abc123def456 doesn't lose anything.

git push origin abc123def456:main --force-with-lease

Wait — --force-with-lease might fail because my local tracking ref doesn't match the current remote. Let me update first.

git fetch origin
git push origin abc123def456:refs/heads/main --force

Mental Model: Force Push Recovery Order

When recovering from a bad force push: (1) Find the correct SHA (reflog, GitHub events API, team's local repos). (2) Verify the SHA has all the expected commits. (3) Verify you're not losing anything from the bad push. (4) Force push the correct SHA back. (5) Notify the team to reset their local branches. Speed matters because every developer who pulls the bad main will have merge conflicts.

Main is restored. Let me verify.

git log --oneline origin/main | head -10

All commits are back. Now I need to alert the team.

# Check if anyone has already pulled the bad main
git log --oneline origin/main | head -1

Let me notify the team to run:

# For developers who pulled the bad main:
git fetch origin
git reset --hard origin/main

And for developers who had local commits that weren't pushed yet, they should:

# Check if their commits are in the restored main
git log origin/main --oneline | grep "their commit message"

# If not, cherry-pick their local commits onto the restored main
git rebase origin/main

Now, the important part — preventing this from happening again.

# Set up branch protection on GitHub
gh api repos/our-org/our-repo/branches/main/protection -X PUT \
  -f required_pull_request_reviews='{"required_approving_review_count":1}' \
  -f enforce_admins=true \
  -F allow_force_pushes=false \
  -F allow_deletions=false

Actually, I should check with the team lead before modifying branch protection. But at minimum, disabling force pushes on main is non-controversial.

Let me also add a pre-push hook that warns before force pushing to main.

cat <<'EOF' > .git/hooks/pre-push
#!/bin/bash
protected_branch='main'
current_branch=$(git symbolic-ref HEAD | sed 's|refs/heads/||')
push_command=$(ps -ocommand= -p $PPID)

if [[ "$current_branch" == "$protected_branch" ]] && [[ "$push_command" =~ "force" ]]; then
    echo "WARNING: You are about to force push to $protected_branch!"
    echo "This will overwrite the remote branch."
    read -p "Are you ABSOLUTELY sure? (type 'yes' to confirm): " confirm
    if [[ "$confirm" != "yes" ]]; then
        echo "Push aborted."
        exit 1
    fi
fi
EOF
chmod +x .git/hooks/pre-push

One more thing — let me check if the developer who force-pushed needs any help cleaning up their local state.

# The force-pusher should:
git fetch origin
git checkout main
git reset --hard origin/main

Crisis resolved. Main is restored, team is notified, and I'll propose branch protection rules in the next team meeting.

What Made This Senior-Level

Junior Would...	Senior Does...	Why
Panic and think the commits are permanently lost	Know that git reflog and GitHub Events API can recover the old branch tip	Git almost never truly deletes data — it just moves the pointer
Force push a fix immediately without checking	Verify the recovery SHA has all expected commits AND that the bad push didn't contain unique commits	You don't want to lose the force-pusher's work while recovering everyone else's
Restore the branch and move on	Also set up branch protection and pre-push hooks to prevent recurrence	The same mistake will happen again without guardrails
Not notify the team about the restoration	Alert everyone to reset their local branches to avoid merge conflicts	Developers who pulled the bad main need to know the remote was restored

Key Heuristics Used

1. Reflog Is the Safety Net: Git's reflog keeps every branch position for 30-90 days. Use it (or GitHub's Events API) to find the pre-force-push SHA.
2. Verify Before Force Pushing the Fix: A recovery force push that loses the original force-pusher's unique commits is a second disaster. Always verify.
3. Guard the Recovery: After restoring, add branch protection to prevent recurrence. Force push to protected branches should be blocked at the platform level.

Cross-References

• Primer — Git object model, refs, and how force push changes branch pointers
• Street Ops — Reflog commands, cherry-pick recovery, and branch protection setup
• Footguns — Force pushing to shared branches, not using --force-with-lease, and missing branch protection

Pages that link here

• Git Footguns
• Git for DevOps Engineers - Street Ops