The Git Disaster Recovery Guide

lesson
git-internals
reflog
reset
rebase-recovery
force-push
bisect
fsck ---# The Git Disaster Recovery Guide

Topics: Git internals, reflog, reset, rebase recovery, force push, bisect, fsck Level: L1–L2 (Foundations → Operations) Time: 60–75 minutes Prerequisites: Basic Git usage (commit, push, pull, branch)

The Mission¶

It's happened. You ran git reset --hard on the wrong branch. Or you force-pushed and overwrote your colleague's work. Or you're in a detached HEAD with commits that belong to no branch. Or git log shows a history that makes no sense.

Git disasters feel catastrophic because losing code feels permanent. But Git is designed so that almost nothing is truly lost — the data is there, you just need to find it. This lesson teaches you to recover from every common Git disaster using the tools Git provides.

Name Origin: Linus Torvalds created Git in approximately two weeks in April 2005 after BitKeeper revoked its free license for the Linux kernel project. He named it "git" — British slang for an unpleasant person. Linus said: "I'm an egotistical bastard, and I name all my projects after myself. First Linux, now git." The first commit was April 7; the Linux kernel was self-hosting on Git by April 18.

How Git Stores Everything¶

Before recovering data, you need to understand where it lives.

Git is a content-addressable filesystem with four object types:

Object	What it stores	Identified by
blob	File contents (no filename!)	SHA-1 hash of content
tree	Directory listing (filename → blob/tree mappings)	SHA-1 hash
commit	Tree pointer + parent(s) + author + message	SHA-1 hash
tag	Annotated pointer to a commit	SHA-1 hash

A branch is not an object — it's a 41-byte text file in .git/refs/heads/ containing one commit SHA. Moving a branch is just writing a different SHA to that file.

# See what's inside a commit
git cat-file -p HEAD
# → tree 4a2e8c...
# → parent 9f3d1a...
# → author Alice <alice@example.com> 1711108800 +0000
# → committer Alice <alice@example.com> 1711108800 +0000
# →
# → Fix authentication bug

# See what's inside the tree
git cat-file -p HEAD^{tree}
# → 100644 blob abc123... README.md
# → 040000 tree def456... src

Mental Model: Git's object store is like an immutable append-only log. Commits are never modified — git commit --amend creates a NEW commit with a new SHA. The old commit still exists in the object store until garbage collection removes it (which takes at least 2 weeks by default). This is why almost everything is recoverable.

The Reflog — Your Safety Net¶

The reflog records every time HEAD moves — every commit, checkout, reset, rebase, merge, pull. It's your 90-day time machine.

# See reflog
git reflog
# → abc123 HEAD@{0}: reset: moving to HEAD~3      ← 2 minutes ago
# → def456 HEAD@{1}: commit: Add authentication   ← 5 minutes ago
# → 789abc HEAD@{2}: commit: Fix login bug         ← 10 minutes ago
# → fedcba HEAD@{3}: commit: Update tests          ← 15 minutes ago

If you see the commit you lost in the reflog, you can recover it.

Gotcha: The reflog is local. It's not pushed, pulled, or cloned. If you delete the repository directory, the reflog is gone. It also expires — entries older than 90 days for reachable commits and 30 days for unreachable commits are pruned by git gc.

Disaster 1: `git reset --hard` — Lost Commits¶

You ran git reset --hard HEAD~3 and lost three commits. The branch pointer moved backward, but the commits still exist:

# Find the lost commits
git reflog
# → abc123 HEAD@{0}: reset: moving to HEAD~3
# → def456 HEAD@{1}: commit: The commit you want back

# Recover by moving the branch forward
git reset --hard def456
# or create a new branch pointing to the lost commit
git branch recovery def456

Disaster 2: Force Push — Overwrote Remote History¶

You force-pushed to main and overwrote your colleague's commits:

# On your machine, the reflog still has the old state
git reflog show origin/main
# → abc123 origin/main@{0}: update by push    ← your force push
# → def456 origin/main@{1}: update by push    ← the state before

# Restore the old state
git push --force origin def456:main

If you don't have the reflog (different machine), your colleague's local repo still has the old commits:

# On your colleague's machine
git fetch
git log origin/main  # Their local copy still has the old history
# Share the commit SHA with you

Gotcha: --force-with-lease is the safer version of --force. It only succeeds if the remote branch is where you think it is — if someone pushed in between, it fails instead of overwriting. Always use --force-with-lease instead of --force.

Disaster 3: Detached HEAD — Commits on No Branch¶

You checked out a tag or a specific commit, made changes, committed, then switched branches. Your commits belong to no branch:

# You were here
git checkout v1.0.0          # Detached HEAD
# Made changes and committed
git commit -am "Important fix"
# Then switched away
git checkout main
# → Warning: you are leaving 1 commit behind

# The commit exists but is "dangling" — no branch points to it
# Find it in the reflog
git reflog | grep "Important fix"
# → abc123 HEAD@{3}: commit: Important fix

# Save it to a branch
git branch hotfix abc123
git checkout hotfix

Disaster 4: Bad Rebase — Mangled History¶

A rebase went wrong — conflicts resolved incorrectly, commits in the wrong order, or the result doesn't compile. The rebase created new commits but the old ones are still in the reflog:

# Find the state before the rebase
git reflog
# → abc123 HEAD@{0}: rebase (finish): ...
# → def456 HEAD@{1}: rebase (continue): ...
# → 789abc HEAD@{5}: rebase (start): checkout main
# → BEFORE HEAD@{6}: commit: My good commit    ← this is what you want

# Abort if rebase is still in progress
git rebase --abort

# If already finished, reset to pre-rebase state
git reset --hard HEAD@{6}

Disaster 5: Committed Secrets¶

You committed an .env file with production credentials. Even if you delete it in the next commit, the credentials exist in Git history forever:

# Remove the file from ALL history
# Option 1: BFG Repo-Cleaner (fastest)
java -jar bfg.jar --delete-files .env

# Option 2: git filter-repo (recommended by Git project)
git filter-repo --invert-paths --path .env

# Both rewrite history — you must force push
git push --force
# All collaborators must re-clone or reset their branches

War Story: In 2016, Uber's AWS credentials were found in a private GitHub repository by attackers. The credentials gave access to an S3 bucket containing 57 million user records (names, emails, phone numbers, driver's license numbers). Uber paid the attackers $100,000 to delete the data and was later fined $148 million for failing to disclose the breach. The credentials should never have been in the repo — and once there, they survived across all clones and forks forever.

Prevention:

# Pre-commit hook to catch secrets
# .git/hooks/pre-commit
if git diff --cached --name-only | grep -qE '\.env$|\.pem$|\.key$|credentials'; then
    echo "ERROR: Potential secrets staged for commit"
    echo "Staged files matching secret patterns:"
    git diff --cached --name-only | grep -E '\.env$|\.pem$|\.key$|credentials'
    exit 1
fi

Disaster 6: Finding Which Commit Broke Things — `git bisect`¶

Something broke between v1.0 and HEAD (200 commits). Instead of checking each one:

git bisect start
git bisect bad HEAD          # Current version is broken
git bisect good v1.0         # This version was working

# Git checks out the midpoint
# Test if it's broken
make test

# Tell git the result
git bisect good              # This one's fine, bug is later
# or
git bisect bad               # This one's broken, bug is earlier

# Git narrows the range (binary search)
# After ~7 steps (log2(200)), you've found the exact commit
git bisect reset             # Return to where you started

Automated bisect (if you have a test script):

git bisect start HEAD v1.0
git bisect run make test
# Git runs the test at each bisect step automatically
# Returns the first commit where the test fails

Under the Hood: git bisect uses binary search — it halves the search space on each step. 200 commits takes at most 8 steps. 1,000 commits takes at most 10. This is logarithmic efficiency — arguably Git's most underused feature.

Disaster 7: Corrupted Repository¶

git status
# → error: object file .git/objects/ab/cd1234... is empty
# → fatal: loose object abcd1234... is corrupt

# Check integrity
git fsck --full
# → broken link from commit abc123 to tree def456
# → dangling commit 789abc

# If objects are damaged, try recovering from remote
git fetch origin

# Last resort: re-clone
# (but save your local branches first)
git branch -a > /tmp/my-branches.txt
git stash list > /tmp/my-stashes.txt
cd ..
git clone origin-url repo-recovered
cd repo-recovered
# Manually recreate local branches

The Recovery Cheat Sheet¶

Disaster	Recovery
`reset --hard` lost commits	`git reflog` → `git reset --hard SHA`
Force push overwrote remote	`git reflog show origin/branch` → force push old SHA
Detached HEAD commits	`git reflog` → `git branch recovery SHA`
Bad rebase	`git reflog` → `git reset --hard` pre-rebase SHA
Committed secrets	`git filter-repo` or BFG → force push → everyone re-clones
Find breaking commit	`git bisect start` → binary search
Corrupted objects	`git fsck` → `git fetch` → worst case re-clone
Deleted branch	`git reflog show branch-name` → `git branch name SHA`
Bad merge	`git reflog` → `git reset --hard` pre-merge SHA
Lost stash	`git fsck --unreachable \\| grep commit` → check each

Flashcard Check¶

Q1: Git reflog — what does it record and how long does it last?

Every HEAD movement (commit, checkout, reset, rebase, merge). 90 days for reachable commits, 30 days for unreachable. Local only — not pushed or cloned.

Q2: git reset --hard HEAD~3 — are the commits gone?

No. The branch pointer moved backward but the commits still exist in the object store. Find them with git reflog and git reset --hard to the desired SHA.

Q3: What's the difference between --force and --force-with-lease?

--force overwrites regardless. --force-with-lease fails if someone pushed since your last fetch — prevents accidentally overwriting others' work.

Q4: How does git bisect work?

Binary search through commit history. You mark a good and bad commit; Git checks out the midpoint; you test and report; Git narrows the range. O(log n) steps.

Q5: You committed credentials. Can you delete them from history?

Not with a normal commit. Use git filter-repo or BFG Repo-Cleaner to rewrite history. Then force push and have everyone re-clone. The credentials are in every clone/fork.

Exercises¶

Exercise 1: Practice reflog recovery (hands-on)¶

# Create a test repo
mkdir /tmp/git-disaster && cd /tmp/git-disaster && git init
echo "v1" > file.txt && git add . && git commit -m "Version 1"
echo "v2" > file.txt && git commit -am "Version 2"
echo "v3" > file.txt && git commit -am "Version 3"

# Disaster: reset back to v1
git reset --hard HEAD~2

# Recover v3
git reflog
# Find the SHA for "Version 3" and reset to it

# Clean up
rm -rf /tmp/git-disaster

Exercise 2: git bisect (hands-on)¶

mkdir /tmp/git-bisect && cd /tmp/git-bisect && git init
for i in $(seq 1 20); do echo "commit $i" > file.txt; git add . ; git commit -m "Commit $i"; done

# "Break" it at commit 13
git log --oneline | head -8  # find commit 13's SHA
# Imagine commit 13 introduced a bug

# Bisect to find it
git bisect start
git bisect bad HEAD
git bisect good HEAD~19
# Test at each step: grep "commit 13" file.txt && git bisect bad || git bisect good

rm -rf /tmp/git-bisect

Takeaways¶

Almost nothing in Git is truly lost. The reflog keeps 90 days of history. git fsck finds dangling objects. The object store is append-only.
git reflog is your time machine. Learn to read it. Every recovery starts there.
Never use --force when --force-with-lease exists. It prevents accidental history overwrites on shared branches.
Secrets in Git history are permanent. Even deletion doesn't remove them from clones and forks. Use pre-commit hooks to prevent, not post-commit surgery to fix.
git bisect is logarithmic. 1,000 commits in 10 steps. Automate it with git bisect run and a test script.

Why YAML Keeps Breaking Your Deploys — when Git merge conflicts in YAML go wrong
The Hanging Deploy — when deploy scripts using Git go wrong