Git for DevOps Engineers - Street Ops

What experienced engineers know about Git that you learn the hard way.

Recovery Playbooks

Recovering from a Bad Merge

Situation: You merged a branch into main and it broke everything.

Option 1: Revert the merge commit (safe for shared branches)
   git log --oneline -5           # Find the merge commit hash
   git revert -m 1 <merge-commit> # -m 1 keeps the main branch side
   git push origin main

   Note: If you later want to re-merge that branch, you need to
   revert the revert first, or the changes won't appear in the diff.

Option 2: Reset (ONLY if no one else has pulled)
   git reset --hard <commit-before-merge>
   git push --force-with-lease origin main

   --force-with-lease is safer than --force: it fails if someone
   else pushed in the meantime.

Reflog Rescue

Situation: You ran git reset --hard and lost commits.

1. Find the lost commit:
   git reflog
   # Shows every HEAD movement. Find the commit before the reset.
   # Output looks like:
   # abc1234 HEAD@{0}: reset: moving to HEAD~3
   # def5678 HEAD@{1}: commit: Add authentication module
   # ghi9012 HEAD@{2}: commit: Update config parser

2. Recover:
   git checkout def5678                    # Detached HEAD at lost commit
   git checkout -b recovered-branch        # Create branch from it

   # Or reset your branch to include it:
   git reset --hard def5678

Reflog entries expire after 90 days by default. Don't wait.

Under the hood: The reflog is local-only — it is never pushed or pulled. Each clone has its own independent reflog. This means git reflog only saves you on the machine where the damage happened. If you destroyed commits on a shared branch, another team member's clone may still have the commits — ask them to push before they pull.

Bisect for Bug Hunting

Situation: Something broke, but you don't know which commit caused it.

1. Start bisect:
   git bisect start
   git bisect bad                          # Current commit is broken
   git bisect good v1.2.0                  # This tag was working

2. Git checks out a middle commit. Test it:
   # Run your test/check
   git bisect good    # If this commit works
   # OR
   git bisect bad     # If this commit is broken

3. Repeat until Git identifies the exact commit.
   # Output: abc1234 is the first bad commit

4. Clean up:
   git bisect reset

Automated bisect (much faster):
   git bisect start HEAD v1.2.0
   git bisect run ./test-script.sh
   # Script must exit 0 for good, non-0 for bad

Cherry-Pick Safely

Situation: You need one specific commit from another branch,
not the whole branch.

1. Find the commit:
   git log --oneline other-branch | head -20

2. Cherry-pick:
   git cherry-pick <commit-hash>

3. If there's a conflict:
   # Resolve conflicts, then:
   git add <resolved-files>
   git cherry-pick --continue

4. To cherry-pick a range:
   git cherry-pick <oldest-hash>^..<newest-hash>

Gotcha: cherry-pick creates a NEW commit with a different hash.
If the original branch is later merged, Git usually handles the
duplicate gracefully, but it can cause confusion in the history.

Debug clue: If a cherry-pick causes unexpected conflicts, it is often because the commit depends on earlier commits in the source branch. Use git log --oneline --graph <commit>~5..<commit> to see the commit's context. You may need to cherry-pick the prerequisite commits first, or manually resolve by understanding the missing context.

Undoing a Pushed Commit

Situation: You pushed a commit to a shared branch that needs to go.

Safe way (revert - creates a new commit that undoes the change):
   git revert <commit-hash>
   git push origin main

Unsafe way (rewrite history - ONLY for your own feature branch):
   git reset --hard HEAD~1
   git push --force-with-lease origin feature/my-branch

NEVER force-push to main/develop/shared branches unless you have
coordinated with every team member and understand the consequences.

Gotchas & War Stories

Secrets in Git history Someone committed an AWS key. They deleted it in the next commit and thought they were safe. The key is still in the history. Tools like trufflehog and GitHub's secret scanning will find it. Fix: rotate the secret immediately, then use git filter-repo (successor to git filter-branch) to purge the file from all history. Force-push all branches. Every clone needs to re-clone.

# Find secrets in history
trufflehog git file://. --only-verified
# Or
git log --all --oneline -p | grep -iE "aws_secret|password|api_key"

Large files bloating the repo Someone committed a 500MB database dump. Even after deleting it, the repo is huge because Git stores every version. Prevention: use .gitignore aggressively and set up a pre-commit hook to reject files over a threshold. For legitimate large files, use Git LFS.

# Find large objects in history
git rev-list --objects --all | \
  git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' | \
  awk '/^blob/ {print $3, $4}' | sort -rn | head -10

Merge commits vs squash merge - Merge commit: preserves individual commits, two-parent merge commit in history - Squash merge: collapses all branch commits into one commit on main - Rebase and merge: replays commits on top of main (linear history)

Team decision. Squash is cleaner for main history but loses granular commit info. Choose one strategy and be consistent.

Detached HEAD panic You checked out a tag or a specific commit and now you're in "detached HEAD state." Your commits won't be on any branch. If you made changes you want to keep:

git checkout -b save-my-work
# Now your commits are on a branch

The .gitkeep convention Git doesn't track empty directories. If you need an empty directory in the repo (e.g., logs/), add an empty .gitkeep file inside it. This is a convention, not a Git feature.

Remember: --force-with-lease mnemonic: "force push, but only if my view of the remote is still current." It checks that the remote ref matches what you last fetched. If someone else pushed in the meantime, it fails safely. Always use --force-with-lease instead of --force — it is the seatbelt for force pushes.

Pre-commit hooks for infrastructure repos Set up hooks to catch problems before they're committed:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    hooks:
      - id: check-yaml
      - id: check-json
      - id: detect-private-key
      - id: no-commit-to-branch
        args: ['--branch', 'main']
  - repo: https://github.com/antonbabenko/pre-commit-terraform
    hooks:
      - id: terraform_fmt
      - id: terraform_validate

Day-to-Day Git Workflow

# Starting work on a new feature
git checkout main
git pull origin main
git checkout -b feature/add-monitoring

# Working on changes
git add -p                     # Stage hunks interactively (review what you're committing)

> **One-liner:** `git add -p` is the most underused Git command. It lets you review every change before staging, catch accidental debug prints, and split a messy working tree into logical commits. The `s` (split) option breaks large hunks into smaller ones for surgical staging.
git commit -m "Add Prometheus scrape config for new service"

# Keeping up with main
git fetch origin
git rebase origin/main         # Rebase your feature on latest main
# Resolve any conflicts

# Before creating a PR, clean up
git rebase -i origin/main      # Squash/reword/reorder commits
git push origin feature/add-monitoring

# After PR is approved and merged
git checkout main
git pull origin main
git branch -d feature/add-monitoring    # Delete local branch

---

Quick Reference

• Cheatsheet: Git

Pages that link here

• Anti-Primer: Git
• Git Cheat Sheet
• Git for DevOps
• Tag Cloud
• Thinking Out Loud: Git