Skip to content

Portal | Level: L1: Foundations | Topics: sed, Bash / Shell Scripting | Domain: CLI Tools

sed — The Stream Editor Primer

Why This Matters

sed is the original Unix stream editor, built for one job: transforming text as it flows through a pipe. Every log file, config file, and command output you encounter in operations is text that sed can slice, substitute, and reshape without opening an editor or writing a script. It composes with pipes, it works on streams, and it is available on every Unix system you will ever touch.

If you cannot reach for sed when you need a quick find-and-replace or a surgical config edit, you are either writing throwaway Python scripts for five-second problems or doing it by hand. Neither scales.

Mental Model

sed reads input line by line into a pattern space, applies commands to it, and prints the result. It never opens a file in an editor — it processes a stream. Think of it as a programmable find-and-replace that operates on every line flowing through a pipe.

The basic cycle: 1. Read a line into the pattern space 2. Apply all commands whose addresses match 3. Print the pattern space (unless -n suppresses it) 4. Clear the pattern space and repeat

Address Selection

Every sed command can be prefixed with an address that limits which lines it applies to.

# Line number: apply only to line 5
sed '5s/old/new/' file.txt

# Line range: lines 10 through 20
sed '10,20s/old/new/' file.txt

# Regex: lines matching a pattern
sed '/ERROR/s/old/new/' file.txt

# Regex range: from first match to second match
sed '/BEGIN/,/END/d' file.txt

# Last line
sed '$d' file.txt

# Every 3rd line starting from line 1 (GNU sed)
sed '1~3s/old/new/' file.txt

# Negation: all lines NOT matching
sed '/^#/!s/foo/bar/' file.txt

Fun fact: sed was written by Lee McMahon at Bell Labs in 1973-1974, making it one of the oldest Unix tools still in daily use. It was designed as a non-interactive version of the ed editor — the name literally means "stream editor." The s/old/new/ syntax comes directly from ed's substitute command. This same syntax later influenced Perl, vim, and even modern IDE find-and-replace.

Remember: sed's default behavior: read, execute, print, repeat. The -n flag suppresses auto-print, so only explicit p commands produce output. This is the key to using sed as a filter: sed -n '/pattern/p' acts like grep.

The Substitute Command

s/pattern/replacement/flags is the command you will use 90% of the time.

# Basic substitution (first occurrence per line)
sed 's/error/ERROR/' logfile.txt

# Global: all occurrences per line
sed 's/error/ERROR/g' logfile.txt

# Case-insensitive (GNU sed)
sed 's/error/ERROR/gI' logfile.txt

# Replace only the 2nd occurrence per line
sed 's/foo/bar/2' file.txt

# Print only lines where substitution happened
sed -n 's/error/ERROR/gp' logfile.txt

Regex: BRE vs ERE

By default, sed uses Basic Regular Expressions (BRE). This means +, ?, {, (, and | are literal characters unless escaped.

# BRE: grouping requires escaped parens
sed 's/\(foo\)\(bar\)/\2\1/' file.txt

# ERE with -E: grouping uses normal parens
sed -E 's/(foo)(bar)/\2\1/' file.txt

# BRE: one or more requires \+
sed 's/[0-9]\+/NUM/' file.txt

# ERE with -E: one or more uses +
sed -E 's/[0-9]+/NUM/' file.txt

Always use -E (extended regex) unless you are writing for ancient systems. It matches what you learned in every regex tutorial.

Gotcha: macOS ships BSD sed, which behaves differently from GNU sed on Linux. The biggest difference: -i (in-place) on macOS requires an explicit backup extension: sed -i '' 's/old/new/' file (empty string = no backup). On Linux, sed -i 's/old/new/' file works without the extra argument. Scripts that work on Linux break on macOS and vice versa. For portability, always use sed -i.bak (works on both) or use perl -pi -e instead.

In-Place Editing

# Edit file in place (GNU/Linux)
sed -i 's/old/new/g' file.txt

# Edit file in place (macOS / BSD) — requires backup suffix
sed -i '' 's/old/new/g' file.txt

# In-place with backup
sed -i.bak 's/old/new/g' file.txt
# Creates file.txt.bak with original content

Delete, Insert, Append

# Delete lines matching a pattern
sed '/^#/d' config.txt          # Remove comment lines
sed '/^$/d' file.txt            # Remove blank lines
sed '1,5d' file.txt             # Delete first 5 lines

# Insert BEFORE a matching line
sed '/\[production\]/i\# Added by deploy script' config.ini

# Append AFTER a matching line
sed '/\[production\]/a\new_setting = true' config.ini

# Change (replace entire line)
sed '/^hostname=/c\hostname=prod-web-01' config.txt

Multi-Line Processing and Hold Space

Under the hood: The hold space was a brilliant design choice for 1974 — it gave sed a second register to store data between lines, enabling multi-line operations on a stream processor with minimal memory. The h, H, g, G, x commands shuttle text between the pattern space and hold space. Think of it as two clipboards: the pattern space is your active workspace, the hold space is your scratch pad. In practice, if you need more than basic hold-space operations, reach for awk or Perl.

sed has two buffers: the pattern space (current line) and the hold space (a scratch buffer).

# N: append next line to pattern space (separated by \n)
# Joins every pair of lines
sed 'N;s/\n/ /' file.txt

# H: append pattern space to hold space
# g: replace pattern space with hold space
# x: swap pattern space and hold space

# Reverse a file (tac equivalent)
sed -n '1!G;h;$p' file.txt

# Delete blank line after every pattern match
sed '/PATTERN/{n;/^$/d}' file.txt

# Print paragraphs containing PATTERN (paragraphs separated by blank lines)
sed -n '/PATTERN/,/^$/p' file.txt

For anything beyond basic hold-space work, consider awk or a real scripting language. Hold-space gymnastics are clever but unmaintainable.

Practical sed Patterns

One-liner: The most useful sed command for DevOps: sed -i 's/^#\(.*SETTING.*\)/\1/' config.file — uncomment a specific setting in a config file without opening an editor. Pair it with Ansible's lineinfile module for fleet-wide config changes.

# Strip leading/trailing whitespace
sed -E 's/^[[:space:]]+//; s/[[:space:]]+$//' file.txt

# Remove HTML tags
sed -E 's/<[^>]+>//g' page.html

# Extract value from key=value lines
sed -n 's/^DB_HOST=//p' .env

# Comment out a line
sed '/^dangerous_setting/s/^/# /' config.txt

# Uncomment a line
sed 's/^# \(dangerous_setting\)/\1/' config.txt

# Replace text between markers (leave markers intact)
sed '/^---START---$/,/^---END---$/{ /^---/!d; }' file.txt

sed vs awk — When to Reach for Which

Task Tool Why
Simple find-and-replace sed One-liner, no field parsing needed
Delete/insert lines sed Address-based line operations
Extract specific fields awk Built-in field splitting
Math on columns awk Has arithmetic operators
In-place file editing sed -i flag
Quick regex substitution in a pipe sed Lighter syntax for simple cases

When the task involves fields, numbers, or any form of aggregation, reach for awk. When the task is substitution or line-level deletion, sed is your tool.

Interview tip: If asked "when would you use sed vs awk vs perl?", the clean answer is: sed for simple substitutions and line deletions (one-liners), awk for field-based extraction and column math, Perl for anything more complex. sed and awk are always installed; Perl usually is. Python is better for anything over ~5 lines. The goal is to pick the lightest tool that gets the job done.

Wiki Navigation

Prerequisites

Related Content