Pipes & Redirection - Footguns¶

Mistakes that cause lost data, silent failures, broken scripts, and hours of debugging.

1. Variable Assignments in Pipelines Are Lost¶

Each pipeline segment runs in a subshell. Variables set inside disappear when the pipeline finishes.

# BROKEN: count is always 0
count=0
cat data.txt | while read -r line; do
    count=$(( count + 1 ))
done
echo "Lines: ${count}"    # Prints 0

# Fix: redirect input instead of piping
count=0
while read -r line; do
    count=$(( count + 1 ))
done < data.txt
echo "Lines: ${count}"    # Correct

This is the most common pipe-related bug. Any variable, array, or flag set inside a while read loop fed by a pipe is silently lost.

Under the hood: POSIX requires that each command in a pipeline may run in a subshell. Bash runs all pipeline segments in subshells by default. Bash 4.2+ offers shopt -s lastpipe, which runs the last command of a pipeline in the current shell (only in non-interactive scripts) — but the safer fix is to avoid the pipe entirely by using redirection (< file) or process substitution (< <(command)).

2. `2>&1` Order Matters¶

Redirections are evaluated left to right. Getting the order wrong sends output to the wrong place.

# WRONG: stderr goes to terminal, stdout to file
command 2>&1 > output.txt

# RIGHT: both go to file
command > output.txt 2>&1

# Or use bash shorthand
command &> output.txt

3. Clobbering Files with `>`¶

> truncates the file before the command starts reading it. You cannot read from and write to the same file.

# DISASTER: data.txt is now empty
sort data.txt > data.txt

# Fix: temp file
sort data.txt > data.txt.tmp && mv data.txt.tmp data.txt

# Fix: sponge (moreutils)
sort data.txt | sponge data.txt

# Prevention
set -o noclobber   # > fails on existing files; use >| to override

4. Buffering in Pipes¶

Programs use full buffering (4-8KB blocks) when stdout is a pipe, versus line-buffering for terminals. This causes delayed output in monitoring pipelines.

# Output comes in bursts, not real-time
tail -f /var/log/app.log | grep "ERROR"

# Fix: force line buffering
tail -f /var/log/app.log | grep --line-buffered "ERROR"

# Fix: stdbuf
tail -f /var/log/app.log | stdbuf -oL grep "ERROR"

# Python is also affected
python3 -u myscript.py | tee output.log   # -u disables buffering

5. Forgetting `pipefail` Causes Silent Failures¶

Without pipefail, a pipeline's exit code is the exit code of the last command. Early failures are invisible.

# curl fails, jq succeeds on empty input — script continues with garbage
curl https://broken-url/data.json | jq '.results' > output.json
echo $?   # 0

# Fix: pipefail catches it
set -o pipefail
curl https://broken-url/data.json | jq '.results' > output.json
echo $?   # non-zero

Every production script needs set -o pipefail. Without it, broken pipelines produce empty or corrupt output silently.

6. PIPESTATUS Only Works in Bash¶

PIPESTATUS does not exist in POSIX sh, dash, or most non-bash shells. Also, it is overwritten by every subsequent command.

#!/usr/bin/env bash
false | true
echo "checking"                # overwrites PIPESTATUS
echo "${PIPESTATUS[0]}"       # shows echo's exit code, not the pipeline

# Fix: capture immediately
false | true
pipe_status=("${PIPESTATUS[@]}")
echo "First command: ${pipe_status[0]}"

7. Here Document Indentation¶

<<- strips leading tabs only, not spaces. Most editors insert spaces, causing the heredoc delimiter to go unrecognized.

# BROKEN: spaces not stripped
function gen_config() {
    cat <<-EOF
        server { listen 80; }
    EOF     # indented with spaces — heredoc never terminates
}

# Safest: do not indent the delimiter
function gen_config() {
    cat <<EOF
server { listen 80; }
EOF
}

8. Process Substitution Race Conditions¶

Process substitution <() and >() run asynchronously. The writing side may not finish before the next command executes.

# Output process may still be running when "Done" prints
command > >(process_output)
echo "Done"

# Named pipes from mkfifo persist if script exits before cleanup
mkfifo /tmp/mypipe
trap 'rm -f /tmp/mypipe' EXIT

9. echo vs printf in Pipelines¶

echo behavior with backslash escapes varies between shells and systems. Data can be silently mangled.

# Inconsistent: some systems print -e literally
echo -e "line1\nline2"

# Safe: printf is consistent everywhere
printf "line1\nline2\n"

# Data integrity in pipelines
echo "${user_input}" | base64        # may mangle backslashes
printf "%s\n" "${user_input}" | base64   # safe

Use printf in scripts that handle arbitrary data.

10. `/dev/null` Does Not Skip Execution¶

Redirecting to /dev/null discards output. The command still runs fully with all side effects.

rm -rf /tmp/data > /dev/null   # still deletes everything
curl https://example.com > /dev/null   # still makes the request

Pipes & Redirection - Footguns¶

1. Variable Assignments in Pipelines Are Lost¶

2. 2>&1 Order Matters¶

3. Clobbering Files with >¶