Skip to content

Portal | Level: L1: Foundations | Topics: Terminal Internals, Bash / Shell Scripting | Domain: Linux

Terminal Internals - Primer

Why This Matters

People say "the terminal" as if it is one thing. It is not. It is four distinct layers — terminal emulator, TTY driver, shell, and programs — stacked on top of each other and pretending to be unified. When something breaks, you cannot fix it unless you know which layer owns the problem.

For DevOps and SRE work, terminal internals explain why SSH sessions behave differently from local terminals, why tmux sometimes eats your keybindings, why Ctrl+C does not always kill things, and why piping behaves differently from interactive use. These are not academic concerns — they show up in automation, remote debugging, and CI/CD pipelines constantly.

Understanding these layers also makes you faster. Job control, redirects, file descriptors, and PATH resolution are daily tools. Knowing how they actually work turns guesswork into precision.

Core Concepts

1. The Four Layers

Every terminal interaction passes through these layers:

+---------------------------------------------+
|  Terminal Emulator                           |
|  (iTerm2, GNOME Terminal, xterm, kitty)      |
|  Converts keystrokes -> bytes, bytes -> display
+---------------------------------------------+
|  TTY Driver / PTY (kernel)                   |
|  Line discipline, signals, terminal modes    |
|  Sits between emulator and programs          |
+---------------------------------------------+
|  Shell (bash, zsh, fish)                     |
|  Parses commands, expands globs,             |
|  manages jobs, sets up redirects/pipes,      |
|  launches programs                           |
+---------------------------------------------+
|  Programs (vim, python, curl, grep, top)     |
|  Read/write bytes, may send escape sequences |
+---------------------------------------------+

When something goes wrong, always ask: which layer owns this behavior?

Symptom Likely Layer
Colors not rendering Terminal emulator or TERM variable
Arrow keys print ^[[D Emulator/TTY mode mismatch
Ctrl+C does nothing Program trapping SIGINT
Command not found Shell (PATH resolution)
Backspace prints ^H TTY driver settings (stty)
Broken prompt after binary cat TTY driver in wrong mode

Name origin: "TTY" is short for "teletypewriter" — a physical device from the 1960s that printed output on paper and sent keystrokes over a serial line. When Unix was developed at Bell Labs (1969-1971), teletypewriters were the standard interface. The kernel's TTY subsystem was written to manage these physical devices. Decades later, we still use the TTY abstraction even though physical teletypewriters are museum pieces. A "pseudo-terminal" (PTY) is a software emulation of a TTY — every terminal emulator, SSH session, and tmux window creates one.

Fun fact: The reason Ctrl+C sends byte 0x03 dates back to ASCII (American Standard Code for Information Interchange, 1963). ASCII codes 0-31 are "control characters" — pressing Ctrl with a letter produces the ASCII code for that letter minus 64. So Ctrl+C = 67 - 64 = 3 (ETX, End of Text). Ctrl+D = 68 - 64 = 4 (EOT, End of Transmission). This mapping is hardwired into the ASCII table and has not changed since 1963.

2. How Keystrokes Become Actions

You press Enter:
  1. Terminal emulator sends byte 0x0D (carriage return)
  2. TTY driver may translate to 0x0A (newline) based on settings
  3. Shell receives the newline, parses the buffered command
  4. Shell forks, sets up redirects, execs the program

You press Ctrl+C:
  1. Terminal emulator sends byte 0x03
  2. TTY driver intercepts it (not the shell or program)
  3. TTY driver sends SIGINT to the foreground process group
  4. Program receives the signal and (usually) dies

Key insight: Ctrl+C is handled by the kernel TTY driver, not the shell. That is why it works even when the shell is not processing input.

3. PATH Resolution

When you type a command, the shell searches for it:

# See your PATH (colon-separated directory list)
echo "$PATH"

# See it one directory per line
echo "$PATH" | tr ':' '\n'

# Find where a command lives
type ls            # shell built-in awareness
command -v python3 # POSIX-portable
which -a python3   # all matches in PATH

Rules: - Directories are searched left to right; first match wins - The shell caches lookups in a hash table (hash -r to clear) - Relative paths and ./ bypass PATH entirely - Scripts invoked by cron, systemd, or CI may have a different PATH

4. File Descriptors and Streams

Every process starts with three open file descriptors:

FD Name Default Purpose
0 stdin keyboard/pipe Input
1 stdout terminal/pipe Normal output
2 stderr terminal Error output

The shell sets up redirects before the program starts:

# Redirect stdout to a file
cmd > out.txt

# Redirect stderr to a file
cmd 2> err.txt

# Redirect both stdout and stderr to the same file
cmd > all.txt 2>&1

# Feed a file as stdin
cmd < input.txt

# Pipe: stdout of producer → stdin of consumer
producer | consumer

# Discard output
cmd > /dev/null 2>&1

Common mistake: cmd 2>&1 > file is not the same as cmd > file 2>&1. Redirections are processed left to right. In the first form, stderr goes to the original stdout (terminal), then stdout goes to the file. Order matters.

5. Job Control

The shell manages background and foreground process groups:

# Run in background
long_command &

# Suspend the foreground job
# Press Ctrl+Z (sends SIGTSTP via TTY driver)

# Resume in background
bg

# Bring back to foreground
fg

# List jobs
jobs

# Send a signal to a job
kill %1        # SIGTERM to job 1
kill -9 %1     # SIGKILL to job 1

When you close a terminal, the kernel sends SIGHUP to all attached jobs. Protect long-running commands:

nohup long_command &       # Ignore SIGHUP
disown %1                  # Remove from shell's job table
tmux / screen              # Full session persistence

6. Escape Sequences and TERM

Programs control the terminal by sending escape sequences — special byte patterns that the terminal emulator interprets as commands:

\033[31m   → set text color to red
\033[0m    → reset formatting
\033[2J    → clear screen
\033[H     → move cursor to top-left

The TERM environment variable tells programs what escape sequences the terminal understands. If it is wrong, things break:

# Check current TERM
echo "$TERM"
# Common values: xterm-256color, screen-256color, tmux-256color

# See what your terminal supports
infocmp "$TERM"

When you see garbage like ^[[D or ^[[32m in output, one of these is true: - TERM is set incorrectly - The program is sending sequences the emulator does not support - You piped terminal-aware output through something that strips nothing

7. Canonical vs Raw Mode

The TTY driver has two major modes:

Mode Behavior Used By
Canonical (cooked) Line-buffered, shell editing works bash prompt, cat
Raw (cbreak) Characters sent immediately, no buffering vim, top, less, REPLs 
# Inspect current TTY settings
stty -a

# If your terminal is broken (no echo, wrong mode):
stty sane    # Reset to sane defaults
reset        # Full terminal reset (also clears screen)

A common cause of "broken terminal": you cat a binary file, and the random bytes flip the TTY into raw mode or change settings. stty sane or reset fixes it.

Debug clue: If typing reset after a broken terminal does not work (because echo is off and you cannot see what you type), try this sequence blind: press Enter, type reset, press Enter. The command executes even if you cannot see it. If that fails, stty sane followed by Enter is the fallback. Both commands restore the TTY to a usable state. In extreme cases, close the terminal and open a new one — tmux users can detach with Ctrl+B d first to preserve the session.

8. SSH and tmux — Extra Layers

SSH adds a layer: local terminal emulator → SSH client → network → SSH server → PTY on the remote host → shell → programs.

tmux/screen adds another: the outer PTY connects to tmux, which creates inner PTYs for each window. This means:

  • TERM inside tmux is screen-256color or tmux-256color, not your outer TERM
  • Some escape sequences are intercepted by tmux before reaching your program
  • Ctrl+B (tmux prefix) is consumed by tmux, not forwarded

When debugging terminal issues over SSH or inside tmux, check TERM at each layer.

What Experienced People Know

  • When something is weird, first ask "which layer?" — most people debug the wrong layer and waste time
  • stty sane fixes most "my terminal is broken" situations faster than closing and reopening
  • Ctrl+C sends SIGINT via the TTY driver, not the shell — programs can and do trap it, which is why it sometimes "does not work"
  • Ctrl+\ sends SIGQUIT, which most programs do not trap — it is the "harder" kill when Ctrl+C fails (and often produces a core dump)
  • Redirect order matters: 2>&1 >file and >file 2>&1 do different things, and getting this wrong is one of the most common shell scripting bugs
  • set -o pipefail in bash makes pipelines fail if any component fails, not just the last one — without it, failing_cmd | grep pattern returns grep's exit code
  • Background jobs in scripts behave differently than interactive jobs — there is no job control in non-interactive shells by default
  • If a program's output looks different when piped vs displayed in a terminal, the program is checking isatty(1) and changing behavior (colors, buffering, progress bars)
  • The shell's hash table can cause "command not found" after installing new software — hash -r clears it
  • In containers, there is often no TTY at all — docker run without -t means stdin is not a terminal, and interactive programs will fail or behave oddly

Wiki Navigation

Prerequisites

Related Content