Terminal Internals - Street Ops

Quick Diagnosis Commands

# Check what TERM is set to (first thing when display is broken)
echo "$TERM"

# See current TTY settings (line discipline, special chars, modes)
stty -a

# Reset a broken terminal (garbled output, no echo, stuck raw mode)
stty sane
# Or the nuclear option — full reset including clearing screen
reset

# Check if stdout is a terminal (useful in scripts)
[ -t 1 ] && echo "interactive" || echo "piped/redirected"

# See terminal dimensions
tput cols; tput lines
# Or: stty size (rows cols)

# Check what terminfo entry is active
infocmp "$TERM" | head -20

# Find which PTY you are on
tty

# Test if your terminal supports 256 colors
tput colors

# List open file descriptors for the current shell
ls -la /proc/$$/fd

Gotcha: Terminal Garbled After Catting a Binary File

Symptom: You accidentally ran cat /usr/bin/something or cat on a binary log file. Now your terminal shows garbage characters, input is not echoed, and the prompt looks wrong. Line endings are broken.

What happened: The binary output contained escape sequences that flipped the TTY into raw mode, changed the character set, or altered terminal settings.

Under the hood: Terminals interpret escape sequences inline with output. A binary file may accidentally contain \x1b[?47h (switch to alternate screen), \x0e (shift to G1 character set), or \x1b[8m (invisible text). These sequences persist until explicitly reversed — which is why reset works but closing and reopening the terminal also works.

# Step 1: Fix it (type blind if you cannot see input)
stty sane
# If that does not work:
reset
# If even that fails, type blindly:
# echo -e '\033c'    (sends ESC-c, the terminal reset sequence)

# Step 2: Prevent it next time
# Use xxd, hexdump, or file instead of cat on unknown files
file suspicious_file
xxd suspicious_file | head -20
# Or use less which handles binary safely:
less suspicious_file

Gotcha: SSH Session Freezes — Appears Dead

Symptom: You are working over SSH and the terminal stops responding. No output, no input, no cursor movement. You are not sure if the connection is dead or if the remote process is hung.

# Step 1: Check if SSH is alive
# Type: ~?  (tilde, question mark — SSH escape help)
# If you see the escape menu, SSH is alive; the remote process is hung
# Type: ~.  (tilde, period — disconnect SSH)

# Important: ~ only works right after a newline
# Press Enter first, then ~.

> **Remember:** SSH escape sequences: `~.` = disconnect, `~^Z` = suspend SSH, `~#` = list forwarded connections, `~?` = help. The tilde must be the first character after a newline. Mnemonic: **tilde-dot drops the connection**.

# Step 2: If SSH escape does not respond, the TCP connection is dead
# Your local terminal is stuck waiting for TCP timeout (can be minutes)
# Configure SSH keepalives to detect this faster:
# In ~/.ssh/config:
#   Host *
#     ServerAliveInterval 15
#     ServerAliveCountMax 3
# This detects dead connections in ~45 seconds instead of minutes

# Step 3: Prevent session loss with tmux
# On remote host, always work inside tmux:
tmux new -s work
# Reconnect after disconnect:
tmux attach -t work

Pattern: tmux Session Management for Ops Work

# Create a named session (always name sessions)
tmux new -s deploy

# Detach cleanly: Ctrl+B, then d

# List sessions
tmux ls

# Reattach to a named session
tmux attach -t deploy

# Kill a session
tmux kill-session -t deploy

# Split panes for monitoring during deploys:
# Ctrl+B %    → vertical split
# Ctrl+B "    → horizontal split
# Ctrl+B o    → cycle between panes
# Ctrl+B z    → zoom/unzoom current pane

# Capture pane output to a file (grab what is on screen)
tmux capture-pane -t deploy:0 -p > /tmp/screen_capture.txt

# Share a session (pair programming/debugging):
# User A: tmux new -s shared
# User B: tmux attach -t shared

Pattern: Debugging Escape Sequence Problems

# See exactly what bytes a key press sends
cat -v
# Press keys; Ctrl+C to exit
# Arrow up shows ^[[A, Home might show ^[[H or ^[OH

# Or use showkey for raw scan codes
sudo showkey -a
# Shows decimal, octal, hex for each keypress

# Check if a terminal supports a specific capability
tput smcup > /dev/null 2>&1 && echo "supports alt screen" || echo "no alt screen"

# Common TERM mismatches and fixes:
# Inside tmux: TERM should be screen-256color or tmux-256color
# Inside screen: TERM should be screen-256color
# Over SSH: TERM should match what the remote terminfo DB has

# Install missing terminfo entries on remote hosts
infocmp tmux-256color > /dev/null 2>&1 || export TERM=screen-256color

> **Default trap:** When TERM is set to `xterm-256color` but the remote host only has `xterm` in its terminfo database, programs like `vim` and `top` will render incorrectly or refuse to start. The fix is to either install the terminfo entry (`tic`) or fall back to a TERM the remote host knows.

# Strip ANSI escape codes from output (for clean logging)
command_with_colors 2>&1 | sed 's/\x1b\[[0-9;]*m//g'

Pattern: Terminal Capability Detection in Scripts

# Check for color support before using colors in scripts
if [ -t 1 ] && [ "$(tput colors 2>/dev/null)" -ge 8 ]; then
    RED=$(tput setaf 1)
    GREEN=$(tput setaf 2)
    RESET=$(tput sgr0)
else
    RED=""
    GREEN=""
    RESET=""
fi

echo "${GREEN}Success${RESET} or ${RED}Failure${RESET}"

# Check terminal width for formatting
COLS=$(tput cols 2>/dev/null || echo 80)
printf '%*s\n' "$COLS" '' | tr ' ' '-'

# Detect if running inside tmux
[ -n "$TMUX" ] && echo "inside tmux"

# Detect if running inside screen
[ "$TERM" = "screen" ] || [ -n "$STY" ] && echo "inside screen"

# Detect if running over SSH
[ -n "$SSH_CLIENT" ] || [ -n "$SSH_TTY" ] && echo "over SSH"

# Handle missing tput gracefully
cols() { tput cols 2>/dev/null || stty size 2>/dev/null | cut -d' ' -f2 || echo 80; }

Pattern: Fixing Locale and Encoding Issues

# Check current locale settings
locale

# If missing locale warnings appear:
sudo locale-gen en_US.UTF-8
sudo update-locale LANG=en_US.UTF-8

# Force UTF-8 for current session
export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8

# Check file encoding:
file -bi document.txt
# Convert encoding
iconv -f ISO-8859-1 -t UTF-8 oldfile.txt > newfile.txt

Pattern: Debugging PTY Allocation Issues

# SSH without a PTY (for scripting — no terminal features)
ssh -T user@host 'uptime; df -h'
# -T disables PTY allocation; command runs and exits

# SSH with forced PTY (for interactive commands over automated SSH)
ssh -tt user@host 'sudo systemctl restart nginx'
# -tt forces PTY; needed for sudo and interactive programs

# When "Pseudo-terminal will not be allocated" appears:
# You are piping into ssh or running non-interactively
# Use -tt to force allocation if you need terminal features

# Check PTY limits (ran out of PTYs — rare but happens)
cat /proc/sys/kernel/pty/nr    # current count
cat /proc/sys/kernel/pty/max   # maximum allowed
# If nr approaches max, find stale PTY holders: fuser /dev/pts/*

Debug clue: If ssh -tt commands hang but ssh -T works fine, the remote host may be out of PTYs. Check /proc/sys/kernel/pty/nr against /proc/sys/kernel/pty/max. Zombie SSH sessions from broken connections are the usual culprit — kill them with pkill -9 -t pts/N.

Useful One-Liners

# Record a terminal session (for documentation or debugging)
script /tmp/session.log
# Everything you type and see is recorded until you type 'exit'

# Broadcast a message to all terminals
wall "Maintenance starting in 5 minutes"

# Find processes attached to a specific terminal
fuser /dev/pts/3

# Check if a background process still has a controlling terminal
ps -o pid,tty,stat,cmd -p $PID
# If TTY is '?', it has no controlling terminal (daemonized)

# Force a program to think it has a terminal (for CI/scripts)
# script -qc 'command' /dev/null   ← wraps command in a PTY
# unbuffer command                  ← from expect package

Pages that link here

• Anti-Primer: Terminal Internals
• Terminal Internals