tmux & screen - Street-Level Ops¶

Real-world workflows for using terminal multiplexers in production operations.

Long-Running Operations in tmux¶

The number one rule: never run a long operation over bare SSH. Always wrap it in tmux first.

# Before starting a database migration
tmux new-session -s db-migration
# Now run the migration inside tmux
pg_dump production | psql staging

# Before a kernel upgrade
tmux new-session -s kernel-upgrade
sudo apt update && sudo apt upgrade -y linux-image-generic && sudo reboot

# Before a large file transfer
tmux new-session -s data-sync
rsync -avz --progress /data/ remote:/backup/data/

If your SSH drops, reconnect and reattach:

ssh prod-db-01
tmux attach -t db-migration
# Your migration is still running

Multi-Pane Monitoring Dashboard¶

Build a monitoring layout for incident response:

#!/bin/bash
# monitoring-layout.sh — 4-pane monitoring dashboard

SESSION="monitor"

tmux new-session -d -s "$SESSION" -x 200 -y 50

# Pane 0: system metrics
tmux send-keys -t "$SESSION" "htop" Enter

# Pane 1: application logs (right side)
tmux split-window -h -t "$SESSION"
tmux send-keys -t "$SESSION" "journalctl -u myapp -f" Enter

# Pane 2: network connections (bottom left)
tmux split-window -v -t "$SESSION:0.0"
tmux send-keys -t "$SESSION:0.2" "watch -n 2 'ss -tunapl | head -30'" Enter

# Pane 3: disk I/O (bottom right)
tmux split-window -v -t "$SESSION:0.1"
tmux send-keys -t "$SESSION:0.3" "iostat -xz 2" Enter

# Set even layout
tmux select-layout -t "$SESSION" tiled

# Attach
tmux attach -t "$SESSION"

Result:

┌───────────────────┬────────────────────┐
│      htop         │  journalctl -f     │
│                   │                    │
├───────────────────┼────────────────────┤
│   ss -tunapl      │    iostat -xz 2    │
│                   │                    │
└───────────────────┴────────────────────┘

Kubernetes Incident Response Layout¶

#!/bin/bash
# k8s-incident.sh — k8s debugging layout

SESSION="k8s-incident"
NS="${1:-default}"

tmux new-session -d -s "$SESSION"

# Pane 0: pod status (auto-refresh)
tmux send-keys "watch -n 5 'kubectl get pods -n $NS -o wide'" Enter

# Pane 1: events stream
tmux split-window -h -t "$SESSION"
tmux send-keys "kubectl get events -n $NS --watch" Enter

# Pane 2: logs from problem pod (manual — fill in pod name)
tmux split-window -v -t "$SESSION:0.0"
tmux send-keys "# kubectl logs -f <pod-name> -n $NS" Enter

# Pane 3: interactive shell for kubectl commands
tmux split-window -v -t "$SESSION:0.1"
tmux send-keys "kubectl config current-context && echo '---' && kubectl get nodes" Enter

tmux select-layout -t "$SESSION" tiled
tmux attach -t "$SESSION"

Shared Sessions for Pair Debugging¶

Two engineers can attach to the same tmux session and see identical output in real time:

# Engineer 1 (on shared server)
tmux new-session -s incident-42

# Engineer 2 (SSHs to the same server)
tmux attach -t incident-42

# Both engineers now see and can type in the same session
# Ctrl-b : to enter command mode and type:
#   setw synchronize-panes on
# Now keystrokes go to ALL panes simultaneously (useful for multi-server ops)

For read-only viewing (one engineer watches, the other drives):

# Viewer attaches in read-only mode
tmux attach -t incident-42 -r

Automating Layouts with tmuxinator¶

tmuxinator defines reusable session layouts in YAML:

# Install
gem install tmuxinator
# or
pip install tmuxp  # Python alternative

# Create a project
tmuxinator new ops-dashboard

# ~/.config/tmuxinator/ops-dashboard.yml
name: ops-dashboard
root: ~/

windows:
  - monitoring:
      layout: tiled
      panes:
        - htop
        - journalctl -u nginx -f
        - watch -n 5 'df -h | grep -v tmpfs'
        - tail -f /var/log/syslog
  - shell:
      panes:
        -   # Empty pane for commands
  - logs:
      layout: even-horizontal
      panes:
        - tail -f /var/log/nginx/access.log
        - tail -f /var/log/nginx/error.log

# Launch the layout
tmuxinator start ops-dashboard

# Stop (kills the session)
tmuxinator stop ops-dashboard

# List available layouts
tmuxinator list

Capturing Output for Postmortems¶

During an incident, capture everything happening in your panes:

# Start logging a pane to a file (do this FIRST when incident starts)
tmux pipe-pane -t incident-42:0.0 "cat >> ~/postmortem/pane0-$(date +%Y%m%d-%H%M%S).log"
tmux pipe-pane -t incident-42:0.1 "cat >> ~/postmortem/pane1-$(date +%Y%m%d-%H%M%S).log"

# Capture current visible content + scrollback
tmux capture-pane -t incident-42:0.0 -p -S - > ~/postmortem/scrollback-pane0.txt

# Stop logging when incident is over
tmux pipe-pane -t incident-42:0.0 ""

Script it for all panes:

#!/bin/bash
# capture-all-panes.sh — capture every pane in a session
SESSION="$1"
OUTDIR="$HOME/postmortem/$(date +%Y%m%d-%H%M%S)"
mkdir -p "$OUTDIR"

for pane in $(tmux list-panes -s -t "$SESSION" -F '#{window_index}.#{pane_index}'); do
    tmux capture-pane -t "$SESSION:$pane" -p -S - > "$OUTDIR/pane-${pane}.txt"
done

echo "Captured to $OUTDIR"
ls -la "$OUTDIR"

tmux + SSH for Serial Console Access¶

When managing bare-metal servers or network gear, tmux keeps your serial console alive:

# Start tmux before connecting to serial console via SSH jump
tmux new-session -s console-sw01

# Connect through jump host to serial console
ssh -J jumphost serialconsole@172.16.0.1

# If SSH to jump host drops, tmux holds the session
# Reconnect: tmux attach -t console-sw01

For multiple console connections:

tmux new-session -s consoles

# Window per device
tmux send-keys "ssh -J jump serial@switch-01" Enter
tmux new-window -t consoles
tmux send-keys "ssh -J jump serial@switch-02" Enter
tmux new-window -t consoles
tmux send-keys "ssh -J jump serial@router-01" Enter

# Navigate: Ctrl-b 0, Ctrl-b 1, Ctrl-b 2

Recovering Orphaned tmux Sessions¶

After a server reboot or crash, check for surviving sessions:

# List all tmux sessions
tmux ls

# If tmux server is dead but socket file exists
ls -la /tmp/tmux-$(id -u)/

# If you get "no server running on /tmp/tmux-1000/default"
# The server died — sessions are gone. This is why tmux-resurrect matters.

# Check if another user has tmux sessions (as root)
ls -la /tmp/tmux-*/

# Force-kill a stuck tmux server
tmux kill-server

# Reconnect to a session that lost its client
tmux attach -d -t infra
# -d detaches any other clients first

Gotcha: After a server reboot, tmux sessions do not survive -- they are held in memory, not on disk. If you need session persistence across reboots, install tmux-resurrect or tmux-continuum. Without these plugins, the only thing that survives a reboot is your session layout YAML files (tmuxinator/tmuxp). ```text

Scripted Pane Layouts for Common Workflows¶

Deploy Watcher¶

```bash

!/bin/bash¶

deploy-watch.sh — monitor a deployment across multiple signals¶

APP="${1:?Usage: deploy-watch.sh }" SESSION="deploy-$APP"

tmux new-session -d -s "$SESSION"

Top-left: deployment status¶

tmux send-keys "watch -n 3 'kubectl rollout status deployment/$APP 2>&1'" Enter

Top-right: pod status¶

tmux split-window -h tmux send-keys "watch -n 3 'kubectl get pods -l app=$APP -o wide'" Enter

Bottom-left: application logs¶

tmux split-window -v -t 0.0 tmux send-keys "kubectl logs -f deployment/$APP --tail=50 --all-containers" Enter

Bottom-right: events¶

tmux split-window -v -t 0.1 tmux send-keys "kubectl get events --field-selector involvedObject.name=$APP --watch" Enter

tmux select-layout tiled tmux attach -t "$SESSION" ```text

Multi-Server Command Execution¶

```bash

!/bin/bash¶

multi-server.sh — open panes to multiple servers, optionally synchronized¶

SERVERS=("web-01" "web-02" "web-03" "db-01") SESSION="multi"

tmux new-session -d -s "$SESSION" "ssh ${SERVERS[0]}"

for server in "${SERVERS[@]:1}"; do tmux split-window -t "$SESSION" "ssh $server" tmux select-layout -t "$SESSION" tiled done

Uncomment to synchronize input across all panes:¶

tmux setw -t "$SESSION" synchronize-panes on¶

tmux attach -t "$SESSION" ```text

With synchronized panes, every keystroke goes to every server simultaneously — useful for verifying config changes across a fleet, but dangerous if you are not careful about what you type.

War story: An engineer had synchronize-panes enabled across 12 production servers and typed rm -rf /tmp/deploy-* in what they thought was a single pane. It ran on all 12 simultaneously. Fortunately it was just tmp files -- but the habit of running destructive commands with sync on has ended careers. Always toggle sync off (Ctrl-b : setw synchronize-panes off) before running anything you would not want on every server.

screen Emergency Workflows¶

When tmux is not available:

```bash

Start a named screen session for the operation¶

screen -S upgrade

Run your long operation¶

apt upgrade -y

Detach: Ctrl-a d¶

Reconnect from another SSH session¶

screen -r upgrade

If screen says "attached" elsewhere, force takeover¶

screen -dr upgrade

-d detaches the other client, -r reattaches to you¶

List all screen sessions¶

screen -ls

There are screens on:¶

12345.upgrade (Detached)¶

12346.monitor (Attached)¶

Kill a dead screen session¶

screen -X -S 12345.upgrade quit ```text

tmux & screen - Street-Level Ops¶

Long-Running Operations in tmux¶

Multi-Pane Monitoring Dashboard¶

Kubernetes Incident Response Layout¶

Shared Sessions for Pair Debugging¶

Automating Layouts with tmuxinator¶

Capturing Output for Postmortems¶

tmux + SSH for Serial Console Access¶

Recovering Orphaned tmux Sessions¶

Scripted Pane Layouts for Common Workflows¶

Deploy Watcher¶

!/bin/bash¶

deploy-watch.sh — monitor a deployment across multiple signals¶

Top-left: deployment status¶

Top-right: pod status¶

Bottom-left: application logs¶

Bottom-right: events¶

Multi-Server Command Execution¶

!/bin/bash¶

multi-server.sh — open panes to multiple servers, optionally synchronized¶

Uncomment to synchronize input across all panes:¶

tmux setw -t "$SESSION" synchronize-panes on¶

screen Emergency Workflows¶

Start a named screen session for the operation¶

Run your long operation¶

Detach: Ctrl-a d¶

Reconnect from another SSH session¶

If screen says "attached" elsewhere, force takeover¶

-d detaches the other client, -r reattaches to you¶

List all screen sessions¶

There are screens on:¶

12345.upgrade (Detached)¶

12346.monitor (Attached)¶

Kill a dead screen session¶

Pages that link here¶