SSH Deep Dive — Street-Level Ops¶
Real-world workflows for debugging connections, securing access, and building tunnels in production.
Debugging SSH Connection Failures¶
# Verbose output — the single most useful debugging flag
ssh -vvv user@host 2>&1 | tee /tmp/ssh-debug.log
# What to look for in verbose output:
# "Connection refused" → sshd not running or wrong port
# "Connection timed out" → firewall blocking, wrong IP, host down
# "Permission denied (publickey)" → key not accepted
# "Host key verification failed" → known_hosts mismatch (server rebuilt?)
# "no matching key exchange" → algorithm mismatch
# Test connectivity without SSH (is the port even open?)
nc -zv host 22 -w 5
# or
nmap -p 22 host
# Check if sshd is running on the target (if you have another way in)
systemctl status sshd
ss -tlnp | grep :22
# Test with password auth explicitly (to isolate key issues)
ssh -o PubkeyAuthentication=no user@host
# Test with a specific key
ssh -i ~/.ssh/specific_key -o IdentitiesOnly=yes user@host
# Check what keys the agent offers
ssh-add -l
# Clear a stale known_hosts entry after server rebuild
ssh-keygen -R hostname
ssh-keygen -R 10.0.1.50
Common Permission Errors and Fixes¶
# "Permissions 0644 for '/home/user/.ssh/id_ed25519' are too open"
chmod 600 ~/.ssh/id_ed25519
# "Bad owner or permissions on /home/user/.ssh/config"
chmod 600 ~/.ssh/config
chmod 700 ~/.ssh
# authorized_keys not working on the server
# Check permissions on the remote side:
ls -la ~/.ssh/
# Must be: drwx------ .ssh
# -rw------- authorized_keys
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
# SELinux context issues (RHEL/CentOS)
restorecon -Rv ~/.ssh/
# Home directory permissions (sshd is strict about this)
chmod 755 ~ # or 750 — must NOT be group-writable
Setting Up a Bastion / Jump Host¶
Architecture¶
Internet → [bastion:2222] → [internal-net:22]
├── app-01 (10.0.1.10)
├── app-02 (10.0.1.11)
└── db-01 (10.0.2.20)
Bastion sshd_config¶
# /etc/ssh/sshd_config on the bastion
Port 2222
PermitRootLogin no
PasswordAuthentication no
AllowUsers jump-user
AllowTcpForwarding yes
AllowAgentForwarding no
X11Forwarding no
MaxSessions 10
ClientAliveInterval 300
ClientAliveCountMax 2
LogLevel VERBOSE
Client Configuration¶
# ~/.ssh/config on your workstation
Host bastion
HostName bastion.example.com
User jump-user
Port 2222
IdentityFile ~/.ssh/bastion_key
Host app-01
HostName 10.0.1.10
User deploy
ProxyJump bastion
IdentityFile ~/.ssh/app_key
Host app-02
HostName 10.0.1.11
User deploy
ProxyJump bastion
IdentityFile ~/.ssh/app_key
Host db-01
HostName 10.0.2.20
User dbadmin
ProxyJump bastion
IdentityFile ~/.ssh/db_key
LocalForward 15432 localhost:5432
Now ssh app-01 transparently jumps through the bastion. ssh db-01 also sets up a PostgreSQL tunnel automatically.
SSH Tunnels for Database Access¶
# Forward local port 15432 to remote PostgreSQL
ssh -L 15432:localhost:5432 user@db-server -N
# Then connect locally:
psql -h localhost -p 15432 -U appuser mydb
# Forward through a jump host (db is not directly reachable)
ssh -L 15432:db-01.internal:5432 user@bastion -N
# Forward to RDS through a bastion
ssh -L 15432:mydb.abc123.us-east-1.rds.amazonaws.com:5432 user@bastion -N
# MySQL through a tunnel
ssh -L 13306:db-host:3306 user@bastion -N
mysql -h 127.0.0.1 -P 13306 -u appuser -p
# Redis through a tunnel
ssh -L 16379:redis.internal:6379 user@bastion -N
redis-cli -h 127.0.0.1 -p 16379
# Multiple forwards in one command
ssh -L 15432:db:5432 -L 16379:redis:6379 -L 18080:admin:8080 user@bastion -N
The -N flag means "no shell" — the SSH session stays open only to maintain the tunnel.
SOCKS Proxy for Web Access Through Remote Host¶
# Create a SOCKS5 proxy
ssh -D 1080 user@remote-host -N
# Use with curl
curl --socks5-hostname localhost:1080 http://internal-dashboard.corp:3000/
# Use with browser: set SOCKS5 proxy to localhost:1080
# Firefox: Settings → Network → Manual proxy → SOCKS Host: localhost, Port: 1080
# Check "Proxy DNS when using SOCKS v5"
# Use with any tool via environment
export ALL_PROXY=socks5h://localhost:1080
wget http://internal-wiki.corp/
# SOCKS through a jump host
ssh -J bastion -D 1080 user@internal-host -N
ControlMaster for Speed¶
# Add to ~/.ssh/config
Host *
ControlMaster auto
ControlPath ~/.ssh/sockets/%r@%h-%p
ControlPersist 600
# Create socket directory
mkdir -p ~/.ssh/sockets
# Performance difference:
# Without multiplexing:
time ssh host 'echo hello' # ~0.8s (TCP + key exchange + auth)
# With multiplexing (after first connection):
time ssh host 'echo hello' # ~0.1s (reuses existing connection)
# Huge benefit for tools that make many SSH connections:
# rsync, Ansible, git push, scp loops, parallel-ssh
# Check status of a multiplexed connection
ssh -O check host
# Kill a multiplexed connection
ssh -O exit host
# Force a new connection (bypass multiplex)
ssh -S none host
SSH Escape Sequences for Stuck Sessions¶
# Terminal frozen? SSH hung? Do NOT close the terminal.
# Press Enter, then:
# ~. — Terminate the connection
# ~^Z — Suspend SSH (background it, resume with fg)
# ~# — List forwarded connections
# ~C — Open SSH command line (add forwards on the fly)
# ~? — Show all escape sequences
# Example: add a port forward to an existing session
# Press Enter, ~C, then:
# ssh> -L 15432:localhost:5432
# Forwarding port.
# Note: escape character only recognized after a newline
# If nested SSH (ssh through ssh), use ~~. for the inner session
Hardening sshd in Production¶
# 1. Disable password auth (keys only)
sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
sed -i 's/^PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
# 2. Disable root login
sed -i 's/^PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config
# 3. Restrict to specific users/groups
echo "AllowGroups ssh-users" >> /etc/ssh/sshd_config
# 4. Change default port (reduces noise, not real security)
sed -i 's/^#Port 22/Port 2222/' /etc/ssh/sshd_config
# 5. Set idle timeout
echo "ClientAliveInterval 300" >> /etc/ssh/sshd_config
echo "ClientAliveCountMax 2" >> /etc/ssh/sshd_config
# 6. Limit auth attempts
echo "MaxAuthTries 3" >> /etc/ssh/sshd_config
# 7. Disable unused features
echo "X11Forwarding no" >> /etc/ssh/sshd_config
echo "AllowTcpForwarding no" >> /etc/ssh/sshd_config
# 8. Test config before restarting
sshd -t && systemctl restart sshd
# 9. ALWAYS keep your current session open while testing
# Open a NEW terminal and try to SSH in before closing the old one
SSH Over Restrictive Networks¶
# Many corporate networks block port 22 but allow 443
# Run sshd on port 443 (if no HTTPS on the same host)
# /etc/ssh/sshd_config:
# Port 443
# Or use SSH config to try port 443
Host bastion
HostName bastion.example.com
Port 443
# Behind a proxy that only allows CONNECT to port 443
Host bastion
HostName bastion.example.com
Port 443
ProxyCommand nc -X connect -x proxy.corp:8080 %h %p
# Using corkscrew through an HTTP proxy
Host bastion
ProxyCommand corkscrew proxy.corp 8080 %h %p
# SSH over websocket (when even port 443 raw TCP is blocked)
# Requires wstunnel on both ends
# Server: wstunnel --server ws://0.0.0.0:443 --restrict-to 127.0.0.1:22
# Client:
Host bastion
ProxyCommand wstunnel -L stdio:%h:%p ws://bastion.example.com:443
Managing known_hosts at Scale¶
# Scan and add host keys for an entire subnet
ssh-keyscan 10.0.1.{1..50} >> ~/.ssh/known_hosts 2>/dev/null
# Scan from a list of hostnames
while read host; do
ssh-keyscan "$host" 2>/dev/null
done < hostnames.txt >> ~/.ssh/known_hosts
# Remove all entries for a host (IP and hostname)
ssh-keygen -R web-01.prod
ssh-keygen -R 10.0.1.10
# Deduplicate known_hosts
sort -u ~/.ssh/known_hosts > ~/.ssh/known_hosts.tmp && \
mv ~/.ssh/known_hosts.tmp ~/.ssh/known_hosts
# Verify a specific host key fingerprint
ssh-keyscan -t ed25519 host 2>/dev/null | ssh-keygen -l -f -
# Output: 256 SHA256:abc123... host (ED25519)
# Compare this fingerprint with what your team published
# Use a per-environment known_hosts
Host prod-*
UserKnownHostsFile ~/.ssh/known_hosts.prod
Host staging-*
UserKnownHostsFile ~/.ssh/known_hosts.staging
SSH CA Setup for Teams¶
# 1. Generate the CA key (keep this extremely safe)
ssh-keygen -t ed25519 -f /secure/ssh-ca/user_ca -C "Team SSH User CA"
ssh-keygen -t ed25519 -f /secure/ssh-ca/host_ca -C "Team SSH Host CA"
# 2. Sign a user's public key
ssh-keygen -s /secure/ssh-ca/user_ca \
-I "alice@company.com" \
-n alice,deploy \
-V +90d \
alice_id_ed25519.pub
# 3. Sign a host's public key
ssh-keygen -s /secure/ssh-ca/host_ca \
-I "web-01.prod.company.com" \
-h \
-n web-01.prod.company.com,10.0.1.10 \
-V +365d \
/etc/ssh/ssh_host_ed25519_key.pub
# 4. Configure servers to trust user CA
echo "TrustedUserCAKeys /etc/ssh/user_ca.pub" >> /etc/ssh/sshd_config
# Copy user_ca.pub (public key only!) to /etc/ssh/user_ca.pub
# 5. Configure clients to trust host CA
echo '@cert-authority *.prod.company.com ssh-ed25519 AAAAC3...' >> ~/.ssh/known_hosts
# 6. Inspect a certificate
ssh-keygen -L -f alice_id_ed25519-cert.pub
# Benefits:
# - No more managing authorized_keys on every server
# - No more known_hosts warnings after server rebuilds
# - Certificates expire automatically
# - Audit trail (every cert has an ID)
Bulk SSH Operations¶
# Run a command on multiple hosts
for host in web-0{1..5}.prod; do
echo "=== $host ==="
ssh "$host" 'uptime; df -h / | tail -1'
done
# Parallel execution with GNU parallel
parallel -j 5 ssh {} 'uptime' ::: web-0{1..5}.prod
# Using pssh (parallel-ssh) for fleet operations
pssh -h hostlist.txt -i 'uptime'
pssh -h hostlist.txt -i 'systemctl restart nginx'
# Copy a file to multiple hosts
for host in web-0{1..5}.prod; do
scp deploy.tar.gz "$host":/tmp/
done
# Or with rsync for efficiency
for host in web-0{1..5}.prod; do
rsync -avz ./config/ "$host":/etc/app/config/
done
Power One-Liners¶
Mount remote filesystem over SSH¶
Breakdown: FUSE-based filesystem that tunnels all file operations over SSH. -o reconnect handles network blips. ServerAliveInterval sends keepalives. Unmount with fusermount -u /local/mount.
[!TIP] When to use: Editing remote files with local tools, browsing remote logs, ad-hoc file access without rsync.
Diff remote file against local¶
Breakdown: cat on the remote side streams the file to stdout through the SSH tunnel into diff's stdin (the - argument). No temp files, no scp.
[!TIP] When to use: Verifying config drift between hosts, validating deployment artifacts.
SSH jump through bastion (legacy and modern)¶
# Legacy (still works everywhere):
ssh -t bastion ssh internal-host
# Modern (OpenSSH 7.3+):
ssh -J bastion internal-host
# Permanent in ~/.ssh/config:
Host internal-*
ProxyJump bastion.example.com
Breakdown: -t forces pseudo-tty allocation so the inner ssh works interactively. -J (ProxyJump) is cleaner — it opens a direct TCP tunnel through the bastion without spawning a shell on it.
[!TIP] When to use: Accessing hosts behind firewalls, private subnets, or VPNs.
Persistent SSH connection multiplexing¶
Breakdown: -M establishes a master connection. -N means no remote command. -f backgrounds. Subsequent connections to the same host reuse the socket (configure in ~/.ssh/config with ControlMaster auto and ControlPath).
[!TIP] When to use: Repeated ssh/scp/rsync to the same host — eliminates handshake overhead. Huge speedup for scripts.
SSH tunnel for database access¶
Breakdown: -L local:remote_host:remote_port creates a local port forward. Traffic to localhost:5432 tunnels through bastion to db.internal:5432. -N means no shell. & backgrounds it.
[!TIP] When to use: Accessing databases, dashboards, or internal APIs from your laptop through a bastion host.
Test SSH throughput¶
Breakdown: yes generates infinite output. pv (pipe viewer) measures throughput. ssh tunnels it to the remote cat > /dev/null (discard). Shows your actual SSH transfer rate.
[!TIP] When to use: Diagnosing slow scp/rsync, validating network capacity, comparing compression options.
Undead remote session (autossh + screen)¶
autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -t server 'screen -raAd mysession'
Breakdown: autossh monitors the SSH connection and reconnects automatically on failure. -M 0 disables the monitoring port (uses ServerAlive instead — more firewall-friendly). screen -raAd reattaches to mysession if it exists, or creates it. The result: a practically immortal remote workspace that survives network blips, laptop suspends, and WiFi hops.
[!TIP] When to use: Long-running ops sessions — migrations, deployments, monitoring.
Quick Reference¶
- Cheatsheet: SSH