Skip to content

Portal | Level: L1: Foundations | Topics: SSH Deep Dive, Linux Fundamentals, Linux Hardening | Domain: Linux

SSH Deep Dive — Primer

Why This Matters

SSH is the front door to every server you manage. It is the protocol behind ssh, scp, sftp, rsync, git push, Ansible, and half the tools in your stack. When SSH is misconfigured, you are locked out. When SSH is poorly secured, attackers walk in. Understanding SSH at the protocol level — not just ssh user@host — is the difference between debugging connection failures in minutes versus hours and between a hardened bastion and an open door.

Protocol Basics

Timeline: SSH was created by Tatu Ylonen at Helsinki University of Technology in 1995, after a password-sniffing attack on the university network. SSH-1 was free software; SSH-2 became proprietary. OpenSSH, started by the OpenBSD project in 1999, is the open-source implementation running on virtually every Linux/Unix system today.

What Happens When You Type ssh user@host

  1. TCP connection to port 22 (or configured port)
  2. Protocol version exchange — client and server agree on SSH-2
  3. Key exchange (Diffie-Hellman or ECDH) — establishes shared secret without transmitting it
  4. Server authentication — server proves its identity via host key
  5. Encryption activated — all traffic from here is encrypted
  6. User authentication — client proves identity (key, password, certificate, FIDO2)
  7. Channel opened — interactive shell, exec command, or port forward

The key exchange is the critical part. It ensures that even if someone records the entire session, they cannot decrypt it later (forward secrecy with ephemeral DH/ECDH keys).

Encryption Layers

SSH provides three security properties: - Confidentiality — AES-256-GCM, ChaCha20-Poly1305, or AES-256-CTR encrypt the payload - Integrity — HMAC or AEAD tags detect tampering - Authentication — host keys verify the server; user keys/passwords verify the client

Key Types

Generating Keys

# Ed25519 — recommended (fast, short keys, strong security)
ssh-keygen -t ed25519 -C "you@example.com"

# RSA — still widely supported, use 4096 bits minimum
ssh-keygen -t rsa -b 4096 -C "you@example.com"

# ECDSA — avoid if possible (NSA-designed curves, less trusted)
ssh-keygen -t ecdsa -b 521

# Generate key to specific path
ssh-keygen -t ed25519 -f ~/.ssh/deploy_key -C "deploy@prod"

# Generate key with no passphrase (automation only)
ssh-keygen -t ed25519 -f ~/.ssh/automation_key -N ""

Name origin: Ed25519 is named after the Edwards curve over the prime field 2^255 - 19. It was designed by Daniel J. Bernstein in 2011 for high speed and resistance to side-channel attacks. The "Ed" prefix denotes an Edwards curve, distinguishing it from Weierstrass-form ECDSA curves.

Key Type Comparison

Type Key Size Security Speed Recommendation
Ed25519 256-bit Excellent Fastest Default choice
RSA 4096 4096-bit Excellent Slower Legacy compatibility
RSA 2048 2048-bit Adequate Slower Minimum acceptable
ECDSA 256/384/521 Good Fast Avoid (curve concerns)

Key File Permissions

# These MUST be correct or SSH refuses to use them
chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_ed25519          # private key
chmod 644 ~/.ssh/id_ed25519.pub      # public key
chmod 600 ~/.ssh/authorized_keys
chmod 600 ~/.ssh/config

SSH Config (~/.ssh/config)

The client config file eliminates repetitive flags and lets you define per-host settings.

# Default settings for all hosts
Host *
    ServerAliveInterval 60
    ServerAliveCountMax 3
    AddKeysToAgent yes
    IdentitiesOnly yes

# Production bastion
Host bastion
    HostName bastion.prod.example.com
    User deploy
    Port 2222
    IdentityFile ~/.ssh/prod_key

# Jump through bastion to reach internal hosts
Host prod-*
    User deploy
    ProxyJump bastion
    IdentityFile ~/.ssh/prod_key

Host prod-web-01
    HostName 10.0.1.10

Host prod-db-01
    HostName 10.0.2.20

# Development server with agent forwarding
Host dev
    HostName dev.example.com
    User developer
    ForwardAgent yes
    IdentityFile ~/.ssh/dev_key

Now ssh prod-web-01 transparently jumps through the bastion. No flags needed.

Important Config Directives

Directive Purpose
HostName Real hostname/IP (overrides the alias)
User Default username
Port Non-standard port
IdentityFile Which private key to use
IdentitiesOnly Only try specified key (prevents agent spam)
ProxyJump Jump host (replaces old ProxyCommand)
ProxyCommand Custom proxy command (nc, socat, etc.)
ForwardAgent Forward SSH agent to remote host
LocalForward Automatic local port forward
RemoteForward Automatic remote port forward
DynamicForward SOCKS proxy
ServerAliveInterval Keepalive interval in seconds
ControlMaster Enable connection multiplexing
ControlPath Socket path for multiplexed connections
ControlPersist How long to keep master connection alive

sshd_config — Server Configuration

The server config lives at /etc/ssh/sshd_config. Changes require a restart or reload: systemctl restart sshd.

Hardened sshd_config

# /etc/ssh/sshd_config — production hardened

# Protocol and port
Port 2222
Protocol 2

# Authentication
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
AuthenticationMethods publickey
ChallengeResponseAuthentication no
UsePAM yes

# Key types (disable weak algorithms)
HostKeyAlgorithms ssh-ed25519,rsa-sha2-512,rsa-sha2-256
PubkeyAcceptedAlgorithms ssh-ed25519,rsa-sha2-512,rsa-sha2-256

# Access control
AllowUsers deploy admin
AllowGroups ssh-users
MaxAuthTries 3
MaxSessions 5
LoginGraceTime 30

# Forwarding
AllowTcpForwarding no
AllowAgentForwarding no
X11Forwarding no
GatewayPorts no

# Logging
LogLevel VERBOSE
SyslogFacility AUTH

# Idle timeout
ClientAliveInterval 300
ClientAliveCountMax 2

# Misc
PermitEmptyPasswords no
PermitUserEnvironment no
Banner /etc/ssh/banner.txt

After editing, always test before restarting:

sshd -t            # Test config syntax
sshd -T            # Dump effective config
sshd -T -C user=deploy,host=10.0.1.1   # Test match blocks

Agent Forwarding

SSH agent forwarding lets you authenticate from a remote host using keys stored on your local machine — without copying private keys to the remote host.

# Start the agent and add your key
eval $(ssh-agent)
ssh-add ~/.ssh/id_ed25519

# Connect with agent forwarding
ssh -A user@jumphost

# On the jumphost, you can now SSH to another host
# using your local key — it is forwarded through the agent
ssh user@internal-host

ProxyJump — The Modern Alternative to Agent Forwarding

ProxyJump is safer because it never exposes your agent socket on the jump host:

# Direct jump (no agent forwarding needed)
ssh -J jumphost user@internal-host

# Multiple jumps
ssh -J jump1,jump2 user@target

# In config
Host target
    HostName 10.0.2.50
    ProxyJump jumphost

Port Forwarding

Local Port Forwarding (-L)

Expose a remote service on a local port. Traffic flows: localhost:localport -> ssh tunnel -> remotehost:remoteport.

# Access remote PostgreSQL (5432) on localhost:15432
ssh -L 15432:localhost:5432 user@db-server

# Access a service on a different host through the SSH server
ssh -L 8080:internal-app.corp:80 user@jumphost

# Now: curl http://localhost:8080 hits internal-app.corp:80

Remote Port Forwarding (-R)

Expose a local service on the remote host. Traffic flows: remotehost:remoteport -> ssh tunnel -> localhost:localport.

# Make your local dev server (port 3000) accessible on the server at port 8080
ssh -R 8080:localhost:3000 user@server

# Anyone connecting to server:8080 reaches your local port 3000

Dynamic Port Forwarding (-D / SOCKS)

Create a SOCKS5 proxy through the SSH tunnel.

# Create SOCKS proxy on localhost:1080
ssh -D 1080 user@remote-host

# Configure browser or tools to use SOCKS5 proxy at localhost:1080
# All traffic routes through the remote host

# Use with curl
curl --socks5-hostname localhost:1080 http://internal-site.corp/

# Use with any tool via proxychains
proxychains4 nmap -sT 10.0.0.0/24

Connection Multiplexing

Reuse a single TCP/SSH connection for multiple sessions. Dramatically speeds up repeated connections.

# In ~/.ssh/config
Host *
    ControlMaster auto
    ControlPath ~/.ssh/sockets/%r@%h-%p
    ControlPersist 600

# Create the socket directory
mkdir -p ~/.ssh/sockets

With this config: - First ssh host establishes the TCP connection and SSH handshake - Subsequent ssh host, scp, rsync, and git operations reuse the existing connection - Connection stays alive for 600 seconds after the last session closes

# Check multiplexed connection status
ssh -O check host

# Close a multiplexed connection
ssh -O exit host

# Stop accepting new sessions on the multiplex
ssh -O stop host

Escape Characters

When you are stuck in an SSH session (frozen terminal, hung connection), escape characters save you.

~.      Close the connection (type Enter first)
~^Z     Suspend SSH (background it)
~#      List forwarded connections
~?      Show escape help
~C      Open command line (add forwards on-the-fly)
~~      Send a literal ~

The escape character (~) only works immediately after a newline. If your terminal is frozen, press Enter, then ~..

Remember: When SSH freezes, the recovery sequence is Enter, tilde, dot (Enter ~ .). This is the single most useful SSH fact that experienced engineers forget under pressure. The ~ must be the first character on a line — that is why Enter comes first.

authorized_keys Format and Options

# Basic entry
ssh-ed25519 AAAAC3...base64... user@laptop

# With command restriction (only allow this one command)
command="/usr/bin/backup-db",no-port-forwarding,no-X11-forwarding,no-agent-forwarding ssh-ed25519 AAAAC3...

# With source IP restriction
from="10.0.1.0/24,192.168.1.100" ssh-ed25519 AAAAC3...

# With environment variable
environment="GIT_AUTHOR_NAME=deploy" ssh-ed25519 AAAAC3...

# Force a specific command for a deploy key
command="cd /opt/app && git pull",no-pty ssh-ed25519 AAAAC3...

These restrictions are powerful for automation keys: a backup key that can only run the backup command, a deploy key restricted to git pull, a monitoring key limited to specific source IPs.

known_hosts and Host Key Verification

# List known hosts (may be hashed)
ssh-keygen -l -f ~/.ssh/known_hosts

# Remove a specific host entry (after server rebuild)
ssh-keygen -R hostname

# Remove by IP
ssh-keygen -R 10.0.1.50

# Hash all entries (privacy — hides hostnames you connect to)
ssh-keygen -H -f ~/.ssh/known_hosts

# Get a server's host key fingerprint manually
ssh-keyscan -t ed25519 hostname
ssh-keyscan hostname 2>/dev/null | ssh-keygen -l -f -

Certificate-Based Authentication

SSH certificates solve the known_hosts and authorized_keys management problem at scale. Instead of distributing individual public keys, you sign them with a Certificate Authority (CA).

# Create the CA key pair
ssh-keygen -t ed25519 -f /etc/ssh/ca_key -C "SSH CA"

# Sign a user key (valid for 52 weeks)
ssh-keygen -s /etc/ssh/ca_key -I "alice@example.com" -n alice,deploy -V +52w ~/.ssh/id_ed25519.pub
# Creates id_ed25519-cert.pub

# Sign a host key (for host key verification)
ssh-keygen -s /etc/ssh/ca_key -I "web-01.prod" -h -n web-01.prod,10.0.1.10 -V +52w /etc/ssh/ssh_host_ed25519_key.pub

# On the server — trust the CA for user auth
echo "TrustedUserCAKeys /etc/ssh/ca_user_key.pub" >> /etc/ssh/sshd_config

# On the client — trust the CA for host auth
echo "@cert-authority *.example.com ssh-ed25519 AAAAC3..." >> ~/.ssh/known_hosts

With certificates, adding a new user or server requires signing a key — not distributing files to every machine.

FIDO2 / Hardware Key Support

Modern SSH supports FIDO2 hardware keys (YubiKey 5, SoloKey) for phishing-resistant authentication.

# Generate a resident key stored on the hardware token
ssh-keygen -t ed25519-sk -O resident -C "you@example.com"

# Generate a non-resident key (requires token for signing, key on disk)
ssh-keygen -t ecdsa-sk -C "you@example.com"

# The -sk suffix means "security key"
# Requires physical touch of the hardware token for every authentication

File Transfer: scp vs sftp vs rsync

Tool Strength Use When
scp Simple, available everywhere Quick single-file copies
sftp Interactive, resume support Browsing remote filesystems
rsync -e ssh Delta transfer, preserves perms Syncing directories, large transfers 
# scp — copy a file
scp file.txt user@host:/tmp/

# scp — recursive directory copy
scp -r ./deploy/ user@host:/opt/app/

# sftp — interactive session
sftp user@host
sftp> put localfile.txt /remote/path/
sftp> get /remote/file.txt ./local/

# rsync over SSH — incremental sync
rsync -avz -e ssh ./app/ user@host:/opt/app/

# rsync with delete (mirror — removes extra files on remote)
rsync -avz --delete -e ssh ./app/ user@host:/opt/app/

# rsync with bandwidth limit
rsync -avz --bwlimit=5000 -e ssh ./backup/ user@host:/backups/

Prefer rsync for anything larger than a single small file. It only transfers changed bytes and can resume interrupted transfers.

Wiki Navigation

Prerequisites

Related Content