SSH Is More Than You Think

• lesson
• ssh-protocol
• key-exchange
• authentication
• tunneling
• agent-forwarding
• proxyjump
• certificates ---# SSH Is More Than You Think

Topics: SSH protocol, key exchange, authentication, tunneling, agent forwarding, ProxyJump, certificates Level: L1–L2 (Foundations → Operations) Time: 60–75 minutes Prerequisites: None (you've used SSH before, this explains what it's doing)

---

The Mission

You type ssh server fifty times a day. The connection is encrypted, the authentication is automatic, and it just works. But SSH is doing far more than giving you a shell. It's a full-featured, encrypted, multiplexed protocol that can tunnel databases, proxy web traffic, and build VPNs — and most people use maybe 10% of it.

This lesson traces what happens when you type ssh server, then explores the features that make SSH one of the most powerful (and misunderstood) tools in DevOps.

---

What Happens When You Type ssh server

Step 1: TCP connection

SSH connects to port 22 (TCP). Both sides exchange version strings:

Client: SSH-2.0-OpenSSH_9.3
Server: SSH-2.0-OpenSSH_9.0

Name Origin: SSH (Secure Shell) was created in 1995 by Tatu Ylönen, a researcher at Helsinki University of Technology, after a password-sniffing attack captured thousands of plaintext Telnet and FTP passwords on the university network. Ylönen wrote the first version in weeks, driven by urgency. He chose port 22 because it was unassigned and sat between FTP (21) and Telnet (23) — the exact protocols SSH was designed to replace. He emailed IANA and got the port assigned the same day. The internet was smaller and more informal in 1995.

Step 2: Key exchange (no passwords involved yet)

Before any authentication, the client and server negotiate encryption using Diffie-Hellman (or ECDH) key exchange. This creates a shared secret without either side transmitting it:

Client generates random private key → derives public key → sends to server
Server generates random private key → derives public key → sends to client
Both compute the same shared secret from the other's public key + their private key

This shared secret becomes the session encryption key. Even if someone recorded the entire exchange, they can't derive the key without one of the private values (which were never transmitted). This is forward secrecy — compromising the server's long-term key later doesn't decrypt past sessions.

Step 3: Server authentication (host key verification)

The server proves its identity with its host key. On first connection:

The authenticity of host 'server (10.0.1.50)' can't be established.
ED25519 key fingerprint is SHA256:xR3fE5...
Are you sure you want to continue connecting (yes/no)?

When you type "yes," the fingerprint is saved to ~/.ssh/known_hosts. On subsequent connections, SSH compares the server's key to the stored fingerprint. If it changes:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

This protects against MITM attacks — if someone intercepts your connection, they can't present the original server's host key.

Mental Model: SSH's security model is Trust On First Use (TOFU). The first connection is vulnerable (you can't verify the fingerprint without an out-of-band channel). But every subsequent connection is protected. This is a pragmatic compromise — true verification would require a PKI infrastructure that SSH was designed to avoid.

Step 4: User authentication

Now the client proves who you are. Methods tried in order (configurable):

1. Public key — client proves it holds the private key matching a public key in the server's authorized_keys
2. Password — fallback (should be disabled in production)
3. Keyboard-interactive — for 2FA (PAM-based)
4. Certificate — signed by a trusted CA (large-scale deployments)

# See what authentication methods the server supports
ssh -v server 2>&1 | grep "Authentications that can continue"
# → publickey,keyboard-interactive

# See which key was accepted
ssh -v server 2>&1 | grep "Accepted"
# → Accepted publickey for deploy

---

Key Types: Which One to Use

# Generate an Ed25519 key (recommended)
ssh-keygen -t ed25519 -C "deploy@company"

# Check key type and size
ssh-keygen -l -f ~/.ssh/id_ed25519
# → 256 SHA256:xR3fE5... deploy@company (ED25519)

Type	Key size	Speed	Security	Recommendation
Ed25519	256-bit	Fastest	Excellent	Use this
RSA	2048-4096 bit	Slow	Good (at 4096)	Legacy, still common
ECDSA	256/384/521 bit	Fast	Good	Works, but Ed25519 is better
DSA	1024-bit	—	Broken	Never use (deprecated in OpenSSH 7.0)

Trivia: Ed25519 was designed by Daniel J. Bernstein (the same person who invented SYN cookies). It was chosen specifically to resist timing side-channel attacks — the implementation runs in constant time regardless of the key material. The public key is 68 characters vs 372+ for RSA. It's faster, smaller, and more secure.

---

The ~/.ssh/config File — SSH's Secret Weapon

Most people type full SSH commands. The config file eliminates that:

# ~/.ssh/config

# Jump through a bastion
Host prod-*
    ProxyJump bastion.example.com
    User deploy
    IdentityFile ~/.ssh/deploy_ed25519

# Specific server with non-standard port
Host db-primary
    HostName 10.0.2.50
    Port 2222
    User postgres

# Multiplexing — share TCP connection across sessions
Host *
    ControlMaster auto
    ControlPath ~/.ssh/control-%h-%p-%r
    ControlPersist 600
    ServerAliveInterval 60
    ServerAliveCountMax 3

Now ssh prod-web-01 automatically jumps through the bastion, uses the right key, and connects as the right user. No flags to remember.

Trivia: SSH multiplexing (ControlMaster) has been available since OpenSSH 3.9 (2004). Multiple sessions share a single TCP connection. The second ssh connection to the same host skips the handshake entirely — it's nearly instant. Three lines in your config dramatically speed up workflows with frequent SSH connections.

---

Tunneling — SSH as a VPN

Local port forwarding (bring a remote service to your machine)

# Forward local port 5432 to the database through a bastion
ssh -L 5432:prod-db:5432 bastion -N

# Now connect to the database as if it were local
psql -h localhost -p 5432 -U myuser mydb

Your laptop:5432 ──[encrypted]──→ bastion ──[internal network]──→ prod-db:5432

Remote port forwarding (expose your machine to a remote network)

# Make your local dev server accessible from a remote server
ssh -R 8080:localhost:3000 remote-server -N

# On remote-server, curl localhost:8080 hits your laptop's port 3000

Dynamic port forwarding (SOCKS proxy)

# Create a SOCKS5 proxy through the SSH connection
ssh -D 1080 bastion -N

# Configure browser to use localhost:1080 as SOCKS proxy
# All browser traffic now goes through the bastion

ProxyJump — the modern way through bastions

# OLD — agent forwarding (dangerous)
ssh -A bastion
ssh internal-server
# Problem: root on bastion can use your agent to authenticate as you

# NEW — ProxyJump (safe)
ssh -J bastion internal-server
# Your key never leaves your laptop. TCP is tunneled through bastion.

Gotcha: ssh -A (agent forwarding) forwards your SSH agent to the remote host. If the remote host is compromised, the attacker can use your agent socket to authenticate as you to any server your key reaches — while your session is active. You won't see anything unusual. Always use ProxyJump (-J) instead of agent forwarding — your key stays local, only the TCP stream is tunneled.

---

SSH Certificates — The Enterprise Alternative

Raw SSH keys don't scale. With 500 servers and 50 engineers, you need 25,000 authorized_keys entries. SSH certificates solve this:

# 1. Create a CA key pair (once)
ssh-keygen -t ed25519 -f /etc/ssh/ca_user_key -C "SSH User CA"

# 2. Sign a user's key (valid for 8 hours)
ssh-keygen -s /etc/ssh/ca_user_key \
    -I "alice@company" \
    -n deploy,alice \
    -V +8h \
    alice_id_ed25519.pub

# 3. Configure server to trust the CA
echo "TrustedUserCAKeys /etc/ssh/ca_user_key.pub" >> /etc/ssh/sshd_config
systemctl reload sshd

Now Alice's signed key works on every server that trusts the CA — no authorized_keys management, keys expire automatically, and revocation is centralized.

Trivia: SSH certificates have been available since OpenSSH 5.4 (2010). Facebook, Netflix, and Google use them at scale. Most organizations don't know the feature exists. HashiCorp Vault can act as an SSH CA, signing keys on demand with short TTLs.

---

Flashcard Check

Q1: What is forward secrecy in SSH?

Ephemeral DH/ECDH keys mean each session has unique encryption keys. Compromising the server's long-term host key later can't decrypt past sessions.

Q2: Why is ssh -A (agent forwarding) dangerous?

Root on the remote host can use your agent socket to authenticate as you to any server your key reaches. Use ssh -J (ProxyJump) instead — key stays local.

Q3: Ed25519 vs RSA — which should you use?

Ed25519. It's faster, smaller (68 chars vs 372+), more secure, and resistant to timing side-channels. Only use RSA for compatibility with very old systems.

Q4: ssh -L 5432:db:5432 bastion — what does this do?

Creates a local port forward. Connections to localhost:5432 are encrypted through the SSH tunnel to the bastion, which connects to db:5432 on the internal network.

Q5: The "REMOTE HOST IDENTIFICATION HAS CHANGED" warning — should you ignore it?

Never blindly. It means the server's host key changed. Could be: server rebuilt (benign), or MITM attack (critical). Verify out-of-band before accepting the new key.

Q6: How does SSH multiplexing work?

ControlMaster shares a single TCP connection across multiple SSH sessions to the same host. Subsequent connections skip the handshake and are nearly instant.

---

Exercises

Exercise 1: Inspect your SSH setup (hands-on)

# What key types do you have?
for f in ~/.ssh/id_*; do ssh-keygen -l -f "$f" 2>/dev/null; done

# What's in your known_hosts?
wc -l ~/.ssh/known_hosts
ssh-keygen -l -f ~/.ssh/known_hosts | head -5

# Check server's host keys
ssh-keyscan -t ed25519 localhost 2>/dev/null | ssh-keygen -l -f -

Exercise 2: Set up a tunnel (hands-on)

# Forward a local port to a remote service
ssh -L 8080:localhost:80 someserver -N &

# Test it
curl http://localhost:8080
# You're now hitting someserver's port 80 through an encrypted tunnel

# Clean up
kill %1

Exercise 3: Optimize your SSH config (hands-on)

Add multiplexing and keepalive to your SSH config:

Host *
    ControlMaster auto
    ControlPath ~/.ssh/control-%h-%p-%r
    ControlPersist 600
    ServerAliveInterval 60
    ServerAliveCountMax 3

Time ssh server hostname before and after. The second connection should be near-instant.

---

Cheat Sheet

Task	Command
Generate Ed25519 key	ssh-keygen -t ed25519 -C "comment"
Copy key to server	ssh-copy-id user@server
Local port forward	ssh -L localport:remotehost:remoteport bastion -N
Remote port forward	ssh -R remoteport:localhost:localport server -N
SOCKS proxy	ssh -D 1080 server -N
Jump through bastion	ssh -J bastion internal-server
Verbose debugging	ssh -vvv server
Check key fingerprint	ssh-keygen -l -f keyfile
Scan host keys	ssh-keyscan -t ed25519 server

Permission Requirements

~/.ssh/              → 700
~/.ssh/id_ed25519    → 600 (private key)
~/.ssh/id_ed25519.pub → 644 (public key)
~/.ssh/authorized_keys → 600
~/.ssh/config        → 644
~/.ssh/known_hosts   → 644

SSH silently refuses keys with wrong permissions.

---

Takeaways

1. SSH does key exchange before authentication. The connection is encrypted before any credentials are transmitted. Forward secrecy means past sessions can't be decrypted.

2. Use Ed25519 for new keys. Faster, smaller, more secure than RSA. Only use RSA for compatibility with old systems.

3. ProxyJump replaces agent forwarding. -J bastion keeps your key local. -A exposes your agent to the remote host — dangerous on untrusted machines.

4. SSH config eliminates repetitive flags. Multiplexing, jump hosts, key selection, user mapping — put it in ~/.ssh/config and forget the flags.

5. SSH certificates scale. For 50+ servers, stop managing authorized_keys files. Use a CA (OpenSSH native or Vault) and sign short-lived certificates.

---

Related Lessons

• What Happens When You Click a Link — TLS, SSH's cousin protocol
• The Container Escape — when SSH tunnels become security risks
• PXE Boot — where SSH access begins after provisioning

Pages that link here

• Cross-Domain Lessons
• Linux Hardening Closing The Doors
• Linux The Complete Guide