Skip to content

TLS Handshake Deep Dive

Scope

This document focuses on modern TLS, especially TLS 1.3 behavior and the operational concepts you actually need:

  • why TLS exists
  • certificate and trust chain basics
  • handshake phases
  • key exchange intuition
  • authentication
  • session resumption
  • common failure cases
  • deployment realities

Reference anchors: - https://www.rfc-editor.org/rfc/rfc8446.html - https://www.rfc-editor.org/rfc/rfc9325.html - https://www.rfc-editor.org/rfc/rfc8996.html

Big Picture

TLS exists to give you:

  • confidentiality
  • integrity
  • peer authentication (usually server authentication, optionally client auth)
  • resistance to active tampering

It sits above TCP and below application protocols like HTTP, SMTP, IMAP, etc.

Think:

app bytes
 -> TLS records/handshake
 -> TCP
 -> IP

What the Handshake Needs To Achieve

Before application data can flow securely, both sides need to establish:

  1. protocol version / capabilities
  2. cipher suite agreement
  3. key exchange parameters
  4. server authentication
  5. shared secrets for traffic protection
  6. optionally client authentication
  7. transcript integrity

That is why the handshake is not just "send cert and go."

TLS 1.3 Handshake Intuition

Very roughly:

  1. ClientHello
  2. supported versions
  3. supported cipher suites
  4. extensions
  5. key share(s)

  6. SNI, ALPN, etc.

  7. ServerHello

  8. selects parameters

  9. returns its key share

  10. EncryptedExtensions

  11. additional negotiated parameters

  12. Certificate

  13. server proves identity chain

  14. CertificateVerify

  15. cryptographic proof tied to server private key

  16. Finished

  17. transcript-based integrity proof

  18. client sends its Finished

  19. application data can flow

TLS 1.3 compresses and cleans up a lot of older TLS complexity.

Certificates and Trust

A certificate says, in effect: "This public key is bound to this identity, signed by an issuer."

The client does not trust a random presented cert because vibes. It validates: - chain to trusted root/intermediate path - name matching - time validity - signature correctness - policy constraints as applicable

If that fails, the secure illusion collapses fast.

Key Exchange Intuition

Modern TLS uses ephemeral key exchange so that session secrets are derived fresh.

That matters because: - compromise of long-term certificate private key later should not expose past session traffic in the same direct way - every session gets fresh cryptographic material

This is part of why modern TLS is much better than old "static everything" thinking.

Authentication vs Encryption

These are different jobs.

Authentication

"Am I really talking to the expected server?"

Encryption

"Can others read the traffic?"

Integrity

"Can others alter it undetected?"

A system that encrypts without authenticating correctly is still in trouble.

ALPN and SNI

SNI

Lets the client indicate which hostname it wants on a shared IP.

ALPN

Lets client/server agree on application protocol, such as HTTP/2.

These extensions matter a lot in modern hosting reality.

Session Resumption

TLS can avoid paying full handshake cost every single time.

Benefits: - lower latency - lower CPU - better scale

But resumption has to be designed carefully; security and key lifetime still matter.

Why TLS 1.0 and 1.1 Are Dead Weight

They are deprecated for good reasons: - weaker design era - outdated cipher realities - modern deployment recommendations moved on

For normal modern infrastructure discussion, TLS 1.2 and especially TLS 1.3 are the relevant baseline.

Common Failure Modes

Certificate name mismatch

Server cert does not match requested hostname.

Expired certificate

The classic self-own.

Missing intermediate chain

The server presents an incomplete chain.

Protocol/cipher mismatch

Old client vs new server or vice versa.

Time skew

Certificate validation can fail because clocks are wrong.

Broken trust store assumptions

Particularly in containers, custom images, private PKI.

"TLS error" blamed on networking

Sometimes the TCP path is fine and crypto policy is what is actually failing.

Operational Commands

openssl s_client -connect host:443 -servername host
openssl x509 -in cert.pem -text -noout
curl -Iv https://host

Look for: - negotiated version - cipher - chain - SAN names - OCSP/stapling status if relevant - ALPN result

Interview-Level Things to Explain

You should be able to explain:

  • what TLS protects
  • what happens in ClientHello/ServerHello at a high level
  • why certificates are not the same thing as encryption keys for the session
  • why name validation matters
  • why TLS 1.3 is cleaner than older handshakes
  • common cert/chain/protocol failure patterns

Fast Mental Model

TLS is a cryptographic negotiation layered on top of TCP that authenticates peers, derives fresh session keys, and then protects application traffic for confidentiality and integrity.

Wiki Navigation

Prerequisites

Related Content