Skip to content

Portal | Level: L0: Entry | Topics: HTTP Protocol, TLS & PKI | Domain: Networking

HTTP Protocol - Primer

Why This Matters

HTTP is the protocol you debug most often in operations work. Every API call, webhook, health check, load balancer probe, CDN interaction, and browser request speaks HTTP. When a service returns 502, when CORS errors block a dashboard, when caching goes wrong and users see stale data, or when a TLS certificate expires at 3 AM — you need to understand HTTP deeply enough to diagnose the problem from headers and status codes alone.

HTTP is also the protocol you configure most often. Load balancer rules, reverse proxy configs, API gateway policies, CDN cache rules, WAF rules, and service mesh routing all operate at the HTTP layer. A misunderstanding of how headers, methods, or status codes work translates directly into misconfigured infrastructure.

Understanding HTTP turns vague symptoms ("the app is broken") into precise diagnoses ("the backend is returning 503 because the upstream health check is failing on the /ready endpoint with a timeout").

Core Concepts

1. URL Anatomy

https://api.example.com:8443/v2/deployments?env=prod&limit=10#results
|____| |_________________||___||____________||________________||______|
scheme       host          port    path         query string   fragment
Component Purpose
Scheme Protocol (http, https)
Host Server name (used in DNS lookup and Host header)
Port TCP port (443 default for HTTPS, 80 for HTTP)
Path Resource location on the server
Query string Key-value parameters after ?
Fragment Client-side only — NOT sent to the server

The fragment (#results) is never included in the HTTP request. This is a common source of confusion when debugging client-side routing.

2. Request Structure

POST /v2/deployments HTTP/1.1
Host: api.example.com
Content-Type: application/json
Authorization: Bearer eyJhbGciOi...
Content-Length: 42
User-Agent: curl/8.4.0

{"image": "myapp:v2.1.0", "replicas": 3}

Components: method, path, protocol version, headers, optional body.

3. HTTP Methods

Method Purpose Body Idempotent Safe
GET Retrieve resource No Yes Yes
HEAD Like GET, headers only No Yes Yes
POST Create or submit Yes No No
PUT Replace resource Yes Yes No
PATCH Partial update Yes No No
DELETE Remove resource Optional Yes No
OPTIONS Describe communication options No Yes Yes

Idempotent means repeating the request produces the same result. This matters for retry logic in load balancers and service meshes. Non-idempotent methods (POST) should not be automatically retried.

4. Response Structure and Status Codes

HTTP/1.1 201 Created
Content-Type: application/json
Location: /v2/deployments/abc123
X-Request-Id: req-7f3a2b

{"id": "abc123", "status": "deploying"}
Range Category Common Codes
1xx Informational 101 Switching Protocols
2xx Success 200 OK, 201 Created, 204 No Content
3xx Redirect 301 Permanent, 302 Found, 307 Temporary, 308 Permanent
4xx Client error 400 Bad Request, 401 Unauthorized, 403 Forbidden
404 Not Found, 405 Method Not Allowed, 429 Too Many Requests
5xx Server error 500 Internal Error, 502 Bad Gateway, 503 Unavailable
504 Gateway Timeout

Status codes you will debug constantly in ops: - 502 — the proxy/LB could not reach the upstream. Check if the backend is running and listening on the expected port. - 503 — service unavailable. Often means the backend exists but is not ready (failed health check, overloaded, draining). - 504 — upstream timed out. Check backend response times and proxy timeout settings. - 429 — rate limited. Check rate limit headers for retry timing.

5. Headers That Matter for Ops

Host              — which virtual host to route to (essential for
                    reverse proxies serving multiple domains)
Content-Type      — MIME type of the body (application/json, etc.)
Authorization     — credentials (Bearer token, Basic, API key)
Cache-Control     — caching directives (no-store, max-age=3600)
ETag / If-None-Match  — conditional requests for cache validation
X-Request-Id      — trace ID for correlating logs across services
X-Forwarded-For   — original client IP behind proxies/LBs
X-Forwarded-Proto — original protocol (http/https) behind TLS
                    termination
Retry-After       — server hint for when to retry (with 429/503)
Strict-Transport-Security — enforce HTTPS (HSTS)

6. Caching

Cache-Control header directives:

  no-store          — do not cache at all
  no-cache          — cache but revalidate every time
  private           — only browser cache, not CDN/proxy
  public            — any cache can store this
  max-age=3600      — fresh for 3600 seconds
  s-maxage=3600     — max-age override for shared caches (CDN)
  must-revalidate   — stale cache must revalidate, not serve stale
  immutable         — content will never change (use with hashed URLs)

Cache validation flow:

Client has cached response with ETag: "abc123"
         |
         v
GET /resource HTTP/1.1
If-None-Match: "abc123"
         |
         v
Server checks: is current ETag still "abc123"?
  Yes → 304 Not Modified (no body, use cache)
  No  → 200 OK (full new response with new ETag)

7. Cookies

Cookies are name-value pairs the server sets via Set-Cookie and the browser sends back on subsequent requests via Cookie.

Set-Cookie: session=abc123; Path=/; HttpOnly; Secure; SameSite=Lax;
            Max-Age=3600
Attribute Purpose
HttpOnly Not accessible via JavaScript (XSS protection)
Secure Only sent over HTTPS
SameSite Controls cross-site sending (Lax, Strict, None)
Domain Which domains receive the cookie
Path Which paths receive the cookie
Max-Age Lifetime in seconds (0 = delete)

8. TLS and HTTPS

HTTPS is HTTP over TLS. The TLS handshake happens before any HTTP traffic.

# Check certificate details
openssl s_client -connect api.example.com:443 \
    -servername api.example.com < /dev/null 2>/dev/null \
    | openssl x509 -noout -dates -subject -issuer

# Check certificate expiration
echo | openssl s_client -connect api.example.com:443 \
    -servername api.example.com 2>/dev/null \
    | openssl x509 -noout -enddate

# Verbose connection debugging
curl -v https://api.example.com/health 2>&1 | grep -E '^\*'

Common TLS issues in ops: - Certificate expired (check with monitoring, automate renewal) - Certificate hostname mismatch (SNI vs cert SAN) - Intermediate certificate missing (chain incomplete) - Mixed content (HTTPS page loading HTTP resources)

9. CORS (Cross-Origin Resource Sharing)

Browsers block cross-origin requests by default. CORS headers allow controlled exceptions.

Origin: https://dashboard.example.com
  → browser sends this header on cross-origin requests

Access-Control-Allow-Origin: https://dashboard.example.com
  → server allows this specific origin

Preflight (OPTIONS request sent automatically for "complex" requests):
  1. Browser sends OPTIONS with Origin and
     Access-Control-Request-Method
  2. Server responds with Access-Control-Allow-* headers
  3. If allowed, browser sends the actual request

CORS errors show up in browser console, not in server logs. If your internal dashboard cannot reach an API, check CORS headers first.

10. Debugging HTTP

# Basic response headers
curl -I https://api.example.com/health

# Full request/response with TLS details
curl -v https://api.example.com/health

# Send JSON POST
curl -X POST https://api.example.com/v2/deployments \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $TOKEN" \
    -d '{"image": "myapp:v2.1.0"}'

# Follow redirects and show each hop
curl -L -v https://example.com 2>&1 | grep -E '^[<>*]'

# Time breakdown (DNS, connect, TLS, TTFB, total)
curl -o /dev/null -s -w \
    "dns: %{time_namelookup}s\nconnect: %{time_connect}s\n\
tls: %{time_appconnect}s\nttfb: %{time_starttransfer}s\n\
total: %{time_total}s\n" \
    https://api.example.com/health

What Experienced People Know

  • Always log the X-Request-Id (or generate one). Without a trace ID, correlating a client error to a server log entry across multiple services is nearly impossible.
  • The difference between 301 and 308 (and 302 vs 307) is whether the client is allowed to change the method. 301/302 may change POST to GET. 307/308 preserve the method. This matters for API redirects.
  • Cache-Control: no-cache does NOT mean "do not cache." It means "cache but revalidate every time." Use no-store to prevent caching.
  • CORS is a browser-only enforcement mechanism. Server-to-server requests are never subject to CORS. If curl works but the browser does not, it is almost certainly a CORS issue.
  • X-Forwarded-For can be spoofed by clients. Trust it only from known proxies. Many rate limiters and geo-IP systems get this wrong.
  • HTTP/2 multiplexes multiple requests over a single TCP connection. This means one slow response can cause head-of-line blocking for all requests on that connection.
  • A 204 response must not have a body. Some poorly implemented APIs return 204 with a body, which breaks strict HTTP parsers in proxies.
  • When debugging intermittent 502/504 errors, check connection keepalive timeouts. If the backend closes idle connections before the proxy expects, the proxy will forward requests to a closed socket.
  • Certificate monitoring is non-negotiable. Automate renewal with Let's Encrypt/ACME, and alert at 30, 14, and 7 days before expiry.
  • HTTP response compression (Content-Encoding: gzip) reduces bandwidth but increases CPU. Know when it helps (JSON APIs, text) and when it does not (already compressed content like images).

Wiki Navigation

Related Content