The Nginx Config That Broke Everything

• lesson
• nginx
• reverse-proxy
• location-blocks
• upstream
• common-misconfigurations
• debugging ---# The Nginx Config That Broke Everything

Topics: Nginx, reverse proxy, location blocks, upstream, common misconfigurations, debugging Level: L1–L2 (Foundations → Operations) Time: 45–60 minutes Prerequisites: None

---

The Mission

You edited /etc/nginx/sites-available/myapp, ran nginx -t (syntax OK!), reloaded, and now the app returns 502 Bad Gateway. The app is running. The port is open. But Nginx can't reach it.

Nginx is the most common reverse proxy in DevOps, and its configuration has subtle traps that nginx -t doesn't catch — it validates syntax, not logic.

---

How Nginx Processes a Request

Client request arrives
  → server {} block selected (by Host header + listen port)
    → location {} block selected (by URI path matching)
      → proxy_pass sends to backend
        → backend responds
          → Nginx returns response to client

Every request walks this chain. Errors at any step produce different symptoms.

---

Trap 1: proxy_pass Trailing Slash

# These are DIFFERENT:

location /api/ {
    proxy_pass http://backend:8080;     # No trailing slash
}
# Request: /api/users → Backend sees: /api/users (path preserved)

location /api/ {
    proxy_pass http://backend:8080/;    # WITH trailing slash
}
# Request: /api/users → Backend sees: /users (path stripped!)

The trailing slash on proxy_pass strips the matching location prefix. This is the #1 source of "it works directly but not through Nginx" bugs.

Mental Model: Without trailing slash: Nginx passes the URL as-is. With trailing slash: Nginx removes the location prefix and appends the rest to proxy_pass URL.

---

Trap 2: DNS Resolution at Load Time

# BAD — hostname resolved ONCE at config load, cached forever
upstream backend {
    server app.example.com:8080;
}

# If the IP changes (container restarts, DNS failover), Nginx keeps the OLD IP
# until you reload the config

# FIX — use a variable to force runtime resolution
resolver 10.0.0.2 valid=30s;   # Use your DNS server

location /api/ {
    set $backend "http://app.example.com:8080";
    proxy_pass $backend;
}

Gotcha: This hits every Nginx installation in dynamic environments (Kubernetes, Docker, cloud). The backend container restarts with a new IP, Nginx keeps sending to the old one → 502 Bad Gateway. nginx -s reload fixes it temporarily but the real fix is runtime DNS resolution.

---

Trap 3: Location Block Matching Order

Nginx doesn't evaluate location blocks top-to-bottom. The matching algorithm is:

1. Exact match (=): location = /api/health { } — highest priority
2. Preferential prefix (^~): location ^~ /static/ { } — stops regex search
3. Regex (~ or ~*): location ~* \.(jpg|png)$ { } — first match wins
4. Prefix (no modifier): location /api/ { } — longest match, but regex can override

# This catches requests you didn't expect:
location / {
    proxy_pass http://frontend:3000;
}

location /api {
    proxy_pass http://backend:8080;
}

# Request: /api-docs → goes to /api block (longest prefix match)
# But you probably wanted /api-docs to go to frontend
# Fix: use /api/ (with trailing slash) to require the slash

---

Trap 4: 502 Bad Gateway — The Usual Suspects

502 means Nginx connected to the backend but got an invalid response (or the connection was refused/reset).

# Diagnostic checklist:
# 1. Is the backend actually running?
ss -tlnp | grep 8080

# 2. Can Nginx reach the backend?
curl -v http://backend:8080/health

# 3. Check Nginx error log (almost always has the answer)
tail -50 /var/log/nginx/error.log
# → "connect() failed (111: Connection refused)"  ← backend not listening
# → "upstream prematurely closed connection"       ← backend crashed during request
# → "no live upstreams"                            ← all backends failed health checks

# 4. Is it a timeout?
# Default proxy_read_timeout is 60s — if backend takes longer, Nginx returns 504

---

Trap 5: Missing Headers

# BAD — backend sees Nginx's IP for all requests
location / {
    proxy_pass http://backend:8080;
}

# GOOD — pass through the real client information
location / {
    proxy_pass http://backend:8080;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

Without these headers: - Backend logs show Nginx's IP (127.0.0.1) for every request — rate limiting breaks - Backend doesn't know if the request was HTTPS — redirect loops - Backend sees wrong Host header — virtual hosting breaks

---

Trap 6: Reload vs Restart

# SAFE — reload: new config, zero downtime, existing connections finish
nginx -s reload

# DANGEROUS — restart: stops then starts, drops all active connections
systemctl restart nginx

# ALWAYS test config before reload
nginx -t && nginx -s reload

# NEVER reload without testing — syntax error + reload = Nginx refuses to start
# and now EVERYTHING is down

Gotcha: nginx -t validates syntax but not logic. A config that passes -t can still 502 (backend unreachable), 404 (wrong root), or loop (circular proxy_pass). Always test the actual behavior after reload.

---

A Complete, Correct Nginx Config

# /etc/nginx/sites-available/myapp
upstream app_backend {
    server 127.0.0.1:8000;
    keepalive 32;
}

server {
    listen 80;
    server_name app.example.com;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name app.example.com;

    ssl_certificate     /etc/letsencrypt/live/app.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/app.example.com/privkey.pem;

    # Security headers
    add_header Strict-Transport-Security "max-age=63072000" always;
    add_header X-Content-Type-Options nosniff always;
    add_header X-Frame-Options DENY always;

    # Proxy settings
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    # Timeouts
    proxy_connect_timeout 5s;
    proxy_read_timeout 30s;
    proxy_send_timeout 30s;

    # Application
    location / {
        proxy_pass http://app_backend;
    }

    # Health check (no access log noise)
    location = /health {
        proxy_pass http://app_backend;
        access_log off;
    }

    # Static files (if applicable)
    location /static/ {
        alias /opt/myapp/static/;
        expires 30d;
        add_header Cache-Control "public, immutable";
    }
}

---

Flashcard Check

Q1: proxy_pass http://backend/; (trailing slash) — what does it do?

Strips the location prefix from the URI before proxying. /api/users with location /api/ becomes /users at the backend.

Q2: Nginx returns 502. First thing to check?

/var/log/nginx/error.log. It almost always tells you exactly what went wrong: connection refused, upstream timeout, premature close.

Q3: Backend container restarts with new IP. Nginx still sends to old IP. Why?

Nginx resolves hostnames at config load and caches the result. Use a resolver directive + variable in proxy_pass for runtime resolution.

Q4: nginx -t says "syntax is ok." Is the config correct?

Syntax is valid. Logic might not be. -t can't check if backends are reachable, locations match correctly, or headers are set right. Test actual behavior.

---

Cheat Sheet

Task	Command
Test config	nginx -t
Reload (safe)	nginx -t && nginx -s reload
Error log	tail -f /var/log/nginx/error.log
Access log	tail -f /var/log/nginx/access.log
Show active config	nginx -T (dumps full merged config)
Check listening ports	ss -tlnp \| grep nginx

Proxy Headers (Always Include)

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

---

Takeaways

1. Trailing slash on proxy_pass changes the URL. With slash: strips location prefix. Without: passes URL as-is. This is the #1 Nginx gotcha.

2. DNS is resolved at load time. Dynamic backends (containers, cloud) need runtime DNS resolution via resolver + variable.

3. Always check error.log. The error log tells you exactly what failed. Don't guess from the status code.

4. nginx -t validates syntax, not logic. Test actual behavior after every change.

5. Reload, don't restart. nginx -s reload is zero-downtime. systemctl restart drops all connections.

---

Related Lessons

• Deploy a Web App From Nothing — Nginx as the reverse proxy layer
• Connection Refused — when Nginx can't reach the backend
• What Happens When Your Certificate Expires — TLS in Nginx