iptables: Following a Packet

lesson
iptables
netfilter
chains
nat
dnat/snat
connection-tracking
nftables
l2 ---# iptables: Following a Packet Through the Chains

Topics: iptables, netfilter, chains, NAT, DNAT/SNAT, connection tracking, nftables Level: L2 (Operations) Time: 60–75 minutes Prerequisites: Basic networking (IP addresses, ports)

The Mission¶

You add an iptables rule. It doesn't work. You add another. Now SSH is broken. You panic, run iptables -F (flush everything), and now the Docker containers can't reach the internet.

iptables is the most powerful and most confusing networking tool on Linux. It controls which packets are accepted, rejected, dropped, or rewritten — but the order of evaluation is non-obvious, and one wrong rule can lock you out of a remote server.

This lesson traces a packet through every iptables chain, explaining what happens at each step.

The Five Chains¶

Every packet entering, leaving, or passing through a Linux system goes through netfilter hooks — points in the kernel's packet processing where iptables rules are evaluated:

Incoming packet from network
        │
        ▼
  ┌──────────────┐
  │  PREROUTING  │ ← DNAT happens here (change destination)
  └──────┬───────┘
         │
    Routing decision: is the packet for this machine?
         │
    ┌────┴────┐
    │         │
    ▼         ▼
┌───────┐ ┌─────────┐
│ INPUT │ │ FORWARD │ ← Packet transiting through (routers, Docker, K8s)
└───┬───┘ └────┬────┘
    │          │
    ▼          ▼
  Local     ┌──────────────┐
  process   │ POSTROUTING  │ ← SNAT/MASQUERADE happens here
            └──────────────┘
    │              │
    ▼              ▼
┌────────┐      Out to
│ OUTPUT │      network
└───┬────┘
    │
    ▼
┌──────────────┐
│ POSTROUTING  │ ← Also for locally-generated packets
└──────────────┘
    │
    ▼
  Out to network

What each chain does¶

Chain	When it runs	What it's for
PREROUTING	Before routing decision	DNAT (change destination IP/port)
INPUT	Packets destined for this host	Firewall: allow/deny incoming traffic
FORWARD	Packets passing through to another host	Router/bridge firewall, Docker, K8s
OUTPUT	Packets generated by this host	Firewall: control outgoing traffic
POSTROUTING	After routing, before sending	SNAT/MASQUERADE (change source IP)

Reading iptables Rules¶

# Show all rules with line numbers and packet counts
iptables -L -n -v --line-numbers

# Show a specific chain
iptables -L INPUT -n -v --line-numbers
# → Chain INPUT (policy ACCEPT)
# → num  pkts bytes target    prot opt in  out  source      destination
# → 1    5234 420K ACCEPT     all  --  lo  *    0.0.0.0/0   0.0.0.0/0
# → 2    23K  1.8M ACCEPT     all  --  *   *    0.0.0.0/0   0.0.0.0/0  state RELATED,ESTABLISHED
# → 3    342  20K  ACCEPT     tcp  --  *   *    0.0.0.0/0   0.0.0.0/0  tcp dpt:22
# → 4    0    0    DROP       all  --  *   *    0.0.0.0/0   0.0.0.0/0

Rules evaluate top to bottom. First match wins. Rule 4 drops everything, but it only reaches packets that didn't match rules 1-3. SSH (port 22) is allowed by rule 3 before the DROP.

Gotcha: If you add a DROP rule before your ACCEPT rules, you block yourself:

# This locks you out of SSH immediately on a remote server:
iptables -I INPUT 1 -j DROP    # Insert at position 1 (before everything)
# You can't SSH in to fix it. Console access or reboot required.

How Docker and Kubernetes Use iptables¶

Docker creates iptables rules for container networking:

# See Docker's NAT rules
iptables -t nat -L -n -v
# → Chain DOCKER (2 references)
# → DNAT  tcp  --  0.0.0.0/0  0.0.0.0/0  tcp dpt:8080 to:172.17.0.2:8080
#   ↑ Incoming traffic to port 8080 is rewritten to container IP

Kubernetes kube-proxy creates even more rules:

# Count iptables rules (can be thousands in K8s)
iptables -L -n | wc -l
# → 5000+ (for a cluster with hundreds of Services)

Trivia: iptables has gone through four generations on Linux: ipfwadm (1994, Linux 1.x) → ipchains (1998, Linux 2.2) → iptables (2001, Linux 2.4) → nftables (2014, Linux 3.13). Each was a complete rewrite. Despite nftables being the official successor, iptables remains more widely used because of the massive ecosystem (Docker, Kubernetes, fail2ban, UFW all generate iptables rules).

Practical: Building a Firewall¶

# Start with a clean slate (CAREFUL on remote servers!)
# Save current rules first:
iptables-save > /tmp/iptables-backup.rules

# Allow established connections (critical — keeps your SSH alive)
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow loopback
iptables -A INPUT -i lo -j ACCEPT

# Allow SSH
iptables -A INPUT -p tcp --dport 22 -j ACCEPT

# Allow HTTP/HTTPS
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT

# Drop everything else
iptables -A INPUT -j DROP

# Verify
iptables -L INPUT -n --line-numbers

War Story: A hospital network engineer added a deny rule for port 22 outbound, intending to restrict which servers staff could SSH to. The rule was broader than intended — it blocked port 22 entirely, including his own management SSH session. The firewall had HA sync, so the rule propagated to the secondary within seconds. Both firewalls now rejected SSH. 40-minute lockout requiring physical console access.

Prevention: Always have an auto-revert: echo "iptables-restore < /tmp/iptables-backup.rules" | at now + 10 minutes. If you lock yourself out, the rules revert in 10 minutes.

Connection Tracking (conntrack)¶

iptables is stateful — it tracks connections. The state ESTABLISHED,RELATED rule works because conntrack remembers:

# See tracked connections
conntrack -L | head -10
# → tcp  6 300 ESTABLISHED src=10.0.1.50 dst=10.0.2.100 sport=45678 dport=443
#   ↑ proto  ↑ TTL  ↑ state     ↑ client         ↑ server

# Count tracked connections
conntrack -C
# → 12345

# Table size limit
cat /proc/sys/net/netfilter/nf_conntrack_max
# → 262144

Gotcha: When conntrack table fills (nf_conntrack: table full, dropping packet in dmesg), new connections are silently dropped. No RST, no ICMP, no logs. Just black holes. This is one of the most common causes of mysterious packet loss under load.

Flashcard Check¶

Q1: iptables chains — which one is for packets destined for this host?

INPUT. FORWARD is for packets transiting through. PREROUTING runs before the routing decision. OUTPUT is for locally-generated packets.

Q2: Rules evaluate ___. First ___ wins.

Top to bottom. First match wins. A DROP at line 1 blocks everything, even if ACCEPT exists at line 5.

Q3: Where does DNAT happen?

PREROUTING chain. The destination is rewritten BEFORE the routing decision, so the kernel routes the packet to the new destination.

Q4: nf_conntrack: table full — what happens?

New connections are silently dropped. No error to the client — just timeout. Increase nf_conntrack_max and monitor the count.

Cheat Sheet¶

Task	Command
List all rules	`iptables -L -n -v --line-numbers`
List NAT rules	`iptables -t nat -L -n -v`
Save rules	`iptables-save > backup.rules`
Restore rules	`iptables-restore < backup.rules`
Flush all rules	`iptables -F` (careful!)
Delete rule by number	`iptables -D INPUT 3`
Insert at position	`iptables -I INPUT 1 -p tcp --dport 22 -j ACCEPT`
Conntrack count	`conntrack -C`
Conntrack max	`cat /proc/sys/net/netfilter/nf_conntrack_max`

Chain Flow for Incoming Packet¶

PREROUTING → routing → INPUT (if for this host) or FORWARD (if transiting)
                                                    → POSTROUTING → out

Chain Flow for Outgoing Packet¶

OUTPUT → POSTROUTING → out to network

Takeaways¶

Rules evaluate top to bottom. First match wins. A misplaced DROP blocks everything below it. Order matters more than content.
Always allow ESTABLISHED,RELATED first. This keeps existing connections (including your SSH session) alive when you modify rules.
PREROUTING = DNAT, POSTROUTING = SNAT. Destination changes happen before routing, source changes happen after. This is how Docker port mapping and Kubernetes Services work.
conntrack table full = silent packet drops. Monitor nf_conntrack_count vs nf_conntrack_max. When it fills, you get the worst kind of failure: no errors.
Auto-revert before changing rules on remote servers. at now + 10 minutes to restore the old rules if you lock yourself out.

Exercises¶

List and interpret existing rules. Run sudo iptables -L -n -v --line-numbers and sudo iptables -t nat -L -n -v. If you are on a machine running Docker, identify the DOCKER chain entries and explain what each DNAT rule does. If no Docker rules exist, note the default policy for each chain (ACCEPT or DROP) and what that implies.
Build a minimal firewall in a container. Run a throwaway container with NET_ADMIN capability: docker run --rm -it --cap-add NET_ADMIN ubuntu bash. Inside, install iptables (apt update && apt install -y iptables). Add rules to: (a) allow loopback, (b) allow ESTABLISHED,RELATED, (c) drop everything else on INPUT. Verify with iptables -L INPUT -n --line-numbers. Confirm the rule order is correct (ACCEPT rules before the DROP).
Test rule ordering. In the same container from exercise 2, insert a DROP rule at position 1: iptables -I INPUT 1 -j DROP. Then try apt update (it will fail because DNS and HTTP are blocked). Remove the DROP rule with iptables -D INPUT 1 and confirm connectivity returns. This demonstrates why first-match-wins ordering matters.
Inspect conntrack state. On a Linux host (or container with NET_ADMIN), install conntrack-tools (apt install -y conntrack). Run conntrack -L to see tracked connections. Make a curl request to any URL, then run conntrack -L again and find the new entry. Identify the protocol, state, source/destination, and TTL. Check the table size limit with cat /proc/sys/net/netfilter/nf_conntrack_max.
Practice safe rule changes with auto-revert. In a container or VM, save current rules with iptables-save > /tmp/backup.rules. Schedule an auto-revert: (sleep 60 && iptables-restore < /tmp/backup.rules) &. Add a restrictive rule that blocks outbound traffic. Wait for the revert to fire and confirm connectivity is restored. This simulates the safety pattern for remote firewall changes.

What Happens When You Click a Link — the packet path that iptables inspects
Connection Refused — when iptables REJECT causes "connection refused"
The Container Escape — container networking through iptables