Skip to content

Firewalls - Primer

Why This Matters

Firewalls are the first line of defense between your infrastructure and everything trying to reach it. A misconfigured firewall rule can either lock you out of your own servers or silently expose services to the internet. In production, firewall mistakes cause two kinds of incidents: outages (legitimate traffic blocked) and breaches (malicious traffic allowed). Every ops engineer needs to understand the Linux firewall stack — not just how to add a rule, but how rules interact, how to debug them, and how to avoid the common mistakes that turn a routine change into a 3 AM page.

Core Concepts

1. The Linux Firewall Stack: netfilter → iptables → nftables

The Linux kernel handles packet filtering through netfilter, a set of hooks in the networking stack. Everything else is a userspace tool that talks to netfilter.

Name origin: "netfilter" was created by Rusty Russell in 1998 for Linux 2.4. The name "iptables" comes from "IP tables" -- the kernel data structures that hold firewall rules. The "nft" in nftables stands for "netfilter tables." Despite the naming, nftables is a complete rewrite, not an evolution of iptables code.

Evolution: - netfilter (kernel framework) — the actual packet processing engine, present since Linux 2.4 - iptables (userspace tool, 2001) — the classic interface to netfilter. Still the most widely documented and deployed - nftables (userspace tool, 2014) — the designated successor. Ships as default on RHEL 8+, Debian 10+, Ubuntu 20.04+

Which to use when: - Legacy systems, existing automation, Kubernetes clusters → you will encounter iptables - New standalone hosts, modern distros → prefer nftables - If firewalld or ufw is already managing the host → use those frontends, do not mix raw iptables calls underneath

# Check what's actually in use
iptables --version
# iptables v1.8.7 (nf_tables)   ← nftables backend, iptables syntax
# iptables v1.8.4 (legacy)      ← true legacy iptables

nft list ruleset   # show nftables rules directly

2. iptables Fundamentals

iptables organizes rules into tables, chains, and targets.

Tables (each serves a different purpose):

Table Purpose Common chains
filter Accept/drop/reject packets INPUT, FORWARD, OUTPUT
nat Network address translation PREROUTING, POSTROUTING, OUTPUT
mangle Packet header modification All five chains
raw Bypass connection tracking PREROUTING, OUTPUT

If you don't specify -t, iptables defaults to the filter table.

Chains: - INPUT — packets destined for the local host - OUTPUT — packets originating from the local host - FORWARD — packets routed through the host (not for it) - PREROUTING — before routing decision (used for DNAT) - POSTROUTING — after routing decision (used for SNAT/masquerade)

Targets: - ACCEPT — allow the packet - DROP — silently discard (sender gets no response) - REJECT — discard and send ICMP error back (polite but reveals firewall exists) - LOG — log to syslog, then continue processing (non-terminating)

Rule ordering matters. iptables evaluates rules top-to-bottom within a chain. First match wins. If no rule matches, the chain's default policy applies.

Remember: Mnemonic: "First match fires." Unlike security groups (which evaluate all rules), iptables and NACLs stop at the first matching rule. A misplaced DROP rule above an ACCEPT rule silently blocks traffic with no error message. Always check rule order with iptables -L --line-numbers.

# List rules with line numbers, numeric addresses, packet counts
iptables -L -n -v --line-numbers

# Example output:
# Chain INPUT (policy DROP 0 packets, 0 bytes)
# num   pkts bytes target   prot opt in  out  source      destination
# 1     1.2M  890M ACCEPT   all  --  lo  *    0.0.0.0/0   0.0.0.0/0
# 2     45M   32G  ACCEPT   all  --  *   *    0.0.0.0/0   0.0.0.0/0   state RELATED,ESTABLISHED
# 3     8234  493K ACCEPT   tcp  --  *   *    0.0.0.0/0   0.0.0.0/0   tcp dpt:22
# 4     1203  72K  ACCEPT   tcp  --  *   *    0.0.0.0/0   0.0.0.0/0   tcp dpt:443

3. Common iptables Patterns

Allow SSH (port 22): 

iptables -A INPUT -p tcp --dport 22 -j ACCEPT

Allow established and related connections (critical — without this, replies to your own outbound connections get dropped): 

iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

Allow loopback: 

iptables -A INPUT -i lo -j ACCEPT

Block everything else (set default policy): 

iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

A minimal secure baseline (order matters): 

# Flush existing rules
iptables -F
iptables -X

# Default policies
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

# Allow loopback
iptables -A INPUT -i lo -j ACCEPT

# Allow established/related
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow SSH
iptables -A INPUT -p tcp --dport 22 -j ACCEPT

# Allow HTTP/HTTPS
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT

# Log dropped packets (optional, useful for debugging)
iptables -A INPUT -j LOG --log-prefix "iptables-dropped: " --log-level 4

Rate limiting SSH (brute-force mitigation): 

iptables -A INPUT -p tcp --dport 22 -m state --state NEW \
    -m recent --set --name SSH
iptables -A INPUT -p tcp --dport 22 -m state --state NEW \
    -m recent --update --seconds 60 --hitcount 4 --name SSH -j DROP

Port forwarding (DNAT): 

# Forward external port 8080 to internal host 10.0.1.5:80
iptables -t nat -A PREROUTING -p tcp --dport 8080 -j DNAT --to-destination 10.0.1.5:80
iptables -A FORWARD -p tcp -d 10.0.1.5 --dport 80 -j ACCEPT
echo 1 > /proc/sys/net/ipv4/ip_forward

4. nftables: The Future of Linux Firewalling

nftables replaces iptables with a cleaner syntax and better performance. The kernel side uses a virtual machine that processes rulesets more efficiently.

Key syntax differences:

Concept iptables nftables
List rules iptables -L -n -v nft list ruleset
Add rule iptables -A INPUT -p tcp ... nft add rule inet filter input tcp ...
Tables Built-in (filter, nat, mangle) User-defined
Chains Built-in (INPUT, OUTPUT, ...) User-defined with hook/priority
Sets ipset (separate tool) Native sets and maps

Basic nftables ruleset: 

#!/usr/sbin/nft -f

flush ruleset

table inet filter {
    chain input {
        type filter hook input priority 0; policy drop;

        # Allow loopback
        iif lo accept

        # Allow established/related
        ct state established,related accept

        # Allow SSH
        tcp dport 22 accept

        # Allow HTTP/HTTPS
        tcp dport { 80, 443 } accept

        # Log and drop everything else
        log prefix "nft-dropped: " drop
    }

    chain forward {
        type filter hook forward priority 0; policy drop;
    }

    chain output {
        type filter hook output priority 0; policy accept;
    }
}

Why nftables is the future: - Single tool replaces iptables, ip6tables, arptables, ebtables - Atomic rule replacement (load entire ruleset at once — no window of partial rules) - Native set support (no separate ipset tool) - Better syntax for complex rulesets - Improved performance for large rule counts

5. firewalld and ufw: Higher-Level Frontends

firewalld (RHEL/CentOS/Fedora default): - Zone-based: each network interface belongs to a zone (public, trusted, internal, dmz, etc.) - Runtime vs permanent: changes are temporary unless you add --permanent - D-Bus interface for programmatic access

# Check status
firewall-cmd --state
firewall-cmd --get-active-zones

# Allow a service
firewall-cmd --zone=public --add-service=https --permanent
firewall-cmd --reload

# Allow a port
firewall-cmd --zone=public --add-port=8080/tcp --permanent
firewall-cmd --reload

# List rules in a zone
firewall-cmd --zone=public --list-all

ufw (Ubuntu/Debian default frontend): - Simpler syntax, designed for single-host use - Wraps iptables under the hood

ufw enable
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow 443/tcp
ufw status verbose

When to use which: - firewalld — multi-zone environments, servers with multiple interfaces, RHEL-family - ufw — single-purpose servers, quick setup, Ubuntu/Debian - Raw iptables/nftables — containers, Kubernetes nodes, custom NAT, automation that needs precise control

6. Debugging Firewall Issues

When traffic is not flowing and you suspect the firewall:

Step 1: Check current rules 

# iptables — all tables
iptables -L -n -v              # filter table
iptables -t nat -L -n -v       # NAT table
iptables -t mangle -L -n -v    # mangle table

# nftables
nft list ruleset

Step 2: Check packet counters The pkts and bytes columns in iptables -L -n -v tell you which rules are matching. If your ACCEPT rule shows zero hits, packets are being matched by an earlier rule or never arriving.

Step 3: Use the LOG target 

# Insert a LOG rule before the DROP to see what's being blocked
iptables -I INPUT 1 -j LOG --log-prefix "FW-DEBUG: " --log-level 4

# Watch the log
tail -f /var/log/kern.log | grep "FW-DEBUG"
# Or: journalctl -k -f | grep "FW-DEBUG"

# REMOVE the debug rule when done (it is rule #1)
iptables -D INPUT 1

Step 4: Check connection tracking 

conntrack -L                   # list all tracked connections
conntrack -L -p tcp --dport 80 # filter by port
conntrack -C                   # count entries
cat /proc/sys/net/netfilter/nf_conntrack_max  # max tracked connections

Connection tracking table exhaustion is a real production issue. When the table fills up, new connections are silently dropped even if firewall rules would allow them.

War story: A load balancer handling 50,000 concurrent connections started silently dropping new connections. No firewall rule changes, no errors in application logs. The only clue was dmesg showing nf_conntrack: table full, dropping packet. The default nf_conntrack_max was 65536, and with connection tracking entries lingering in TIME_WAIT, the table filled up. Fix: increase nf_conntrack_max and decrease nf_conntrack_tcp_timeout_time_wait.

Debug clue: If traffic is being dropped but your firewall rules look correct, check dmesg for nf_conntrack: table full before anything else. This is one of the most commonly missed causes of mysterious connection failures.

Step 5: Verify with packet capture 

# Are packets even reaching the interface?
tcpdump -i eth0 -n port 443 -c 10

7. Persistence

iptables rules live in kernel memory. They vanish on reboot unless you persist them.

Debian/Ubuntu: 

# Save
iptables-save > /etc/iptables/rules.v4
ip6tables-save > /etc/iptables/rules.v6

# Restore (happens automatically via iptables-persistent package)
apt install iptables-persistent
# Rules load from /etc/iptables/rules.v4 on boot

# Manual restore
iptables-restore < /etc/iptables/rules.v4

RHEL/CentOS (without firewalld): 

service iptables save          # saves to /etc/sysconfig/iptables
systemctl enable iptables

nftables persistence: 

# Save
nft list ruleset > /etc/nftables.conf

# The nftables systemd unit loads /etc/nftables.conf on boot
systemctl enable nftables

firewalld persists rules by default when you use --permanent and --reload. This is one of its main advantages over raw iptables.

8. Container and Kubernetes Implications

Docker and Kubernetes both manipulate iptables heavily. Understanding this is critical for ops.

Docker iptables rules: - Docker creates a DOCKER chain and inserts rules in FORWARD and nat tables - -p 8080:80 creates DNAT rules in the nat PREROUTING chain - Docker sets net.ipv4.ip_forward = 1 automatically - Docker inserts its rules in the FORWARD chain, which can bypass your INPUT rules

# See Docker's iptables rules
iptables -t nat -L -n | grep DOCKER
iptables -L FORWARD -n -v

Kubernetes (kube-proxy) iptables rules: - kube-proxy creates thousands of iptables rules for Service → Pod routing - Each Service gets KUBE-SERVICES, KUBE-SVC-, and KUBE-SEP- chains - NodePort services add KUBE-NODEPORTS rules

# Count kube-proxy rules (can be thousands in large clusters)
iptables-save | wc -l
# On a cluster with 200 services: easily 5000+ rules

# List service chains
iptables -t nat -L -n | grep KUBE-SVC | head -10

Why iptables -F on a Kubernetes node is catastrophic: - Flushes ALL rules including kube-proxy's Service routing - All Service ClusterIPs stop working immediately - Pod-to-Service communication breaks cluster-wide (if the node was routing traffic) - kube-proxy will eventually recreate the rules, but there is an outage window - If you also flush the nat table (iptables -t nat -F), NodePort and LoadBalancer services break too

Safe approach: If you need to debug on a k8s node, use iptables-save to snapshot first, add specific rules rather than flushing, and let kube-proxy manage its own chains.

9. Common Gotchas

First match wins: A rule at line 3 that DROPs port 22 will override an ACCEPT at line 10. Rule ordering is the #1 source of firewall misconfigurations.

# WRONG — SSH is blocked because DROP comes first
iptables -A INPUT -p tcp --dport 22 -j DROP
iptables -A INPUT -p tcp --dport 22 -s 10.0.0.0/8 -j ACCEPT

# RIGHT — specific ACCEPT before general DROP
iptables -A INPUT -p tcp --dport 22 -s 10.0.0.0/8 -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j DROP

Forgetting ESTABLISHED/RELATED: Without this rule, the host can initiate outbound connections but never receive the responses. This is the most common "everything is broken" mistake.

# This MUST be near the top of your INPUT chain
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

Locking yourself out via SSH: If you set INPUT policy DROP before adding an SSH ACCEPT rule, you lose access immediately. Always add the SSH rule first, or use a cron job to flush rules after 5 minutes as a safety net:

# Safety net: schedule a flush in case you lock yourself out
echo "iptables -F; iptables -P INPUT ACCEPT" | at now + 5 minutes
# Now make your changes. Cancel the at job once you confirm access.

iptables vs firewalld conflict: Running raw iptables commands on a host managed by firewalld creates confusion. firewalld will overwrite your manual rules on reload. Pick one management method and stick with it.

# Check if firewalld is managing the firewall
systemctl is-active firewalld
# If active, use firewall-cmd, not raw iptables

conntrack table exhaustion: High-traffic hosts (load balancers, proxies) can exhaust the connection tracking table. Symptoms: new connections silently dropped, dmesg shows nf_conntrack: table full.

# Check current vs max
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max

# Increase if needed
sysctl -w net.netfilter.nf_conntrack_max=262144

Wiki Navigation

Related Content