Skip to content

Firewalls - Street-Level Ops

Real-world firewall diagnosis and management workflows for production Linux systems.

Task: Emergency — Locked Out After Setting DROP Policy

# You set INPUT DROP before adding SSH rule. Console access required.
# From the console or BMC:
$ iptables -P INPUT ACCEPT
$ iptables -F

# Now rebuild properly — SSH rule FIRST
$ iptables -A INPUT -i lo -j ACCEPT
$ iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
$ iptables -A INPUT -p tcp --dport 22 -j ACCEPT
$ iptables -P INPUT DROP

# Safety net for next time
$ echo "iptables -F; iptables -P INPUT ACCEPT" | at now + 5 minutes
# Make changes, then cancel: atrm <job_id>

War story: The at safety net is the single most important trick in firewall management. Without it, a typo in a remote iptables rule locks you out permanently — the only recovery is console/BMC access or a datacenter visit. Always set the at timer before changing any remote firewall rule.

Task: Find Why a Port Is Being Blocked

# App cannot reach port 8080. Check if firewall is the cause.
$ iptables -L -n -v --line-numbers | grep 8080
# No output — no explicit rule. Check default policy:
$ iptables -L INPUT | head -1
Chain INPUT (policy DROP)

# Policy is DROP and no ACCEPT rule for 8080. Add it:
$ iptables -A INPUT -p tcp --dport 8080 -j ACCEPT

# Verify traffic is flowing
$ ss -tlnp | grep 8080
LISTEN  0  128  *:8080  *:*  users:(("myapp",pid=4521,fd=6))

Task: Debug with LOG Before DROP

# Insert LOG rule at position 1 to see what is being dropped
$ iptables -I INPUT 1 -j LOG --log-prefix "FW-DEBUG: " --log-level 4

# Watch the log
$ journalctl -k -f | grep "FW-DEBUG"
# FW-DEBUG: IN=eth0 OUT= SRC=10.0.0.50 DST=10.0.0.10 PROTO=TCP DPT=3306

# Found it — port 3306 from 10.0.0.50 is being dropped
# Add the allow rule
$ iptables -I INPUT 2 -p tcp -s 10.0.0.50 --dport 3306 -j ACCEPT

# CRITICAL: Remove the debug LOG rule when done
$ iptables -D INPUT 1

Gotcha: LOG rules on a busy server can flood /var/log/kern.log and fill the disk within minutes. Always use --log-level 4 (warning) to avoid cluttering higher-severity logs, and add -m limit --limit 5/min to rate-limit the logging. Never leave a debug LOG rule in place overnight.

Task: Check Conntrack Table Under Load

# Load balancer dropping new connections intermittently
$ dmesg | grep conntrack
[89012.456] nf_conntrack: table full, dropping packet

$ cat /proc/sys/net/netfilter/nf_conntrack_count
65432
$ cat /proc/sys/net/netfilter/nf_conntrack_max
65536

# Nearly full. Increase limits:
$ sysctl -w net.netfilter.nf_conntrack_max=262144
$ sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30

# Persist
$ cat >> /etc/sysctl.d/99-conntrack.conf <<'EOF'
net.netfilter.nf_conntrack_max = 262144
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30
EOF

Scale note: Each conntrack entry consumes ~300 bytes of kernel memory. At the default max of 65536, that is ~20MB. At 1 million entries (busy load balancer), it is ~300MB. Size nf_conntrack_max based on your expected concurrent connection count plus 20% headroom. Monitor with node_nf_conntrack_entries in Prometheus node_exporter.

Task: Save and Restore iptables Rules

# Snapshot current rules before making changes
$ iptables-save > /root/iptables-backup-$(date +%Y%m%d).rules

# Make changes...

# If something goes wrong, roll back:
$ iptables-restore < /root/iptables-backup-20260315.rules

# Persist rules for reboot (Debian/Ubuntu)
$ iptables-save > /etc/iptables/rules.v4

# Persist rules (RHEL/CentOS)
$ service iptables save

One-liner: Before any firewall change: iptables-save > /root/iptables-$(date +%Y%m%d-%H%M).rules. This 5-second habit has saved countless engineers from hours of manual rule reconstruction after a bad change.

Task: Allow a Service Through firewalld

# Check current zone and rules
$ firewall-cmd --get-active-zones
public
  interfaces: eth0

$ firewall-cmd --zone=public --list-all
public (active)
  services: ssh dhcpv6-client

# Add HTTPS permanently
$ firewall-cmd --zone=public --add-service=https --permanent
$ firewall-cmd --reload

# Verify
$ firewall-cmd --zone=public --list-services
ssh dhcpv6-client https

# Add a custom port
$ firewall-cmd --zone=public --add-port=9090/tcp --permanent
$ firewall-cmd --reload

Task: Rate-Limit SSH to Mitigate Brute Force

# Track new SSH connections, drop if >4 in 60 seconds
$ iptables -A INPUT -p tcp --dport 22 -m state --state NEW \
    -m recent --set --name SSH
$ iptables -A INPUT -p tcp --dport 22 -m state --state NEW \
    -m recent --update --seconds 60 --hitcount 4 --name SSH -j DROP
$ iptables -A INPUT -p tcp --dport 22 -j ACCEPT

# Test: rapid SSH attempts from another host get blocked after 4th

Task: Inspect Docker/Kubernetes iptables Rules

# See what Docker has added
$ iptables -t nat -L -n | grep -A3 DOCKER
Chain DOCKER (2 references)
DNAT  tcp  --  0.0.0.0/0  0.0.0.0/0  tcp dpt:8080 to:172.17.0.2:80

# Count kube-proxy rules on a k8s node
$ iptables-save | wc -l
6842

# NEVER run iptables -F on a k8s node — it wipes all kube-proxy Service rules
# Instead, snapshot first:
$ iptables-save > /root/iptables-k8s-snapshot.rules

Gotcha: On a Kubernetes node, iptables-save | wc -l returning thousands of rules is normal — kube-proxy creates two rules per Service endpoint. Flushing these rules (iptables -F) instantly breaks all Service-to-Pod routing on that node. If you must debug iptables on a k8s node, use iptables -L -t nat -n | grep <svc-clusterIP> to inspect specific Service chains without touching anything.

Task: Set Up Basic nftables Ruleset

$ cat > /etc/nftables.conf <<'NFTEOF'
#!/usr/sbin/nft -f
flush ruleset

table inet filter {
    chain input {
        type filter hook input priority 0; policy drop;
        iif lo accept
        ct state established,related accept
        tcp dport 22 accept
        tcp dport { 80, 443 } accept
        log prefix "nft-dropped: " drop
    }
    chain forward {
        type filter hook forward priority 0; policy drop;
    }
    chain output {
        type filter hook output priority 0; policy accept;
    }
}
NFTEOF

$ nft -f /etc/nftables.conf
$ systemctl enable nftables

Under the hood: nftables replaces iptables, ip6tables, arptables, and ebtables with a single framework. The inet address family handles both IPv4 and IPv6 in one ruleset — no more maintaining parallel rule sets. Debian 11+ and RHEL 9+ use nftables as the default backend even when you type iptables (via the iptables-nft compatibility layer).

Task: Verify Firewall Is Not the Problem

# Quick test: temporarily accept all, see if traffic flows
# (Only on non-production or with safety net)
$ iptables-save > /tmp/fw-backup.rules
$ iptables -P INPUT ACCEPT
$ iptables -F

# Test the connection. If it works, the firewall was blocking it.
# Restore immediately:
$ iptables-restore < /tmp/fw-backup.rules

Remember: iptables chain processing mnemonic: P-I-F-O — PREROUTING (DNAT, before routing decision), INPUT (to local processes), FORWARD (through the box), OUTPUT (from local processes), then POSTROUTING (SNAT, after routing decision). Packets to the box itself hit INPUT, not FORWARD. Packets passing through hit FORWARD, not INPUT.