Firewall Footguns¶

Mistakes that lock you out, expose services, or cause cascading outages.

1. Setting DROP policy before adding SSH rule¶

You run iptables -P INPUT DROP remotely. Your current SSH session stays alive (it is ESTABLISHED), but the moment it drops, you cannot reconnect. You need console access or a BMC to recover.

Fix: Always add ESTABLISHED,RELATED and SSH ACCEPT rules before setting DROP policy. Use at now + 5 minutes to schedule an automatic flush as a safety net.

One-liner: Before any remote firewall change: at now + 5 minutes <<< 'iptables -F; iptables -P INPUT ACCEPT'. If you lock yourself out, the rules auto-reset in 5 minutes. Cancel with atrm <job-id> once you verify connectivity.

2. Forgetting the ESTABLISHED,RELATED rule¶

You add ACCEPT rules for specific ports and set DROP policy. The host can receive new connections but cannot make outbound connections — DNS, NTP, package updates all fail because return packets are dropped.

Fix: This rule must be near the top of INPUT: iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT.

3. Running `iptables -F` on a Kubernetes node¶

You flush iptables to "start clean." This destroys all kube-proxy rules — thousands of Service-to-Pod DNAT entries. Every ClusterIP and NodePort stops working. Pods cannot reach services. kube-proxy eventually recreates them, but you have a multi-minute outage.

Fix: Never flush iptables on a k8s node. Snapshot first with iptables-save. Add specific rules rather than flushing.

War story: On Kubernetes nodes, kube-proxy maintains thousands of iptables rules for Service-to-Pod DNAT. Flushing iptables destroys all of them instantly. Every ClusterIP, NodePort, and LoadBalancer service stops working. kube-proxy eventually recreates the rules, but you get a multi-minute outage on that node. Always iptables-save > /tmp/rules-backup.txt before touching anything on a k8s node.

4. Mixing raw iptables with firewalld¶

You add iptables rules directly on a host managed by firewalld. Everything works until someone runs firewall-cmd --reload. Your manual rules vanish because firewalld regenerates its ruleset from its own state.

Fix: Check systemctl is-active firewalld. If active, use firewall-cmd exclusively. If you need raw iptables, disable firewalld first.

5. Rule ordering — specific ACCEPT after general DROP¶

You append a DROP rule for port 22, then later append an ACCEPT for SSH from a specific subnet. The DROP matches first. First match wins in iptables. Your management subnet is locked out.

Fix: Put specific ACCEPT rules before general DROP rules. Use iptables -I (insert) to place rules at specific positions, not iptables -A (append).

6. Not persisting rules across reboot¶

You spend 30 minutes building a careful ruleset. The host reboots. All rules are gone — iptables rules live in kernel memory only. The host comes up with no firewall.

Fix: Save rules: iptables-save > /etc/iptables/rules.v4 (Debian) or service iptables save (RHEL). Install iptables-persistent on Debian/Ubuntu.

Default trap: iptables rules live only in kernel memory. A reboot, a power cycle, or a kernel panic wipes them all. The host comes back up with no firewall. On Debian/Ubuntu, install iptables-persistent and it auto-loads on boot. On RHEL/CentOS, systemctl enable iptables does the same.

7. Conntrack table exhaustion on busy hosts¶

Your load balancer or proxy handles thousands of connections per second. The default nf_conntrack_max (65536) fills up. New connections are silently dropped. dmesg shows nf_conntrack: table full but your monitoring does not watch dmesg.

Fix: Pre-size conntrack: sysctl -w net.netfilter.nf_conntrack_max=262144. Monitor nf_conntrack_count vs nf_conntrack_max in your metrics.

War story: A Kubernetes cluster running Consul service mesh experienced cascading connection drops when the conntrack table filled up. The kernel logged nf_conntrack: table full, dropping packet but the default monitoring stack did not scrape dmesg. SSH to the affected node took 4+ seconds (vs. <1s normally) due to 50% packet loss. The default nf_conntrack_max of 65536 is far too low for any node handling thousands of connections per second.

8. Leaving LOG rules in production¶

You add iptables -I INPUT 1 -j LOG for debugging. Every packet hitting INPUT gets logged. On a busy host, syslog fills the disk within hours. The LOG rule also adds latency to every packet.

Fix: Always remove LOG rules after debugging: iptables -D INPUT 1. If you need persistent logging, add it only to the final DROP rule with rate limiting: -m limit --limit 5/min.

9. Using REJECT instead of DROP for external traffic¶

REJECT sends an ICMP error back to the sender. This tells attackers your host exists and which ports are filtered. It also consumes bandwidth and CPU generating rejection responses during a DDoS.

Fix: Use DROP for external-facing rules (silent discard). Use REJECT for internal networks where you want fast failure feedback.

10. Forgetting ip_forward when setting up NAT¶

You configure DNAT rules to forward traffic to an internal host. Packets arrive but never reach the backend. The kernel discards forwarded packets by default because ip_forward is disabled.

Fix: Enable forwarding: sysctl -w net.ipv4.ip_forward=1. Persist in /etc/sysctl.d/. Also add matching FORWARD chain ACCEPT rules.