Skip to content

Solution: Firewall Shadow Rule

Summary

The ACCEPT rule for port 8443 was appended to the end of the INPUT chain (rule 47). However, rule 12 is a broad DROP rule that blocks all TCP traffic to destination ports 8000-9999 from non-localhost sources. Since iptables evaluates rules top to bottom and stops at the first match, the DROP at position 12 catches the traffic before the ACCEPT at position 47 is ever reached. This is called a "shadow rule."

Senior Workflow

Step 1: List the full INPUT chain with counters and line numbers

iptables -L INPUT -n -v --line-numbers

Look for: - The position of the ACCEPT rule for port 8443 - Any earlier rules that match the same traffic (same source, dest, port range) - Packet counters: the ACCEPT rule should have 0 packets (shadowed)

Step 2: Identify the shadowing rule

# Look for rules matching port 8443 traffic:
iptables -L INPUT -n -v --line-numbers | grep -E "8443|8000.*9999|dpt:8"

In this case, rule 12 drops all TCP to ports 8000-9999: 

12   14892  1.2M  DROP  tcp  --  *  *  0.0.0.0/0  0.0.0.0/0  tcp dpts:8000:9999

Step 3: Verify with packet counters

The ACCEPT rule at position 47 shows 0 packets: 

47   0     0     ACCEPT  tcp  --  *  *  10.60.10.0/24  0.0.0.0/0  tcp dpt:8443

Meanwhile, rule 12's counter keeps incrementing -- it is eating the traffic.

Step 4: Test with a LOG rule (optional diagnostic)

# Insert a LOG rule just before the DROP to see matched packets:
iptables -I INPUT 12 -p tcp -s 10.60.10.0/24 --dport 8443 -j LOG --log-prefix "SHADOW-DEBUG: "

# Attempt the connection, then check:
dmesg | grep SHADOW-DEBUG

Step 5: Apply the fix

# Remove the shadowed ACCEPT rule from the end:
iptables -D INPUT -p tcp -s 10.60.10.0/24 --dport 8443 -j ACCEPT

# Insert the ACCEPT rule ABOVE the DROP rule (at position 12):
iptables -I INPUT 12 -p tcp -s 10.60.10.0/24 --dport 8443 -j ACCEPT

# The old DROP rule is now at position 13

Step 6: Verify the fix

# From the API gateway:
curl -k https://10.60.5.30:8443/health
# Should return 200 OK

# Verify counters:
iptables -L INPUT -n -v --line-numbers | head -15
# The new ACCEPT rule at position 12 should show incrementing packets

Step 7: Persist the rules

iptables-save > /etc/iptables/rules.v4
# Or use:
netfilter-persistent save

Step 8: Clean up the ruleset

With 47 rules grown organically, schedule a full review: - Remove obsolete rules for decommissioned services - Consolidate overlapping rules - Document the purpose of each rule - Consider migrating to nftables or firewalld for better management

Common Pitfalls

  • Appending when inserting is needed: -A always adds to the end. With deny rules earlier in the chain, the new rule will never be reached. Use -I INPUT <position> for proper placement.
  • Not checking counters: The zero-packet counter on the ACCEPT rule is the clearest indicator of a shadow rule.
  • Moving the DROP rule instead of inserting the ACCEPT: Moving a broad DROP rule to the end of the chain could expose other ports unintentionally.
  • Not persisting changes: iptables rules are lost on reboot unless saved.
  • Large unmanaged rulesets: 47+ rules without documentation or structure is a maintenance nightmare. Use chain organization or a management tool.
  • Not cleaning up the LOG rule: If you added a LOG rule for debugging, remove it after diagnosis to avoid log spam.