Solution¶

Triage¶

Check iptables rules with counters:

iptables -L -v -n --line-numbers
iptables -L OUTPUT -v -n --line-numbers

Test direct TCP connectivity:

nc -zv 203.0.113.50 443 -w 5
curl -v --connect-timeout 10 https://payments.provider.com

Trace the connection attempt at the syscall level:

strace -e trace=connect curl -s https://payments.provider.com 2>&1 | grep connect

Compare working vs. non-working destinations:

nc -zv google.com 443 -w 5      # works
nc -zv 203.0.113.50 443 -w 5    # blocked

Root Cause¶

The security team added an iptables rule to the OUTPUT chain that drops all traffic to the 203.0.113.0/24 subnet as part of a "block known bad IP ranges" hardening script. The payment gateway payments.provider.com resolves to 203.0.113.50, which falls within this blocked range.

The rule was inserted as rule #3 in the OUTPUT chain, before the general ACCEPT rule for established connections. The DROP rule matches outgoing SYN packets to port 443, silently dropping them. The application's TCP connection attempt times out because no RST or ICMP unreachable is returned.

Fix¶

Identify the rule number:

iptables -L OUTPUT -v -n --line-numbers | grep 203.0.113

Delete the offending rule:

iptables -D OUTPUT 3    # or by specification:
iptables -D OUTPUT -d 203.0.113.0/24 -j DROP

If the subnet block is needed, add an exception for the payment gateway:
```
iptables -I OUTPUT 3 -d 203.0.113.50/32 -p tcp --dport 443 -j ACCEPT
```

Verify the fix:

curl -v --connect-timeout 10 https://payments.provider.com

Persist the rules:

iptables-save > /etc/iptables/rules.v4    # Debian/Ubuntu
# or
service iptables save                      # RHEL/CentOS

Rollback / Safety¶

Before modifying rules, save a backup: iptables-save > /tmp/iptables-backup.rules.
To restore: iptables-restore < /tmp/iptables-backup.rules.
Test changes immediately; do not leave SSH and rely on at or a cron job to restore rules if locked out.

Common Traps¶

Not using -n flag. Without -n, iptables does reverse DNS lookups on every IP, making the output extremely slow and potentially misleading.
Forgetting rule order matters. Inserting an ACCEPT after a DROP does nothing. The ACCEPT must come before the DROP in the chain.
Confusing DROP and REJECT. DROP silently discards (connection times out). REJECT sends an ICMP unreachable (connection refused). DROP is harder to diagnose.
Not checking all tables. Rules can exist in filter, nat, mangle, and raw tables. Check with iptables -t nat -L -v -n, etc.
Assuming iptables rules persist. On most distributions, iptables rules are ephemeral and lost on reboot unless explicitly saved.