Skip to content

Routing - Primer

Why This Matters

Every packet that leaves a host makes a routing decision. When traffic takes an unexpected path — or no path at all — the routing table is the first place to look. Misrouted traffic causes latency spikes, asymmetric path failures, blackholes, and security policy violations. Understanding routing at the Linux host level and at the network level is a core ops skill.

Routing Table Basics

A routing table is an ordered set of rules that tells the kernel where to send packets based on their destination address. Every host has one.

Viewing the Routing Table

# Modern (iproute2) — preferred
ip route show

# Legacy
route -n
netstat -rn

Typical output:

default via 192.168.1.1 dev eth0 proto dhcp metric 100
10.0.0.0/8 via 10.255.0.1 dev wg0
172.16.0.0/16 dev docker0 proto kernel scope link src 172.16.0.1
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.50

Key Fields

Field Meaning
default Catch-all route (0.0.0.0/0) — the default gateway
via Next-hop IP address
dev Outbound interface
proto How the route was learned (kernel, dhcp, static, bgp)
metric Priority — lower wins when multiple routes match
scope link Destination is directly attached (no next-hop needed)

Longest Prefix Match

The kernel selects the most specific matching route. A packet to 10.0.5.3 matches 10.0.5.0/24 over 10.0.0.0/8 over default. This is the single most important routing concept.

Default Gateway

The default gateway handles all traffic that does not match a more specific route. If it is missing or points at an unreachable next-hop, the host can only talk to directly-connected networks.

# Check default gateway
ip route show default

# Set a default gateway (transient)
ip route add default via 192.168.1.1 dev eth0

# Delete default route
ip route del default

Common failure: DHCP lease expires, default route disappears, host loses external connectivity but local network still works.

Static Routes

Static routes are manually configured and do not change unless an administrator modifies them.

# Add a static route
ip route add 10.100.0.0/16 via 10.0.0.1 dev eth0

# Add with a specific metric
ip route add 10.100.0.0/16 via 10.0.0.1 metric 200

# Persistent (Debian/Ubuntu — /etc/network/interfaces)
up ip route add 10.100.0.0/16 via 10.0.0.1

# Persistent (RHEL/CentOS — /etc/sysconfig/network-scripts/route-eth0)
10.100.0.0/16 via 10.0.0.1

Static routes are simple and predictable, but they do not adapt to topology changes. Use them for stable paths — VPN tunnels, management networks, known internal subnets.

Dynamic Routing Protocols

Dynamic protocols let routers exchange reachability information and adapt to failures automatically.

OSPF (Open Shortest Path First)

  • Interior gateway protocol (IGP) — used within a single organization
  • Link-state: every router builds a complete topology map
  • Converges fast (sub-second with tuning)
  • Uses areas to scale (Area 0 is the backbone)
  • Metric is cost, typically based on bandwidth
  • Common in enterprise and datacenter east-west fabrics

BGP (Border Gateway Protocol)

  • Exterior gateway protocol — the protocol that runs the internet
  • Path-vector: routes carry the full AS (autonomous system) path
  • Slow convergence by design (stability over speed)
  • eBGP: between organizations (different AS numbers)
  • iBGP: within an organization (same AS, used for internal route distribution)
  • Increasingly used inside datacenters (BGP-on-the-host, Calico in k8s)
  • Ops relevance: cloud VPN/Direct Connect, k8s networking, anycast

When You See Dynamic Routing in Ops

You rarely configure OSPF or BGP directly as a DevOps engineer, but you need to: - Understand why a route exists (proto bgp or proto ospf in the routing table) - Know that route convergence takes time after a link failure - Recognize when a routing protocol is injecting unexpected routes - Read BGP looking glass output when debugging cloud connectivity

Asymmetric Routing

Asymmetric routing occurs when the forward path (A to B) differs from the return path (B to A).

Why It Happens

  • Multiple uplinks with different default gateways
  • BGP paths differ inbound vs outbound
  • Policy-based routing sends traffic out one path, return comes back another
  • Load balancers or NAT devices only see one direction

Symptoms

  • Stateful firewalls drop return traffic (they never saw the outbound SYN)
  • TCP connections hang or reset intermittently
  • Packet captures on one interface show requests but no responses
  • Traceroute works but application traffic fails

Debugging

# Check if forward and reverse paths differ
traceroute -n <destination>     # from host A
# Then from destination, traceroute back

# Check conntrack on firewalls
conntrack -L | grep <ip>

# Watch for RSTs
tcpdump -ni eth0 'tcp[tcpflags] & tcp-rst != 0' and host <ip>

Fix: ensure stateful devices see both directions, or use rp_filter settings carefully.

Policy-Based Routing

Standard routing decides based on destination only. Policy-based routing (PBR) can route based on source address, port, protocol, or packet mark.

Linux Implementation: ip rule + Multiple Tables

# View policy rules (priority order)
ip rule show

# Default output:
# 0:    from all lookup local
# 32766: from all lookup main
# 32767: from all lookup default

# Create a custom routing table
echo "100 custom" >> /etc/iproute2/rt_tables

# Add a route to the custom table
ip route add default via 10.0.1.1 table custom

# Add a rule: traffic from 192.168.5.0/24 uses the custom table
ip rule add from 192.168.5.0/24 table custom priority 100

# Rule based on source port (using fwmark + iptables)
iptables -t mangle -A OUTPUT -p tcp --sport 8080 -j MARK --set-mark 1
ip rule add fwmark 1 table custom priority 200

Use Cases

  • Multi-homed hosts: different source subnets exit via different ISPs
  • VPN split tunneling: only certain traffic goes through the tunnel
  • Container networking: pod traffic uses a different route than host traffic
  • Compliance: traffic from certain applications must use a specific path

Traceroute and MTR Interpretation

traceroute

Shows the path packets take, hop by hop:

traceroute -n 8.8.8.8

 1  192.168.1.1    1.2 ms   1.0 ms   1.1 ms
 2  10.0.0.1       5.3 ms   5.1 ms   5.4 ms
 3  * * *
 4  72.14.215.65  12.1 ms  11.8 ms  12.3 ms
  • * * * means the hop did not respond (ICMP rate-limited or filtered — not necessarily a problem)
  • Latency jump between hops indicates the slow segment
  • Asymmetric paths mean traceroute latency per-hop can be misleading (each hop's reply may take a different return path)

mtr (My Traceroute)

Combines traceroute with continuous ping — better for spotting intermittent issues:

mtr -n --report -c 100 8.8.8.8

Key columns: - Loss%: packet loss at that hop (loss only at the final hop matters most — intermediate loss is often ICMP deprioritization) - Avg/Best/Wrst: latency statistics - StDev: high standard deviation suggests congestion or flapping

Common Ops Scenarios

Traffic Takes an Unexpected Path

Diagnosis: 

ip route get <destination>        # Shows exactly which route the kernel picks
ip route show table all           # Check all tables, not just main
ip rule show                      # Check for PBR rules

Common causes: more specific route exists, PBR rule overrides, VPN injecting routes, Docker/k8s adding routes for container networks.

Blackhole Routes

A blackhole route silently drops matching traffic:

# Create a blackhole (used for DDoS mitigation, blocking ranges)
ip route add blackhole 198.51.100.0/24

# Check if a blackhole exists
ip route show type blackhole

Symptoms: no ICMP unreachable, no RST — traffic simply vanishes. ip route get reveals the blackhole.

Unreachable Next-Hop

The route exists but the next-hop gateway is down:

# Check if next-hop is reachable
ping -c 2 <gateway-ip>

# Check ARP table — is the gateway's MAC resolved?
ip neigh show <gateway-ip>

# Look for INCOMPLETE or FAILED entries
ip neigh show | grep -E 'INCOMPLETE|FAILED'

Symptoms: traffic to remote networks fails, traffic to local network works. The route looks correct but the gateway is not answering ARP.

Route Flapping

Routes appearing and disappearing — often from an unstable link or misconfigured dynamic routing. Watch in real time with ip monitor route.

Container and Kubernetes Routing

CNI plugins (Calico, Cilium, Flannel) add routes automatically. When debugging pod connectivity, check ip route show on the node — missing or stale CNI routes indicate plugin misconfiguration.

Quick Reference

Task Command
Show routes / specific dest ip route show / ip route get <ip>
Add/delete static route ip route add/del <net> via <gw>
All tables / policy rules ip route show table all / ip rule show
Watch changes / ARP table ip monitor route / ip neigh show
Trace path traceroute -n <ip> / mtr -n --report <ip>
Blackhole routes ip route show type blackhole

Wiki Navigation

Next Steps

Related Content