Subnetting and IP Addressing - Street-Level Ops¶

What experienced network operators know from years of CIDR planning, VPC buildouts, and 2 AM "why can't pod A reach service B" incidents.

VPC / Cloud Subnet Planning¶

The Standard Pattern: /16 VPC, /24 Subnets¶

VPC: 10.100.0.0/16 (65,534 usable IPs)

  Public subnets (one per AZ):
    10.100.0.0/24   (us-east-1a)  — 254 hosts
    10.100.1.0/24   (us-east-1b)  — 254 hosts
    10.100.2.0/24   (us-east-1c)  — 254 hosts

  Private subnets (one per AZ):
    10.100.10.0/24  (us-east-1a)  — 254 hosts
    10.100.11.0/24  (us-east-1b)  — 254 hosts
    10.100.12.0/24  (us-east-1c)  — 254 hosts

  Database subnets:
    10.100.20.0/24  (us-east-1a)  — 254 hosts
    10.100.21.0/24  (us-east-1b)  — 254 hosts

  Spare: 10.100.30.0/24 - 10.100.255.0/24

AWS gotcha: AWS reserves 5 IPs per subnet (first four + last). A /24 gives you 251, not 254.

Remember: AWS reserved IPs per subnet: .0 (network), .1 (VPC router), .2 (DNS), .3 (future use), .255 (broadcast). Mnemonic: N-R-D-F-B — Network, Router, DNS, Future, Broadcast.

Multi-VPC Strategy¶

Production:  10.100.0.0/16
Staging:     10.101.0.0/16
Development: 10.102.0.0/16
Shared Svcs: 10.103.0.0/16

Keep them in the same /8 but different /16 blocks. Makes VPC peering and transit gateway routes clean.

K8s CIDR Planning¶

The Three CIDRs That Must Not Overlap¶

Node CIDR:    10.100.0.0/16    (VPC — where nodes live)
Pod CIDR:     10.244.0.0/16    (overlay — where pods get IPs)
Service CIDR: 10.96.0.0/12     (virtual — ClusterIP range)

Check for overlap:

# If you can AND two network addresses with the wider mask
# and get the same result, they overlap.
# Quick check — does 10.96.0.0/12 overlap 10.100.0.0/16?
ipcalc 10.96.0.0/12
# Network: 10.96.0.0, HostMax: 10.111.255.254
# 10.100.x.x falls inside 10.96-10.111 range — OVERLAP!

# Fix: use a different service CIDR
# Service CIDR: 10.200.0.0/16

Pod CIDR Sizing¶

Each node gets a /24 slice of the pod CIDR by default (256 pod IPs per node).

Pod CIDR /16 = 256 /24 slices = 256 nodes max
Pod CIDR /14 = 1024 /24 slices = 1024 nodes max

If you need more nodes, widen the pod CIDR or use --node-cidr-mask-size to give each node a smaller slice (e.g., /25 = 128 pods per node).

Gotcha: Changing the pod CIDR or service CIDR on a running cluster is effectively impossible without rebuilding. Plan these ranges generously at cluster creation time. Doubling back later requires a cluster migration.

Docker Bridge Conflicts¶

The Problem¶

Docker's default bridge is 172.17.0.0/16. Custom docker networks allocate from 172.18.0.0/16, 172.19.0.0/16, etc. Corporate networks often use 172.16-31.x ranges internally.

# See what Docker has allocated
docker network ls
docker network inspect bridge | jq '.[0].IPAM.Config'

# See all docker networks and their subnets
docker network inspect $(docker network ls -q) | jq '.[] | {Name: .Name, Subnet: .IPAM.Config[0].Subnet}'

Fix: Override Docker's Default Pool¶

# /etc/docker/daemon.json
{
  "default-address-pools": [
    {"base": "10.200.0.0/16", "size": 24}
  ],
  "bip": "10.200.0.1/24"
}

Restart Docker after changing. This moves all Docker networks into 10.200.x.x space.

War story: A company's VPN stopped working after a developer ran docker-compose up on their laptop. Docker allocated 172.18.0.0/16 for a bridge network, which shadowed the corporate 172.18.0.0/16 subnet. All traffic to the office network went to Docker instead. Fix: always set default-address-pools in daemon.json on developer machines.

Calculating "How Many IPs Do I Have Left"¶

# AWS: check available IPs in a subnet
aws ec2 describe-subnets --subnet-ids subnet-abc123 \
  --query 'Subnets[0].AvailableIpAddressCount'

# On a Linux box: count IPs in use on a /24
ip neigh show dev eth0 | wc -l

# Calculate total vs used
# /24 = 254 usable. If 200 are in use, you have 54 left.
# /25 = 126 usable. If 100 are in use, you have 26 left.

# ipcalc gives you the total
ipcalc 10.0.1.0/24 | grep Hosts
# Hosts/Net: 254

When to Worry¶

Below 20% free: plan expansion
Below 10% free: you are one autoscale event from exhaustion
IP exhaustion in K8s pod CIDR: pods stuck in ContainerCreating, events show "no available IPs"

IPv6 in Practice¶

Where You Actually Encounter It¶

Dual-stack K8s clusters: Pods get both IPv4 and IPv6 addresses. Services can be dual-stack.
AWS/GCP/Azure VPCs: Dual-stack is increasingly the default. EC2 instances get both.
Load balancers: ALBs and NLBs serve on both stacks.
Internal services: Most still speak IPv4. IPv6 is primarily for external-facing paths.

Dual-Stack K8s Quick Checks¶

# Verify cluster has dual-stack
kubectl get nodes -o wide
# Look for both IPv4 and IPv6 addresses

# Check a pod's addresses
kubectl get pod mypod -o jsonpath='{.status.podIPs}'
# [{"ip":"10.244.1.5"},{"ip":"fd00::5"}]

# Check service type
kubectl get svc mysvc -o jsonpath='{.spec.ipFamilyPolicy}'
# SingleStack, PreferDualStack, or RequireDualStack

IPv6 One-Liners¶

# Show all IPv6 addresses
ip -6 addr show

# Show IPv6 routes
ip -6 route show

# Test IPv6 connectivity (link-local requires interface)
ping -6 fe80::1%eth0

# Test global IPv6
ping -6 2001:db8::1

# DNS lookup for AAAA records
dig AAAA example.com +short

# curl over IPv6
curl -6 http://[2001:db8::1]:8080/health

Common Subnet One-Liners¶

# What subnet is this IP in?
ipcalc 10.0.1.137/24

# Split a /24 into four /26 subnets
ipcalc 10.0.1.0/24 -s 62 62 62 62

# List all /24 subnets in a /16
for i in $(seq 0 255); do echo "10.0.$i.0/24"; done

# Check if two CIDRs overlap (python one-liner)
python3 -c "
import ipaddress
a = ipaddress.ip_network('10.0.0.0/16')
b = ipaddress.ip_network('10.0.1.0/24')
print(a.overlaps(b))  # True — b is inside a
"

# Find which of your subnets an IP belongs to
python3 -c "
import ipaddress
ip = ipaddress.ip_address('10.0.1.137')
for cidr in ['10.0.0.0/24','10.0.1.0/24','10.0.2.0/24']:
    if ip in ipaddress.ip_network(cidr):
        print(f'{ip} is in {cidr}')
"

Troubleshooting IP Conflicts and Overlapping Ranges¶

Symptom: Intermittent Connectivity, ARP Flapping¶

Two hosts with the same IP. ARP table flips between MACs.

# Check for duplicate IPs
arping -D -I eth0 10.0.1.50
# If you get a reply, someone else has that IP

# Watch ARP table for flapping
ip neigh show | grep 10.0.1.50
# If the MAC keeps changing, you have a duplicate

Symptom: Route Goes to Wrong Destination¶

Overlapping CIDRs in the routing table. More specific route wins.

# Show routing table — look for overlapping entries
ip route show | sort -t/ -k2 -n

# Check which route a specific IP matches
ip route get 10.0.1.50

Symptom: VPN Tunnel Up But No Traffic¶

Local subnet and remote subnet overlap. Traffic stays local instead of going through the tunnel.

# Compare local subnets vs VPN remote subnets
ip route show          # local routes
ip route show table 220  # strongSwan routes (or check your VPN's table)

# If both show 10.0.1.0/24, traffic for 10.0.1.x stays local
# Fix: re-IP one side or use NAT translation

One-liner: To quickly check CIDR math in your head: each bit in the prefix doubles the number of subnets and halves the hosts. /24 = 254 hosts, /25 = 126, /26 = 62, /27 = 30, /28 = 14.