Skip to content

Portal | Level: L2: Operations | Topics: Cloud Deep Dive | Domain: Cloud

Runbook: VPC Subnet IP Exhaustion

Symptoms

  • Pods stuck in Pending with error: failed to allocate for range: no available addresses
  • aws-node (VPC CNI) logs: InsufficientFreeAddressesInSubnet
  • New nodes can't join the cluster
  • Existing pods are fine, only new pods fail

Fast Triage (under 2 minutes)

# Check for pending pods
kubectl get pods -A --field-selector=status.phase=Pending

# Check node events
kubectl get events -A --sort-by=.lastTimestamp | grep -i "address\|ip\|subnet"

# Check VPC CNI logs
kubectl logs -n kube-system -l k8s-app=aws-node --tail=20

Causes and Fixes

1. Subnet IPs Exhausted (EKS)

# Check available IPs per subnet
aws ec2 describe-subnets --subnet-ids subnet-aaa subnet-bbb subnet-ccc \
  --query 'Subnets[].{SubnetId:SubnetId, AZ:AvailabilityZone, AvailableIPs:AvailableIpAddressCount, CIDR:CidrBlock}'

# Check IPs allocated per node
kubectl get nodes -o json | jq '.items[] | {
  name: .metadata.name,
  pods: .status.allocatable.pods,
  zone: .metadata.labels["topology.kubernetes.io/zone"]
}'

Immediate fix: Enable prefix delegation (16 IPs per slot instead of 1). 

kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true
kubectl set env daemonset aws-node -n kube-system WARM_PREFIX_TARGET=1

Medium-term: Add secondary CIDR block. 

aws ec2 associate-vpc-cidr-block --vpc-id vpc-xxx --cidr-block 100.64.0.0/16
# Then create new subnets in the secondary CIDR

2. ENI Limits Per Instance Type

[!NOTE] Each EC2 instance type has fixed, non-configurable limits on ENIs and IPs per ENI. These limits are hard ceilings — no amount of configuration can raise them. The only options are larger instance types or enabling prefix delegation. Check the AWS docs for your specific instance type before capacity planning.

Each instance type has a max number of ENIs and IPs per ENI: - m5.large: 3 ENIs × 10 IPs = 29 pod IPs - m5.xlarge: 4 ENIs × 15 IPs = 58 pod IPs - m5.2xlarge: 4 ENIs × 15 IPs = 58 pod IPs

Fix: Use larger instance types or enable prefix delegation.

3. WARM_IP_TARGET Too Aggressive

# Check current settings
kubectl get daemonset aws-node -n kube-system -o json | \
  jq '.spec.template.spec.containers[0].env'

Fix: Reduce warm IP settings. 

kubectl set env daemonset aws-node -n kube-system \
  WARM_IP_TARGET=2 \
  MINIMUM_IP_TARGET=5

Verification

# Check pod scheduling resumes
kubectl get pods -A --field-selector=status.phase=Pending

# Check VPC CNI is healthy
kubectl get pods -n kube-system -l k8s-app=aws-node

# Check available IPs increased
aws ec2 describe-subnets --subnet-ids subnet-aaa \
  --query 'Subnets[].AvailableIpAddressCount'

Prevention

  • Monitor awscni_assigned_ip_addresses and awscni_total_ip_addresses
  • Alert when available subnet IPs < 20% of total
  • Use /19 or larger subnets for pod networking
  • Plan CIDR ranges for growth (can't shrink subnets)
  • Enable prefix delegation from the start on new clusters

Wiki Navigation

Related Content

  • AWS CloudWatch (Topic Pack, L2) — Cloud Deep Dive
  • AWS Devops Flashcards (CLI) (flashcard_deck, L1) — Cloud Deep Dive
  • AWS EC2 (Topic Pack, L1) — Cloud Deep Dive
  • AWS ECS (Topic Pack, L2) — Cloud Deep Dive
  • AWS General Flashcards (CLI) (flashcard_deck, L1) — Cloud Deep Dive
  • AWS IAM (Topic Pack, L1) — Cloud Deep Dive
  • AWS Lambda (Topic Pack, L2) — Cloud Deep Dive
  • AWS Networking (Topic Pack, L1) — Cloud Deep Dive
  • AWS Route 53 (Topic Pack, L2) — Cloud Deep Dive
  • AWS S3 Deep Dive (Topic Pack, L1) — Cloud Deep Dive