Skip to content

Solution: ARP Flux / Duplicate IP

Summary

A developer's test VM was assigned the static IP 10.30.1.100, which is already in use by the production server app-web-05. Both hosts respond to ARP requests for this IP, causing ARP tables on other devices to flip-flop between the two MACs. When a client's ARP cache points to the rogue VM, traffic goes to the wrong host and the production server becomes unreachable.

Senior Workflow

Step 1: Confirm the ARP flapping

# From any client on the same subnet:
arp -a | grep 10.30.1.100
# Run multiple times -- watch for MAC changing

# Or use arping for definitive proof:
arping -I eth0 10.30.1.100
# If TWO different MACs respond, you have a duplicate IP

Step 2: Identify the legitimate and rogue MAC

# Known production server MAC (from CMDB / server team):
# app-web-05: 00:1a:2b:3c:4d:5e

# Unknown/rogue MAC:
# 00:ff:aa:bb:cc:11

Step 3: Locate the rogue device on the network

# On the access switch:
sw-access-01# show mac address-table address 00ff.aabb.cc11
Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
30      00ff.aabb.cc11    DYNAMIC     Gi0/22

# Gi0/22 is where the rogue device is connected

Step 4: Check DHCP lease table

# On the DHCP server:
cat /var/lib/dhcp/dhcpd.leases | grep -A 6 "10.30.1.100"

# OR if using ISC DHCP:
dhcp-lease-list | grep 10.30.1.100

This may show two leases, or the rogue device may have a static assignment outside DHCP.

Step 5: Remove the duplicate IP

# Option A: Contact the developer, remove the static IP from the test VM
# Option B: Shut the switch port administratively
sw-access-01(config)# interface Gi0/22
sw-access-01(config-if)# shutdown

Step 6: Clear ARP caches

# On affected clients:
ip neigh flush dev eth0

# On the router/gateway:
arp -d 10.30.1.100

Step 7: Verify connectivity is restored

arping -I eth0 -c 5 10.30.1.100
# Should now see only one MAC: 00:1a:2b:3c:4d:5e

ping 10.30.1.100
# Should be consistent with no intermittent drops

Step 8: Implement preventive measures

# Enable DHCP snooping on the switch
sw-access-01(config)# ip dhcp snooping
sw-access-01(config)# ip dhcp snooping vlan 30

# Enable Dynamic ARP Inspection
sw-access-01(config)# ip arp inspection vlan 30

# Mark uplink as trusted
sw-access-01(config-if)# ip dhcp snooping trust
sw-access-01(config-if)# ip arp inspection trust

Common Pitfalls

  • Assuming the problem is the server: The production server is the victim, not the cause. Always check for duplicate IPs before diving into server diagnostics.
  • Not using arping for detection: ping alone cannot detect duplicates. arping sends ARP requests and shows which MACs respond.
  • Clearing ARP on only one client: All devices on the subnet may have cached the wrong MAC. The gateway ARP cache is especially important.
  • Shutting down the wrong port: Verify the MAC-to-port mapping before shutting down switch ports.
  • Not implementing DHCP snooping/DAI: Without these controls, any device can claim any IP. This will happen again.
  • Ignoring IPAM: Manual IP assignment ("picking a free IP") is the process failure. Implement proper IP address management.