BGP EVPN / VXLAN — Street-Level Ops¶

Quick Diagnosis Commands¶

# NX-OS: Overall fabric health
show nve peers                          # VTEP-to-VTEP tunnel state
show nve vni                            # VNI table — status and peer count
show bgp l2vpn evpn summary             # BGP EVPN session status
show mac address-table dynamic          # L2 MAC forwarding table
show ip route vrf TENANT_A             # L3 routing in tenant VRF

# Arista EOS: Overall fabric health
show vxlan vtep                         # VTEP peers
show vxlan address-table                # MAC-to-VTEP mappings
show bgp evpn summary                   # EVPN BGP sessions
show bgp evpn route-type mac-ip         # Type 2 routes
show bgp evpn route-type imet           # Type 3 routes (VTEP membership)
show interfaces Vxlan1                  # VTEP interface counters

# Underlay reachability (always check this first)
ping 10.0.0.12 source loopback1         # Can local VTEP reach remote VTEP?
traceroute 10.0.0.12 source loopback1   # Trace the underlay path
show ip route 10.0.0.12                 # How many ECMP paths exist?

Remember: EVPN route types by number — 2-3-5: Type 2 = MAC/IP (host), Type 3 = IMET (BUM flooding membership), Type 5 = IP prefix (inter-subnet routing). If you remember nothing else, remember "2 is who, 3 is where to flood, 5 is how to route."

Gotcha: VTEP Cannot Reach Another VTEP¶

Symptom: show nve peers shows a peer in Init or Down state. VM traffic between two leaves is dropped.

Rule: VTEP reachability is an underlay problem. Diagnose underlay routing before touching EVPN config.

Debug clue: If ping 10.0.0.12 source loopback1 df-bit packet-size 1550 fails but packet-size 1400 works, you have an MTU black hole. VXLAN adds 50 bytes of overhead (8 bytes VXLAN header + 8 bytes UDP + 20 bytes outer IP + 14 bytes outer Ethernet). Every link in the underlay needs MTU 9216 or higher to carry standard 1500-byte inner frames without fragmentation.

# Step 1: Check if the underlay route exists
show ip route 10.0.0.12   # remote VTEP loopback

# Step 2: Ping from loopback (source matters — BGP session is loopback-to-loopback)
ping 10.0.0.12 source loopback1 count 5

# Step 3: Check OSPF/BGP underlay
show ip ospf neighbor
show bgp ipv4 unicast summary

# Step 4: Verify MTU — VXLAN adds ~50 bytes overhead
ping 10.0.0.12 source loopback1 df-bit packet-size 1550
# If this fails but 1400 works, you have an MTU problem

# Step 5: Fix MTU on physical links
interface Ethernet1/1
  mtu 9216
  ip mtu 9000

# Step 6: Check NVE interface is up
show interface nve1
# NVE1 is up  ← required
# NVE1 is down  ← interface is admin-down or source loopback is missing route

Gotcha: MAC Mobility — Host Moved, Traffic Still Goes to Old VTEP¶

Symptom: After a VM live migration, traffic to the VM is still forwarded to the old leaf. Or you see the VM appearing on two VTEPs simultaneously.

Rule: EVPN handles MAC mobility with a sequence number. If the sequence number is not incrementing, the old VTEP's Type 2 route is winning.

# Check MAC mobility on NX-OS
show bgp l2vpn evpn route-type 2 aabb.cc00.1234
# Look for the "MAC Mobility" extended community and sequence number
# Extcommunity: RT:65001:10100 ENCAP:8 MAC Mobility Sequence:1

# The VTEP that last saw the MAC should advertise seq:N+1
# If old VTEP is still advertising with higher sequence, check:
# 1. Is the host actually gone from the old leaf? Check MAC table:
show mac address-table address aabb.cc00.1234

# 2. Clear the stale MAC entry on the old VTEP
clear mac address-table dynamic address aabb.cc00.1234

# 3. Check the MAC mobility timer
show running-config | include mac-move
# mac move retry-limit 1  ← how fast old VTEP gives up after mobility

Gotcha: Symmetric IRB — Host Cannot Route to Another Subnet¶

Symptom: Hosts in VLAN 100 can ping the local gateway but cannot reach hosts in VLAN 200 on another leaf.

Rule: Symmetric IRB requires the L3 VNI to be configured on every VTEP that participates in the VRF, not just the VTEPs serving specific VLANs.

# Verify L3 VNI is present in NVE on both VTEPs
show nve vni
# VNI      Mode  BD    IngressRep  cp-group    state up-time
# 10001    L3    N/A   N/A         N/A         Up    ...   ← L3 VNI must be present

# Verify VRF is associated with L3 VNI
show vrf TENANT_A detail | include L3
# VNI: 10001  ← must match

# Verify Type 5 IP prefix routes are being exchanged
show bgp l2vpn evpn route-type 5
# Look for 192.168.200.0/24 advertised by the remote VTEP

# Verify the anycast gateway IP is configured on the local SVI
show interface Vlan100
# IP address: 192.168.100.1/24  ← anycast gateway
# Fabric forwarding mode: anycast-gateway  ← NX-OS requirement

# Arista: verify anycast gateway MAC
show ip virtual-router mac-address
# Virtual MAC: aabb.cc00.fffe  ← must be consistent across all leaves

Pattern: Verify EVPN Type 2 Route for a Specific Host¶

Use this when a specific host is unreachable and you want to confirm the fabric knows where it is:

# On the leaf where the host is connected:
show mac address-table address aabb.cc00.5678
# VLAN   MAC Address    Type    Age    Port
# 100    aabb.cc00.5678 dynamic 0:00   Ethernet1/3  ← host connected here

# Check if this leaf is advertising a Type 2 route:
show bgp l2vpn evpn route-type 2 aabb.cc00.5678
# If no route, EVPN is not advertising the MAC — check:
show nve vni 10100   # Is VNI 10100 in Up state?

# On a remote leaf, confirm the route is received:
show bgp l2vpn evpn route-type 2 aabb.cc00.5678
# Should show the route with Next Hop = 10.0.0.11 (local leaf's VTEP IP)

# Confirm the VXLAN address table on the remote leaf:
show vxlan address-table    # Arista
show mac address-table vni 10100  # NX-OS equivalent

Scenario: New VTEP Not Forming EVPN Sessions with Spines¶

You've added a new leaf. Underlay is up (OSPF/BGP sees the loopback) but EVPN BGP sessions won't come up.

# Check BGP EVPN session status
show bgp l2vpn evpn summary
# Neighbor      AS    State
# 10.0.0.1     65000  Idle   ← not connecting

# Verify BGP is configured for L2VPN EVPN AFI/SAFI
show running-config | section bgp
# router bgp 65011
#   neighbor 10.0.0.1 remote-as 65000
#   address-family l2vpn evpn
#     send-community both    ← REQUIRED for EVPN extended communities
#     route-reflector-client  ← on spine only

# NX-OS: enable EVPN feature
feature bgp
feature nv overlay
feature vn-segment-vlan-based
nv overlay evpn     ← required global command on NX-OS

# Gotcha: 'send-community both' is REQUIRED for EVPN to work
# Without it, route-targets (extended communities) are stripped
# and the remote leaf never imports the route into its VRF

# Verify the spine is configured as route reflector
# On spine, check the peer policy for the new leaf:
show bgp l2vpn evpn neighbors 10.0.0.11 policy

# Check if extended communities are being sent (required for RT matching)
show bgp l2vpn evpn neighbors 10.0.0.11 advertised-routes
# If empty, community propagation is broken

Scenario: BUM Traffic Flooding — VXLAN Ingress Replication Misconfigured¶

Symptom: High broadcast traffic, ARP storms visible across the fabric.

# Check how BUM traffic is being handled on NX-OS
show nve vni
# VNI      Mode  IngressRep
# 10100    L2    static      ← static means multicast or static peer list, NOT BGP-driven
# 10100    L2    bgp         ← correct for EVPN ingress replication

# Fix: enable BGP-driven ingress replication
interface nve1
  member vni 10100
    ingress-replication protocol bgp
    no mcast-group   # remove if multicast was configured

# Verify the replication list is being built from Type 3 routes
show bgp l2vpn evpn route-type 3
# Each VTEP advertising Type 3 for VNI 10100 should appear in replication list

# On Arista, check flood list per VNI
show vxlan flood vtep vni 10100
# This should list all remote VTEPs that have hosts in VNI 10100

Gotcha: ingress-replication protocol bgp and mcast-group are mutually exclusive per VNI. If you see both configured, remove one. Having both causes undefined BUM forwarding behavior that is extremely difficult to debug.

Emergency: Complete Loss of East-West Traffic in Fabric¶

All inter-leaf VM traffic is black-holed. North-south to external routers is fine.

War story: A fabric lost all east-west traffic after a maintenance window. Root cause: an engineer changed the loopback IP on one spine without updating the BGP neighbor config on leaves. Underlay OSPF reconverged, but EVPN sessions used the old loopback as the BGP neighbor address. The NVE interface went down on every leaf that peered with that spine. Always change loopback IPs as a coordinated pair: IGP advertisement + BGP neighbor config.

# Step 1: Check NVE interface state
show interface nve1
# If NVE1 is down — source loopback route is gone or interface was shut

# Step 2: Check underlay
show ip route | grep "L, 10.0.0."   # local loopback routes
# If loopback route is missing, NVE cannot form tunnels

# Step 3: Check that loopback is advertised in underlay IGP
show ip ospf interface loopback0
# or:
show bgp ipv4 unicast neighbors <spine1> advertised-routes | grep 10.0.0.11

# Step 4: Check for hardware programming issues on NX-OS
show system internal pixm info vni 10100
show forwarding route vxlan vni 10100

# Step 5: Try bouncing NVE interface (impacts all VXLAN tunnels — get change approval)
interface nve1
  shutdown
  no shutdown

# Step 6: Clear EVPN routes and let BGP reconverge
clear bgp l2vpn evpn * soft

Useful One-Liners¶

# Show all EVPN MAC/IP routes for a specific VNI on NX-OS
show bgp l2vpn evpn | grep -A 3 "10100"

# Find which VTEP is advertising a specific MAC
show bgp l2vpn evpn route-type 2 | grep -B 2 "aabb.cc00.1234"

# Count MAC/IP routes per VTEP (detect imbalance)
show bgp l2vpn evpn route-type 2 | grep "^BGP" | awk '{print $NF}' | sort | uniq -c

# Check ECMP path count for remote VTEP
show ip route 10.0.0.12 | grep "via"

# Verify VXLAN packet counters are incrementing
show interface nve1 | grep -E "packets|bytes"

# Check MTU end-to-end (inner host perspective — test from a VM)
ping -c 3 -M do -s 1472 192.168.200.10  # 1472 + 28 = 1500 inner frame

# Find all VLANs/VNIs mapped on a leaf
show vlan brief | awk '$4 != "" {print $1, $4}'  # NX-OS VLAN to VNI

# Arista: check VTEP-to-VTEP tunnel status with counters
show vxlan counters