BGP EVPN / VXLAN Footguns¶
Misconfigurations that cause silent packet drops, routing black holes, or split-brain fabric states.
1. Mismatched VNI-to-VLAN Mapping Across Leaves¶
Leaf-A maps VLAN 100 to VNI 10100. Leaf-B maps VLAN 100 to VNI 20100. Traffic from a host on Leaf-A is VXLAN-encapsulated with VNI 10100. Leaf-B receives it but looks up VNI 10100 — which is either unmapped or maps to a different VLAN. Traffic is silently dropped. Both leaves look healthy individually.
Fix: Maintain a central VNI-to-VLAN mapping table (a spreadsheet, IPAM, or Netbox). Automate configuration generation from this single source of truth. Never configure VNI mappings ad-hoc on individual devices.
2. Physical MTU Not Accounting for VXLAN Overhead¶
You set physical interface MTU to 1500 everywhere. VXLAN adds ~50 bytes of overhead (outer Ethernet + IP + UDP + VXLAN header). The inner VM traffic with an MSS of 1460 creates VXLAN frames of ~1510 bytes — slightly over the physical MTU. Packets are silently dropped or fragmented. TCP sessions establish but stall on large transfers. ICMP (small packets) works fine.
Debug clue: The classic symptom of VXLAN MTU issues is "small packets work, large transfers stall." ICMP pings succeed (small), SSH sessions establish (small handshake), but SCP/HTTP transfers of large files hang. If you see TCP retransmissions only on segments near 1500 bytes, suspect MTU mismatch on the underlay.
Fix: Set physical interface MTU to at least 1600 (commonly 9216 for jumbo frames) on all fabric links. Verify with an oversized ping:
ping <remote-vtep> source loopback1 df-bit packet-size 1550
# Must succeed for VXLAN to work correctly
3. Spines Learning EVPN MAC/IP Routes¶
In a standard EVPN fabric, spines are only route reflectors — they reflect EVPN routes between leaves but do not process them for forwarding. If you misconfigure a spine as an EVPN-speaking VTEP (enable NVE, add VNI config), it starts advertising its own EVPN routes. Leaves send some traffic to the spine as if it were a VTEP, where it is black-holed.
Fix: Spines should have BGP configured as route reflectors only, with no NVE interface and no VNI configuration. Strictly separate the control plane role (RR) from the data plane role (VTEP).
4. Missing send-community both on EVPN BGP Neighbors¶
BGP EVPN relies heavily on extended communities: Route Targets (RT) for VPN membership, encapsulation type, MAC mobility sequence numbers, and Router MAC. If send-community both (or send-community extended) is missing from the BGP neighbor configuration, these extended communities are stripped. The receiving VTEP cannot import the routes because the RT doesn't match — it silently ignores all EVPN routes from that neighbor.
Fix: Always configure send-community both (standard + extended) on all EVPN BGP sessions. Verify with show bgp l2vpn evpn neighbors <ip> advertised-routes — if you see routes without extended community attributes, this is the problem.
5. Anycast Gateway MAC Inconsistency Across Leaves¶
Each leaf must present the same virtual MAC address for the anycast gateway. If one leaf is configured with a different MAC (copy-paste error or manual misconfiguration), hosts that move to that leaf see a different gateway MAC and send traffic to the wrong destination until their ARP cache expires (up to several minutes).
Fix: The anycast gateway MAC must be a single globally-configured value, identical on every leaf. On Arista, ip virtual-router mac-address is set once; on NX-OS, fabric forwarding anycast-gateway-mac is global. Audit with:
6. Asymmetric IRB Without All VLANs on Every VTEP¶
Asymmetric IRB requires the ingress VTEP to perform both bridging and routing. This means the ingress VTEP must have the destination VLAN and its SVI configured locally — even if no hosts in that VLAN are directly connected. In a large fabric with hundreds of VLANs, this means every leaf must carry every VLAN, regardless of whether it has hosts in those VLANs.
Fix: For large deployments, use symmetric IRB instead. Symmetric IRB only requires each leaf to know about the VLANs it directly serves. If you're already on asymmetric IRB and hitting scale problems, the migration to symmetric IRB requires adding the L3 VNI (VRF VNI) to every participating VTEP.
7. Not Configuring ECMP on Spine Uplinks¶
You have two spine switches and two uplinks per leaf, but forget to enable ECMP (maximum-paths). All VTEP-to-VTEP traffic takes a single path (typically the lower-cost one). The fabric is effectively running at half capacity, and one spine failure causes traffic to reconverge through the remaining spine instead of being pre-distributed.
Fix: Configure maximum-paths 4 ecmp 4 (or appropriate value) in the BGP router configuration. Verify with:
8. Using Multicast BUM Handling Without PIM Configured in Underlay¶
You configure mcast-group 239.1.1.1 on NVE interfaces for BUM traffic handling. The VTEPs need to join the multicast group, which requires PIM to be running in the underlay. If PIM is not configured on the spine and leaf interfaces, multicast traffic is not forwarded. BUM traffic (ARP requests, broadcasts) is dropped. VMs cannot discover their gateway. The overlay appears completely broken.
Fix: Either configure PIM sparse-mode throughout the underlay (and a rendez-vous point), or switch to ingress replication (head-end replication) with EVPN Type 3 routes. For new deployments, ingress replication + EVPN is simpler and avoids the multicast dependency.
9. Forgetting nv overlay evpn (NX-OS Global Command)¶
On NX-OS, EVPN requires a global command in addition to the BGP configuration:
Without this, BGP EVPN sessions may establish but EVPN routes are not properly processed or installed. The symptom is BGP sessions showing Established but no MAC/IP routes in the EVPN table, and VTEPs not seeing each other's hosts.
Fix: Always verify the NX-OS global EVPN config:
show running-config | include "nv overlay\|feature nv"
# Should show:
# feature nv overlay
# nv overlay evpn
10. VTEP Source Loopback Not Advertised in Underlay IGP¶
The NVE interface uses a loopback as the VTEP source address. If this loopback is not redistributed or advertised into the underlay routing protocol (OSPF, ISIS, or BGP underlay), remote VTEPs cannot reach it. EVPN BGP sessions (which are loopback-to-loopback) fail to establish. No NVE tunnels form.
Fix: Verify the loopback is advertised:
# OSPF
show ip ospf interface loopback0 # Must show OSPF enabled on loopback
show ip ospf database self-originate # Must include the loopback prefix
# BGP underlay (Arista)
show bgp ipv4 unicast neighbors <spine> advertised-routes | grep "10.0.0.11"
The loopback must use a /32 prefix mask (not /24) so it is advertised as a host route, not summarized.
Gotcha: If the VTEP loopback is advertised as a /24 instead of a /32, route summarization in the underlay IGP may cause traffic to be sent to the wrong device. Each VTEP must be reachable at its exact loopback IP. Using /32 ensures the route is never summarized or aggregated away.