Solution: Jumbo Frames Enabled But Some Paths Failing¶

Triage¶

Confirm server MTU settings:
```
ip link show eth0 | grep mtu
```
Expect MTU 9000 on both source and destination.
Test path MTU with large ping from source to destination:
```
ping -s 8972 -M do -c 3 10.2.2.100
```
8972 + 20 (IP header) + 8 (ICMP header) = 9000. If this fails, something in the path does not support MTU 9000.

Test hop by hop to isolate the failure point:

ping -s 8972 -M do -c 3 10.1.1.1    # default gateway (ToR switch)
ping -s 8972 -M do -c 3 10.0.0.1    # aggregation switch
ping -s 8972 -M do -c 3 10.0.0.2    # next hop
ping -s 8972 -M do -c 3 10.2.2.100  # destination

The first hop that fails identifies the MTU bottleneck.

Binary search for actual path MTU:

ping -s 1472 -M do -c 1 10.2.2.100   # 1500 MTU test - should work
ping -s 4000 -M do -c 1 10.2.2.100   # midpoint test

Check the suspected switch's MTU configuration:
```
show system mtu
show interface mtu
```

Root Cause¶

Jumbo frames (MTU 9000) were enabled on all servers, top-of-rack switches, and most aggregation switches. However, one aggregation switch (or its uplink ports) in the cross-rack path has a maximum MTU of 1500 (default) or a hardware limitation preventing jumbo frame support.

When a server sends a 9000-byte frame with DF (Don't Fragment) set, the intermediate switch cannot forward it because the egress port MTU is only 1500. The switch should generate an ICMP "Fragmentation Needed" message, but some switches silently drop oversized frames without generating ICMP, causing a "black hole" for large packets.

Small packets (< 1500 bytes) traverse the path without issue, which is why ping, SSH, and small transfers work fine. Large transfers (NFS, iSCSI, bulk replication) fail because they use the negotiated MSS of 8960.

Fix¶

Option A -- Enable Jumbo on the Missing Switch (Preferred):

Switch(config)# system mtu jumbo 9000
Switch(config)# exit
Switch# reload    ! Some switches require reload for system MTU change

Verify the switch hardware supports 9000 MTU first.

Option B -- If Switch Cannot Support Jumbo: Reduce MTU to 1500 on all devices in paths that traverse this switch, or re-architect the network to avoid routing jumbo traffic through the non-compliant switch.

Option C -- Partial Fix with Clamped MSS: If only TCP traffic is affected, clamp TCP MSS at the router:

interface GigabitEthernet0/0
  ip tcp adjust-mss 1460

This forces TCP to use smaller segments but does not fix non-TCP traffic.

After fixing, verify end-to-end:

ping -s 8972 -M do -c 5 10.2.2.100

Rollback / Safety¶

Changing system MTU on many switch platforms requires a reload -- schedule in a maintenance window.
Verify all ports on the switch can handle 9000 MTU (some have per-port limits).
If rolling back, reduce server MTU to 1500 to restore full connectivity.
Test after change with both large pings and actual application traffic.

Common Traps¶

Assuming that enabling jumbo frames on servers and ToR switches is sufficient -- every device in the Layer 2 path must support it.
Forgetting uplink/trunk ports -- access ports may be configured for jumbo but uplinks left at default 1500.
Switch "system mtu" vs. "system mtu jumbo" -- some platforms have separate settings for routed and switched MTU.
ICMP black hole: if the switch silently drops oversized frames without sending ICMP "frag needed," PMTUD never adjusts and TCP retransmits endlessly.
Not testing after the fix with actual application workload, only with ping.