--- tags: - networking - l1 - flashcard-deck - mellanox-switches --- [Portal](../../../../library/portal/index.md) | **Level:** [L1: Foundations](../../../../library/portal/levels.md) | **Topics:** [Mellanox Switches](../../../../library/portal/topics.md) | **Domain:** Networking id category difficulty tags question answer source_path mellanox-switches/a1b2c3d4e5f6 mellanox-switches easy mellanox, history What company originally made Mellanox switches, and who owns them now? "Mellanox Technologies designed the switches. NVIDIA acquired Mellanox in 2020. The switch line continues under the ""NVIDIA Networking"" brand.\n\nFun fact: Mellanox acquired by NVIDIA (2020, $6.9B). Powers AI/HPC networking.\n\nNumber anchor: NVIDIA paid $6.9 billion for Mellanox in April 2020 — the largest acquisition in NVIDIA\'s history at the time." training/library/topics/mellanox-switches/primer.md mellanox-switches/b2c3d4e5f6a7 mellanox-switches easy mellanox, onyx, cli What is the current name of the Mellanox switch operating system? Onyx (formerly MLNX-OS). The name MLNX-OS still appears in firmware filenames.\n\nFun fact: Mellanox acquired by NVIDIA (2020, $6.9B). Powers AI/HPC networking.\n\nName origin: Onyx is a dark gemstone, fitting for networking firmware. The name change from MLNX-OS happened around 2018. training/library/topics/mellanox-switches/primer.md mellanox-switches/c3d4e5f6a7b8 mellanox-switches easy mellanox, onyx, cli What is the Onyx command to save the running configuration to persistent storage? "configuration write — not ""write memory"" or ""copy running-config startup-config"" as in Cisco IOS.\n\nRemember: ONYX = Mellanox switch OS. Cumulus = Linux-based NOS. Both on Spectrum ASICs.\n\nGotcha: Forgetting `configuration write` before reboot loses all changes. Unlike Cisco\'s `wr mem`, the command is `configuration write` — muscle memory from other vendors will fail you." training/library/topics/mellanox-switches/primer.md mellanox-switches/d4e5f6a7b8c9 mellanox-switches easy mellanox, spectrum, asic Name the four generations of the Spectrum ASIC family in order. Spectrum (SN2000), Spectrum-2 (SN3000), Spectrum-3 (SN4000), Spectrum-4 (SN5000). Each roughly doubles bandwidth density.\n\nRemember: Spectrum ASICs: -1(100G), -2(200G), -3(400G), -4(800G). Each gen ~doubles.\n\nTimeline: Spectrum (2016), Spectrum-2 (2019), Spectrum-3 (2020), Spectrum-4 (2022). Each generation roughly doubles bandwidth density. training/library/topics/mellanox-switches/primer.md mellanox-switches/e5f6a7b8c9d0 mellanox-switches easy mellanox, spectrum What maximum port speed does the Spectrum-3 (SN4000) ASIC support? 400G per port.\n\nRemember: Spectrum ASICs: -1(100G), -2(200G), -3(400G), -4(800G). Each gen ~doubles.\n\nUnder the hood: 400G is achieved using 8 lanes of 50G PAM4 signaling. Spectrum-4 reaches 800G with 8 lanes of 100G PAM4.\n\nAnalogy: Think of port speed generations like highway lanes — each generation widens the highway while also making each lane faster. training/library/topics/mellanox-switches/primer.md mellanox-switches/f6a7b8c9d0e1 mellanox-switches easy mellanox, wjh What does WJH stand for on Mellanox switches? What Just Happened — a hardware-level feature that logs every packet dropped by the ASIC with the drop reason.\n\nFun fact: Mellanox acquired by NVIDIA (2020, $6.9B). Powers AI/HPC networking.\n\nUnder the hood: WJH uses dedicated ASIC hardware counters to log drops with zero performance impact. Traditional packet capture can\'t catch packets the ASIC drops.\n\nDebug clue: `show what-just-happened forwarding` shows the most recent drops with reason codes — faster than tcpdump for ASIC-level drops. training/library/topics/mellanox-switches/street_ops.md mellanox-switches/a7b8c9d0e1f2 mellanox-switches easy mellanox, ufm What is UFM in the Mellanox ecosystem? Unified Fabric Manager — a centralized management platform for multi-switch fabrics providing topology discovery, health monitoring, firmware orchestration, and telemetry.\n\nFun fact: Mellanox acquired by NVIDIA (2020, $6.9B). Powers AI/HPC networking.\n\nAnalogy: UFM is to Mellanox switches what vCenter is to VMware ESXi hosts — a single pane of glass for multi-device management. training/library/topics/mellanox-switches/primer.md mellanox-switches/b8c9d0e1f2a3 mellanox-switches easy mellanox, mlag What does MLAG stand for on Mellanox switches? Multi-Chassis Link Aggregation — allows two physical switches to present as one logical switch for link aggregation to downstream devices.\n\nFun fact: Mellanox acquired by NVIDIA (2020, $6.9B). Powers AI/HPC networking.\n\nAnalogy: MLAG is like two switches pretending to be one for downstream devices. The downstream device sees a single LAG partner.\n\nGotcha: MLAG requires an ISL (Inter-Switch Link) between peers. If the ISL fails, split-brain occurs. training/library/topics/mellanox-switches/street_ops.md mellanox-switches/c9d0e1f2a3b4 mellanox-switches easy mellanox, onyx, show What Onyx command shows all LLDP neighbor information? show lldp interfaces\n\nRemember: ONYX = Mellanox switch OS. Cumulus = Linux-based NOS. Both on Spectrum ASICs.\n\nRemember: LLDP = Link Layer Discovery Protocol. It\'s vendor-neutral, unlike Cisco\'s CDP. Both discover neighbors but LLDP works across vendors. training/library/topics/mellanox-switches/primer.md mellanox-switches/d0e1f2a3b4c5 mellanox-switches medium mellanox, roce, rdma What is RoCEv2 and why does it need lossless Ethernet? RDMA over Converged Ethernet v2 — it carries RDMA traffic over standard Ethernet using UDP. Because RDMA assumes a reliable transport, any packet drop causes expensive retransmissions, so PFC is used to make the Ethernet fabric lossless.\n\nRemember: RDMA = bypass CPU for low latency. RoCE = RDMA over Ethernet. v2 uses UDP.\n\nName origin: RDMA = Remote Direct Memory Access. RoCE = RDMA over Converged Ethernet. v2 added UDP encapsulation for L3 routing. training/library/topics/mellanox-switches/street_ops.md mellanox-switches/e1f2a3b4c5d6 mellanox-switches medium mellanox, pfc, roce What is Priority Flow Control (PFC) and what priority is typically used for RoCE? PFC allows pausing traffic on a specific 802.1p priority class without stopping other classes. Priority 3 is the conventional choice for RoCE traffic.\n\nRemember: RDMA = bypass CPU for low latency. RoCE = RDMA over Ethernet. v2 uses UDP.\n\nAnalogy: PFC is like a traffic light that only stops one lane (priority class) while letting others flow freely. training/library/topics/mellanox-switches/street_ops.md mellanox-switches/f2a3b4c5d6e7 mellanox-switches medium mellanox, ecn What role does ECN play alongside PFC in a lossless fabric? "ECN (Explicit Congestion Notification) marks packets when buffer utilization crosses a threshold, signaling senders to reduce rate. This prevents buffers from filling to the point where PFC must pause traffic, avoiding PFC storms.\n\nRemember: DCB = lossless Ethernet. PFC=flow control, ECN=congestion notification. For RoCE.\n\nRemember: ""ECN = early warning, PFC = emergency brake."" ECN tells senders to slow down before buffers fill. PFC stops traffic when buffers are about to overflow." training/library/topics/mellanox-switches/street_ops.md mellanox-switches/a3b4c5d6e7f8 mellanox-switches medium mellanox, dcbx What is DCBX and why does trust mode matter for RoCE? Data Center Bridging Capability Exchange — negotiates PFC, ETS, and priority settings between switch and NIC. Trust mode (L2 CoS vs DSCP) must match on both sides, or RDMA traffic lands in the wrong queue and is not protected by PFC.\n\nRemember: RDMA = bypass CPU for low latency. RoCE = RDMA over Ethernet. v2 uses UDP.\n\nName origin: DCBX = Data Center Bridging Capability eXchange. It auto-negotiates lossless Ethernet parameters between switch and NIC. training/library/topics/mellanox-switches/footguns.md mellanox-switches/b4c5d6e7f8a9 mellanox-switches medium mellanox, firmware Describe the Onyx firmware upgrade workflow (key commands in order). 1) image fetch scp://... to download image
2) show images to verify
3) image install <img> to install to next partition
4) image boot next to set boot partition
5) configuration write to save config
6) reload\n\nRemember: ONYX = Mellanox switch OS. Cumulus = Linux-based NOS. Both on Spectrum ASICs.\n\nGotcha: Always verify the new firmware version supports your optics and ASIC generation before upgrading. Incompatible firmware can brick ports. training/library/topics/mellanox-switches/street_ops.md mellanox-switches/c5d6e7f8a9b0 mellanox-switches medium mellanox, firmware How does Onyx support firmware rollback? Onyx has dual boot partitions. If a new firmware fails, you can select the previous partition at boot to roll back to the last known-good version.\n\nRemember: ONYX = Mellanox switch OS. Cumulus = Linux-based NOS. Both on Spectrum ASICs.\n\nUnder the hood: Onyx uses an A/B partition scheme — firmware installs to the inactive partition. If boot fails, the bootloader falls back to the previous partition. training/library/topics/mellanox-switches/street_ops.md mellanox-switches/d6e7f8a9b0c1 mellanox-switches medium mellanox, mlag, isl What is an ISL in MLAG and what happens if it fails? The Inter-Switch Link connects MLAG peer switches. If the ISL fails, both switches become primary (split-brain), causing duplicate packets and MAC flapping downstream.\n\nRemember: MLAG = two switches as one. Like LACP but across a switch pair.\n\nDebug clue: If MLAG peers disagree on state, check `show mlag` and compare peer status. ISL failure is the #1 cause of MLAG split-brain. training/library/topics/mellanox-switches/footguns.md mellanox-switches/e7f8a9b0c1d2 mellanox-switches medium mellanox, wjh, drops Name three common WJH drop reasons and what they indicate. BUFFER_CONGESTION: traffic burst exceeded buffer capacity. INGRESS_ACL: packet matched a deny ACL. BLACKHOLE_ROUTE: packet matched a null or missing route.\n\nRemember: Mellanox/NVIDIA = high-performance networking. InfiniBand for HPC, Spectrum for Ethernet.\n\nUnder the hood: Spectrum ASICs provide wire-speed L2/L3 with programmable pipeline.\n\nDebug clue: BUFFER_CONGESTION + no ECN marks = ECN is not configured. INGRESS_ACL drops = check `show access-list`. BLACKHOLE_ROUTE = check `show ip route`. training/library/topics/mellanox-switches/street_ops.md mellanox-switches/f8a9b0c1d2e3 mellanox-switches medium mellanox, dom, optics What DOM readings should you monitor on Mellanox switch transceivers? "Rx power (below -10 dBm is concerning), temperature (above 70C is a warning), and Tx bias current (increasing bias indicates optic degradation). Access via show interfaces ethernet 1/1 transceiver diagnostics.\n\nFun fact: Mellanox acquired by NVIDIA (2020, $6.9B). Powers AI/HPC networking.\n\nRemember: ""Rx power is king."" Below -10 dBm means dirty fiber, bad splice, or failing optic. Above -1 dBm means the optic is overdriving the receiver." training/library/topics/mellanox-switches/street_ops.md mellanox-switches/a9b0c1d2e3f4 mellanox-switches medium mellanox, config, backup How do you back up and restore configuration on an Onyx switch? Backup: configuration upload scp://user@server/path.cfg
Restore: configuration fetch scp://... then configuration switch-to fetched-config.cfg then configuration write\n\nRemember: ONYX = Mellanox switch OS. Cumulus = Linux-based NOS. Both on Spectrum ASICs.\n\nGotcha: Restoring a config from a different firmware version may fail silently. Always upgrade firmware first, then restore config. training/library/topics/mellanox-switches/street_ops.md mellanox-switches/b0c1d2e3f4a5 mellanox-switches medium mellanox, buffers How does the Spectrum shared buffer architecture differ from per-port buffers? Spectrum uses a shared memory pool that any port can draw from, rather than fixed per-port allocations. This is better for bursty traffic but means one congested port can consume buffer space needed by other ports.\n\nRemember: Spectrum ASICs: -1(100G), -2(200G), -3(400G), -4(800G). Each gen ~doubles.\n\nGotcha: One elephant flow can consume shared buffers meant for other ports. Use buffer management profiles to set per-port limits. training/library/topics/mellanox-switches/footguns.md mellanox-switches/c1d2e3f4a5b6 mellanox-switches medium mellanox, breakout What is port breakout on a Mellanox switch and what is the oversubscription risk? Breaking one high-speed port into multiple lower-speed ports (e.g., 1x100G into 4x25G). Total bandwidth does not increase — the existing bandwidth is split. With full breakout, effective oversubscription may increase.\n\nFun fact: Mellanox acquired by NVIDIA (2020, $6.9B). Powers AI/HPC networking.\n\nExample: Breaking 1x100G into 4x25G is common for connecting 25G servers. Total bandwidth stays 100G — you\'re splitting, not multiplying. training/library/topics/mellanox-switches/footguns.md mellanox-switches/d2e3f4a5b6c7 mellanox-switches medium mellanox, syslog How do you configure syslog forwarding on an Onyx switch? logging host 10.0.0.100 port 514 and logging level info. Feed into a centralized logging system (ELK, Splunk, Loki).\n\nRemember: ONYX = Mellanox switch OS. Cumulus = Linux-based NOS. Both on Spectrum ASICs.\n\nGotcha: Default syslog level may be too verbose for production. Start with `logging level warning` and increase if needed. training/library/topics/mellanox-switches/street_ops.md mellanox-switches/e3f4a5b6c7d8 mellanox-switches medium mellanox, pfc, watchdog What is the PFC watchdog and why should you enable it? PFC watchdog detects stuck PFC states (where a port is paused indefinitely). When detected, it disables PFC on that port to prevent a PFC storm from cascading across the fabric. Enable with priority-flow-control watchdog enable.\n\nRemember: DCB = lossless Ethernet. PFC=flow control, ECN=congestion notification. For RoCE.\n\nWar story: Without PFC watchdog, a single misbehaving NIC can cascade PFC pauses across an entire datacenter fabric, causing minutes of downtime. training/library/topics/mellanox-switches/footguns.md mellanox-switches/f4a5b6c7d8e9 mellanox-switches hard mellanox, roce, tuning What three mechanisms must be configured together for a properly functioning RoCEv2 lossless fabric? "1) PFC — to pause traffic before buffer overflow
2) ECN — to signal senders to back off before PFC activates
3) DCBX trust mode — to ensure traffic is classified into the correct priority queue on both switch and NIC\n\nRemember: RDMA = bypass CPU for low latency. RoCE = RDMA over Ethernet. v2 uses UDP.\n\nName origin: RDMA = Remote Direct Memory Access. RoCE = RDMA over Converged Ethernet. v2 added UDP encapsulation for L3 routing.\n\nRemember: ""PFC + ECN + DCBX = the lossless trio."" Miss any one and RoCE performance degrades unpredictably." training/library/topics/mellanox-switches/street_ops.md mellanox-switches/a5b6c7d8e9f0 mellanox-switches hard mellanox, pfc, storm Explain how a PFC storm propagates through a datacenter fabric. A slow receiver causes its ingress port to buffer. When the buffer fills, PFC sends pause frames upstream. The upstream switch port buffers, fills, and sends PFC pauses further upstream. This cascading pause can reach spine switches, pausing unrelated traffic across the entire fabric.\n\nRemember: DCB = lossless Ethernet. PFC=flow control, ECN=congestion notification. For RoCE.\n\nDebug clue: Check `show interfaces ethernet counters pfc` for pause frame counts. A port sending millions of pause frames per second is the storm source. training/library/topics/mellanox-switches/footguns.md mellanox-switches/b6c7d8e9f0a1 mellanox-switches hard mellanox, mlag, issu What is ISSU in the context of Mellanox MLAG, and what is the correct upgrade procedure? "In-Service Software Upgrade — upgrading firmware on one MLAG peer at a time without traffic disruption. Procedure: upgrade peer 1, verify MLAG reconverges and traffic is healthy, then upgrade peer 2. Both peers must end up on the same firmware version.\n\nFun fact: Mellanox acquired by NVIDIA (2020, $6.9B). Powers AI/HPC networking.\n\nRemember: ""ISSU = upgrade one peer at a time."" Traffic flows through the surviving peer during each upgrade. Both peers must end on the same version." training/library/topics/mellanox-switches/street_ops.md mellanox-switches/c7d8e9f0a1b2 mellanox-switches hard mellanox, api, security What is the security risk of the Onyx REST API and how should it be secured in production? The REST API provides full read-write access to switch configuration. If enabled without authentication (common in lab), it is a critical vulnerability. Secure with: HTTPS, strong authentication, management ACLs limiting source IPs, and regular access log auditing.\n\nRemember: ONYX = Mellanox switch OS. Cumulus = Linux-based NOS. Both on Spectrum ASICs.\n\nGotcha: Some lab environments enable the REST API with default credentials. This is a critical vulnerability — any script can reconfigure the switch remotely. training/library/topics/mellanox-switches/footguns.md mellanox-switches/d8e9f0a1b2c3 mellanox-switches hard mellanox, wjh, debug Why might WJH show no drops even when packets are being lost, and how do you fix this? "WJH channels (forwarding, ACL) may not be enabled by default on some firmware versions. Enable with: what-just-happened forwarding enable and what-just-happened acl enable. Also enable auto-export.\n\nRemember: Mellanox/NVIDIA = high-performance networking. InfiniBand for HPC, Spectrum for Ethernet.\n\nUnder the hood: Spectrum ASICs provide wire-speed L2/L3 with programmable pipeline.\n\nRemember: ""No WJH output ≠ no drops."" Always verify WJH channels are enabled: `what-just-happened forwarding enable`." training/library/topics/mellanox-switches/footguns.md mellanox-switches/e9f0a1b2c3d4 mellanox-switches hard mellanox, thermal What happens when a Spectrum ASIC exceeds its thermal threshold, and how do you prevent it? The switch silently thermal-throttles, reducing port speeds to lower heat. Monitor via show environment and SNMP. Alert when ASIC temp exceeds 85C. Ensure proper hot/cold aisle separation and adequate rack airflow.\n\nRemember: Spectrum ASICs: -1(100G), -2(200G), -3(400G), -4(800G). Each gen ~doubles.\n\nNumber anchor: Spectrum ASICs throttle around 105°C junction temperature. Alert at 85°C to catch airflow problems before performance degrades. training/library/topics/mellanox-switches/footguns.md mellanox-switches/f0a1b2c3d4e5 mellanox-switches easy mellanox, models Which Mellanox switch model series is most commonly deployed as a ToR/leaf switch? The SN2100 (16x100G) and SN2700 (32x100G) from the Spectrum (SN2000) family are the most common leaf/ToR choices.\n\nRemember: Mellanox/NVIDIA = high-performance networking. InfiniBand for HPC, Spectrum for Ethernet.\n\nUnder the hood: Spectrum ASICs provide wire-speed L2/L3 with programmable pipeline.\n\nUnder the hood: SN2100 is popular as a leaf switch because 16x100G fits most rack sizes. SN2700 (32x100G) is used when higher density is needed. training/library/topics/mellanox-switches/primer.md mellanox-switches/a1c2d3e4f5a6 mellanox-switches medium mellanox, licensing What features are included in the Mellanox/Onyx base license? L2/L3 switching, BGP, OSPF, VXLAN, MLAG, RDMA/RoCE, WJH, REST API, and SNMP. The base license is more feature-rich than many competitors.\n\nFun fact: Mellanox acquired by NVIDIA (2020, $6.9B). Powers AI/HPC networking.\n\nFun fact: Unlike Cisco where BGP and OSPF often require separate licenses, Mellanox/NVIDIA includes L3 routing in the base license — a significant cost advantage. training/library/topics/mellanox-switches/primer.md mellanox-switches/b2d3e4f5a6b7 mellanox-switches hard mellanox, optics, qualified What can happen with non-qualified optics on Mellanox switches? Non-qualified optics may be silently degraded: speed-limited, DOM unavailable, or intermittent CRC errors. The switch validates transceiver vendor/part against a qualified list. Some firmware allows overriding the check but this is unsupported.\n\nFun fact: Mellanox acquired by NVIDIA (2020, $6.9B). Powers AI/HPC networking.\n\nWar story: Using cheap third-party optics saved $500 per port but caused intermittent CRC errors under load, resulting in packet loss that was invisible to monitoring until WJH was enabled. training/library/topics/mellanox-switches/footguns.md mellanox-switches/c3e4f5a6b7c8 mellanox-switches easy mellanox, onyx, show What Onyx command shows hardware temperature, fan speed, and PSU status? "show environment\n\nRemember: ONYX = Mellanox switch OS. Cumulus = Linux-based NOS. Both on Spectrum ASICs.\n\nRemember: ""show environment = health check."" Check it after any maintenance window or airflow changes." training/library/topics/mellanox-switches/primer.md --- ## Wiki Navigation ### Related Content - [Mellanox Switches](../../../../library/topics/mellanox-switches/index.md) (Topic Pack, L2) — Mellanox Switches