Skip to content

Incident Replay: Duplex Mismatch Symptoms

Setup

  • System context: Production server connected via 1GbE to a ToR switch. Network performance is severely degraded — throughput is only 10Mbps on a 1Gbps link.
  • Time: Tuesday 16:00 UTC
  • Your role: Network engineer / on-call SRE

Round 1: Alert Fires

[Pressure cue: "Application team reports file transfers to stor-prod-02 take 100x longer than expected. 'The server is really slow.'"]

What you see: iperf3 between the server and another host shows 8Mbps throughput. Interface shows 1000Mbps/Full. But the switch port shows 1000Mbps/Half. Duplex mismatch.

Choose your action: - A) Restart the network interface - B) Check ethtool settings and switch port configuration - C) Replace the network cable - D) Check for packet loss with ping

If you chose A:

[Result: Interface renegotiates but lands on the same mismatch. Auto-negotiation is failing.]

[Result: ethtool eth0 shows speed 1000Mbps, duplex Full. Switch port is hardcoded to 1000/Half. Auto-negotiation on the server detected the speed correctly but the switch's hardcoded half-duplex overrides. Proceed to Round 2.]

If you chose C:

[Result: New cable, same mismatch. The issue is configuration, not physical.]

If you chose D:

[Result: ping -f shows 30% packet loss. Confirms a problem but does not identify duplex mismatch.]

Round 2: First Triage Data

[Pressure cue: "Problem scoped. Apply the fix."]

What you see: Root cause from Round 1 narrows the investigation. You need to apply the correct fix and verify.

Choose your action: - A) Apply the quick targeted fix - B) Apply the comprehensive fix with verification - C) Apply a workaround while planning the proper fix - D) Escalate to a specialist team

If you chose A:

[Result: Quick fix resolves the immediate issue but may not be durable. Proceed cautiously.]

[Result: Comprehensive fix applied with verification steps. Issue resolved. Proceed to Round 3.]

If you chose C:

[Result: Workaround buys time but the root cause remains. Acceptable short-term.]

If you chose D:

[Result: Specialist is unavailable or adds delay. Try the fix yourself first.]

Round 3: Root Cause Identification

[Pressure cue: "Fix applied. Document root cause and prevention."]

What you see: Root cause is confirmed. Process or configuration gap that allowed this to happen is identified.

Choose your action: - A) Fix the specific instance only - B) Fix the instance and add monitoring - C) Fix the instance, add monitoring, and update procedures - D) Comprehensive: fix + monitor + procedure + automation

[Result: All layers addressed. Immediate fix, detection, process, and automation. Proceed to Round 4.]

If you chose A:

[Result: Fixes this case but the same mistake can recur.]

If you chose B:

[Result: Better detection next time but does not prevent recurrence.]

If you chose C:

[Result: Good coverage but automation reduces human error further.]

Round 4: Remediation

[Pressure cue: "Service restored. Verify and close."]

Actions: 1. Verify the service is functioning correctly 2. Verify monitoring detects the fix 3. Update runbooks and procedures 4. Schedule follow-up actions (automation, infrastructure changes) 5. Close the incident with a post-mortem

Damage Report

  • Total downtime: Varies based on path taken
  • Blast radius: Affected service and dependent systems
  • Optimal resolution time: 10 minutes
  • If every wrong choice was made: 60 minutes + additional damage

Cross-References