Solution: Serial-over-LAN Output Garbled

Triage

1. Since SSH is available, log in and check the current serial console configuration:
2. cat /proc/cmdline -- look for console= parameters
3. cat /etc/default/grub -- check GRUB_CMDLINE_LINUX and GRUB_SERIAL_COMMAND
4. grep -i serial /boot/grub2/grub.cfg -- check compiled GRUB config
5. systemctl status serial-getty@ttyS0.service -- check if getty is running on serial
6. Check the BIOS settings (via SSH to BMC or from documentation):
7. BIOS Serial Console Redirection: COM1, 19200 baud, 8N1, VT100+, no flow control
8. Compare baud rates across all layers:
9. BIOS: 19200
10. GRUB: ? (check config)
11. Kernel: ? (check cmdline)
12. SOL client: ? (default varies)

Root Cause

The BIOS serial console redirection is configured for 19200 baud (set during original installation 2 years ago). The RHEL 7 GRUB and kernel were configured to match at 19200 baud.

During the RHEL 8 upgrade, the GRUB configuration was regenerated with RHEL 8 defaults. RHEL 8's default serial console speed is 115200 baud. The kernel command line was also updated to console=ttyS0,115200. The BIOS was not changed.

This creates a baud rate mismatch: - BIOS sends at 19200 baud -> SOL client expects 19200 -> BIOS output is readable (but GRUB is not) - GRUB and kernel send at 115200 baud -> SOL/BIOS expects 19200 -> output is garbled

The garbled output is the classic symptom of mismatched baud rates: the receiver interprets bits at the wrong speed, producing random characters.

Fix

1. Option A: Update BIOS to 115200 (preferred, matches modern defaults):
2. Enter BIOS Setup via SOL during POST (garbled but Esc/Del key still works)
3. Navigate to: Advanced > Serial Port Console Redirection
4. Change Baud Rate from 19200 to 115200
5. Save and exit

6. SOL output should be clean from BIOS through GRUB through kernel

7. Option B: Update GRUB and kernel to 19200 (if BIOS cannot be changed):

8. SSH into the server and edit /etc/default/grub:

   GRUB_TERMINAL_OUTPUT="console serial"
   GRUB_SERIAL_COMMAND="serial --speed=19200 --unit=0 --word=8 --parity=no --stop=1"
   GRUB_CMDLINE_LINUX="... console=tty0 console=ttyS0,19200n8"
   

9. Regenerate GRUB config:
   • BIOS boot: grub2-mkconfig -o /boot/grub2/grub.cfg
   • UEFI boot: grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg
10. Update the getty baud rate:
    • systemctl edit serial-getty@ttyS0.service
    • Set: ExecStart=-/sbin/agetty -o '-p -- \\u' 19200 ttyS0 vt100

11. Reboot and test SOL

12. Verify the fix:

13. ipmitool -I lanplus -H <bmc-ip> -U admin -P <pass> sol activate
14. Confirm clean output at BIOS POST, GRUB menu, and kernel boot
15. Confirm interactive login prompt on the serial console

Rollback / Safety

• If Option B breaks the boot (wrong GRUB config), use IPMI SOL to interrupt GRUB and edit the boot entry manually. Even garbled output allows blind typing of known key sequences.
• Always keep a backup of /etc/default/grub before modifying.
• The server is otherwise healthy (SSH works); serial console changes do not affect runtime operation.

Common Traps

• Fixing only one layer: All four layers (BIOS, GRUB, kernel, getty) must agree on baud rate. Fixing GRUB but not the kernel still produces garbled output during kernel boot.
• Wrong COM port: Supermicro boards typically use COM1 (ttyS0) for SOL, but some models use COM2 (ttyS1). Using the wrong port produces no output at all, not garbled output.
• Forgetting to regenerate grub.cfg: Editing /etc/default/grub has no effect until grub2-mkconfig is run.
• Flow control mismatch: If BIOS enables hardware flow control (RTS/CTS) but the OS does not, data can stall. Set flow control to "none" on all layers for SOL.
• Assuming the BMC is broken: Garbled output almost always means baud rate mismatch, not a hardware fault.