Skip to content

Solution: BMC Clock Skew - Certificate Failure

Triage

  1. Check the BMC's current date/time: 

    # Via ipmitool from the host OS
sudo ipmitool sel time get

# Or via iLO SSH CLI
ssh admin@10.50.2.111
show /map1/firmware1 | grep date

  2. Check the certificate validity dates: 

    echo | openssl s_client -connect 10.50.2.111:443 2>/dev/null | openssl x509 -noout -dates

  3. Compare: if the BMC clock is set to a date before the certificate's "Not Before" date, TLS clients will reject it as "not yet valid."

Root Cause

The server was powered off for 3 weeks during the datacenter power upgrade. The BMC's real-time clock (RTC) is backed by a small battery (CR2032 or similar). After extended power loss, the BMC clock either: - Drifted significantly due to a weak battery, or - Reset to a factory default date (e.g., January 1, 2000)

When the server was powered back on, the BMC booted with the wrong date. The self-signed SSL certificate (valid from 2024-06-15 to 2029-06-15) appears "not yet valid" because the BMC thinks it is January 2000. The host OS corrected its clock via NTP at boot, but the BMC has its own independent clock that was not corrected.

Fix

  1. Set the BMC clock to the correct time: 

    # From the host OS
sudo ipmitool sel time set "03/05/2026 10:30:00"

# Or via iLO SSH
ssh admin@10.50.2.111
set /map1/firmware1 oemhp_date=03/05/2026
set /map1/firmware1 oemhp_time=10:30:00

  2. Configure NTP on the BMC so the clock stays synchronized: 

    # Via iLO web UI:
#   Administration -> Network -> SNTP Settings
#   Primary NTP Server: 10.10.0.123
#   Secondary NTP Server: 10.10.0.124
#   Timezone: UTC

# Or via iLO SSH:
set /map1/enetwork1 oemhp_sntp_server1=10.10.0.123
set /map1/enetwork1 oemhp_sntp_server2=10.10.0.124

  3. Verify the fix: 

    # Test HTTPS access
curl -k https://10.50.2.111/redfish/v1/Systems/1 -u admin:password

# Check cert validity from client perspective
echo | openssl s_client -connect 10.50.2.111:443 2>/dev/null | openssl x509 -noout -dates -subject

  4. If the certificate has genuinely expired (not just a clock issue), regenerate it: 

    # Via iLO web UI: Security -> SSL Certificate -> Generate CSR / Generate Self-Signed
# This triggers an iLO reset (brief management plane interruption)

  5. Re-enable monitoring:

  6. Verify Redfish API calls work from monitoring server.
  7. Check that historical data gap is documented.

Rollback / Safety

  • Setting the BMC clock and configuring NTP are non-disruptive operations. They do not affect the host OS.
  • If you regenerate the SSL certificate, the iLO will reset (1-2 minute interruption to management access only; host OS unaffected).
  • Any scripts or tools that pin the old certificate thumbprint will need updating after certificate regeneration.

Common Traps

  • Trap: Regenerating the certificate without fixing the clock first. If the BMC clock is still wrong, the new certificate will also be generated with wrong dates.
  • Trap: Using curl -k (skip validation) as a permanent workaround. This disables certificate validation and masks the real issue.
  • Trap: Not configuring NTP on the BMC. Without NTP, the BMC clock will drift again over time, especially if the RTC battery is weak.
  • Trap: Forgetting to check other servers that were offline during the same maintenance. If one BMC has clock issues, others likely do too.
  • Trap: Not testing that monitoring scripts actually resume after the fix. The scripts may have error-handling that requires a manual restart.