Skip to content

Symptoms: Disk Full Root - Services Down

  • Multiple services on api-gateway-03 (Ubuntu 22.04) are failing to start or crashing.
  • Monitoring alerts fired: "Disk space critical: / at 100%" at 04:17 UTC.
  • SSH login works but shows: "No space left on device" when running some commands.
  • Nginx is returning 502 errors because upstream application processes cannot write temp files.
  • Systemd journal is also failing to write, causing some log gaps.
  • The server has a 50GB root partition; no separate /var partition.
  • Last successful config management run was 3 days ago.
  • The on-call engineer was paged and needs to restore service quickly.