Skip to content

Lab 1: Linux Triage

Field Value
Tier 1 — Foundations
Estimated Time 30 minutes
Prerequisites Basic Linux CLI, SSH access
Auto-Grade Yes

Scenario

You have just joined an on-call rotation at a mid-size SaaS company. At 02:17 AM, PagerDuty fires an alert: the main application server is unresponsive. Customers are reporting slow page loads and intermittent 502 errors. The previous on-call engineer left no runbook for this host.

You SSH in and find a mess. The disk is nearly full because nobody rotated logs. A zombie process is consuming a process slot. A misconfigured cron job is flooding the system with failed tasks every minute. File permissions on the application config directory are wide open. And DNS resolution is broken because someone edited /etc/resolv.conf incorrectly.

Your job is to stabilize the server. Fix each issue, verify the fix, and leave the system in a clean state. Time is ticking — customers are affected.

Objectives

  • Free at least 500 MB of disk space by cleaning up oversized log files
  • Identify and terminate the zombie process (and its parent)
  • Fix the broken cron job in /etc/cron.d/broken-task
  • Correct file permissions on /opt/app/config/ (owner root, mode 750)
  • Repair DNS resolution by fixing /etc/resolv.conf

Setup

./setup.sh

The setup script creates a simulated broken environment under /tmp/lab-linux-triage/. All paths referenced above are relative to that root (e.g., /tmp/lab-linux-triage/etc/resolv.conf).

Hints

Hint 1: Finding disk hogs Use `du -sh /tmp/lab-linux-triage/var/log/*` to find the largest log files. You can truncate them with `> filename` or remove old rotated copies.
Hint 2: Zombie processes Zombies show as state `Z` in `ps aux`. You cannot kill a zombie directly — you need to kill or signal the parent process. Check `ps -eo pid,ppid,stat,comm` to find the parent.
Hint 3: Cron debugging Look at the cron file syntax. Common mistakes: wrong field count, missing user field in `/etc/cron.d/` files, or invalid command paths. Use `crontab -l` and check `/var/log/syslog` for cron errors.
Hint 4: File permissions `chmod 750 /opt/app/config/` sets rwxr-x---. Use `chown root:root` for ownership. Check with `stat -c '%a %U:%G' /opt/app/config/`.
Hint 5: DNS resolution `/etc/resolv.conf` needs at least one valid `nameserver` line. Common DNS servers: `8.8.8.8`, `1.1.1.1`. Test with `nslookup example.com` or `dig example.com`.

Grading

./grade.sh

The grading script checks each objective independently and prints pass/fail status.

Solution

See the solution/ directory for a walkthrough of each fix.