From Init Scripts to systemd

• lesson
• sysv-init
• upstart
• systemd
• process-supervision
• boot-sequence
• linux-history ---# From Init Scripts to systemd

Topics: SysV init, Upstart, systemd, process supervision, boot sequence, Linux history Level: L1–L2 (Foundations → Operations) Time: 45–60 minutes Prerequisites: None (everything is explained from scratch)

---

The Mission

Your coworker asks: "Why does everyone argue about systemd? It's just an init system, right?"

It's not "just" an init system. systemd is the most controversial project in Linux history, and the arguments reveal deep disagreements about how Unix systems should work. Understanding why it exists — and what it replaced — makes you better at operating Linux systems, because you'll understand the design decisions behind every systemctl command, every unit file, and every journal entry.

This lesson traces the evolution of process management on Linux: what problem each generation solved, what new problems it created, and why the current answer still makes people angry.

---

Generation 0: The Simplest Thing That Works

In the beginning, Unix had a simple init: PID 1 read /etc/inittab, spawned getty processes for terminal login, and ran shell scripts for each runlevel (system state).

# /etc/inittab (simplified)
id:3:initdefault:                        # Boot to runlevel 3 (multi-user, no GUI)
si::sysinit:/etc/rc.d/rc.sysinit         # Run system init script
l3:3:wait:/etc/rc.d/rc 3                 # Run scripts for runlevel 3
1:2345:respawn:/sbin/getty tty1 9600     # Spawn login on tty1

Runlevels were numbered 0-6:

Runlevel	Meaning
0	Halt
1	Single user (rescue)
2	Multi-user, no networking (Debian: full multi-user)
3	Multi-user with networking
5	Multi-user with GUI
6	Reboot

To start services in a runlevel, you put numbered shell scripts in /etc/rc.d/rcN.d/:

/etc/rc.d/rc3.d/
├── S01networking     # S = Start, 01 = order
├── S02sshd           # Start SSH second
├── S03docker         # Start Docker third
├── K01bluetooth      # K = Kill, stop this service in runlevel 3
└── K02avahi          # Stop this too

S scripts run in numeric order at startup. K scripts run in order at shutdown. Each script was a full bash script with start, stop, restart, and status cases:

#!/bin/bash
# /etc/init.d/myapp

case "$1" in
    start)
        echo "Starting myapp..."
        /usr/bin/myapp &
        echo $! > /var/run/myapp.pid
        ;;
    stop)
        echo "Stopping myapp..."
        kill $(cat /var/run/myapp.pid)
        rm /var/run/myapp.pid
        ;;
    restart)
        $0 stop
        sleep 2
        $0 start
        ;;
    status)
        if [ -f /var/run/myapp.pid ] && kill -0 $(cat /var/run/myapp.pid) 2>/dev/null; then
            echo "myapp is running"
        else
            echo "myapp is stopped"
        fi
        ;;
esac

What worked

• Dead simple. Anyone who can write bash can write an init script.
• No dependencies on anything except the shell.
• Easy to read, easy to debug (bash -x /etc/init.d/myapp start).

What broke

Sequential startup. 200 services booted one after another. On modern hardware with SSDs and multi-core CPUs, this meant 30-60 seconds of wasted time waiting for services that could have started in parallel.

PID file tracking. The init system had no actual connection to the process — it relied on a PID file written by the service. If the PID file was stale, wrong, or the process double-forked, init lost track entirely. Stopping a service meant killing whatever PID was in the file, which might be a completely different process by now.

No dependency management. The "S01, S02, S03" numbering was manual ordering, not dependency resolution. If S03 needed S01 but S01 was slow, S03 would start and fail. There was no way to express "wait until the database is actually ready."

No automatic restart. If a service crashed at 3am, it stayed down until someone noticed. The only built-in restart mechanism was respawn in /etc/inittab, which was designed for getty processes, not application servers.

No resource isolation. A runaway service could consume all CPU, all RAM, all disk I/O, and there was nothing to contain it. The only answer was manual nice/ionice and hope.

Trivia: SysV init came from AT&T's System V Unix (1983). The rc in rc.d stands for "run commands." The numbering convention (S01, S02) was a clever hack — the shell just globbed S* and the filesystem sort order handled sequencing. It worked for 30 years.

---

Generation 1: Upstart — Event-Driven Init

Ubuntu shipped Upstart in 2006 (created by Scott James Remnant at Canonical) to solve the parallelism and dependency problems.

Upstart's key idea: instead of sequential scripts, services react to events. A service starts when its triggering event fires:

# /etc/init/myapp.conf (Upstart)
description "My Application"

start on (filesystem and net-device-up IFACE!=lo)
stop on runlevel [016]

respawn
respawn limit 10 5

exec /usr/bin/myapp --config /etc/myapp.conf

Feature	SysV init	Upstart
Start order	Numbered scripts (manual)	Events (automatic)
Parallelism	None (sequential)	Yes (event-driven)
Restart on crash	No (manual)	respawn directive
Process tracking	PID file (fragile)	Direct child tracking
Config format	Shell scripts	Declarative stanzas

What Upstart fixed

• Parallel startup (event-driven, not sequential)
• Automatic restart (respawn)
• Cleaner config (declarative, not imperative)
• Better process tracking (no PID file games for simple services)

What Upstart didn't fix

• Still no cgroup-based process tracking (double-forking daemons could escape)
• Event model was confusing ("start on started networking" — which event exactly?)
• Ubuntu-specific — Fedora, RHEL, and others never adopted it
• No socket activation, no timer units, no resource control

Upstart was a solid step forward, but it was a Canonical project that never gained cross-distribution adoption. When systemd appeared, the writing was on the wall.

---

Generation 2: systemd — The Full Rewrite

Lennart Poettering (Red Hat) announced systemd in 2010. It was not an incremental improvement — it was a complete rethinking of what PID 1 should be.

The core ideas

1. Declarative, not imperative. Unit files describe what you want, not how to do it:

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application
After=network.target postgresql.service
Requires=postgresql.service

[Service]
Type=simple
User=appuser
ExecStart=/usr/bin/myapp --config /etc/myapp.conf
Restart=on-failure
RestartSec=5
MemoryMax=1G

[Install]
WantedBy=multi-user.target

Compare this to the 30-line bash init script above. The unit file is shorter AND more powerful — it includes restart logic, resource limits, dependency management, and user isolation that the bash script doesn't have.

2. Parallel startup via dependency graph. systemd reads all unit files, builds a dependency graph, and starts everything it can simultaneously:

systemd-analyze critical-chain
# → multi-user.target @8.2s
# → └─docker.service @4.1s +2.3s
# →   └─network-online.target @3.8s
# →     └─NetworkManager-wait-online.service @1.2s +2.6s
# →       └─NetworkManager.service @0.8s +0.3s

3. Cgroup-based process tracking. Every service runs in its own cgroup. If a service forks 50 child processes, systemd knows about all of them — not just the PID in a file. When you systemctl stop myapp, it kills the entire cgroup tree. No orphans, no stragglers.

4. Socket activation. systemd can listen on a socket before starting the service. When a connection arrives, systemd starts the service and hands over the socket. This means: - Services can start in any order (the socket is ready before the service is) - Idle services use zero resources until needed - Graceful restart — the socket stays open during restart, buffering connections

Name Origin: The "d" in systemd stands for "daemon." The lowercase "s" is intentional — Poettering has said it's meant to be lowercase, like a Unix command.

---

The Controversy

systemd is the most controversial project in Linux history. The arguments are real and technical, not just tribal.

The case for systemd

• Parallel boot is 5-10x faster than sequential SysV scripts
• Unit files are testable and parseable — they're not arbitrary shell scripts that might do anything
• Cgroup tracking solves the double-fork problem that PID files never could
• One consistent interface across all distributions (Fedora, Debian, Ubuntu, RHEL, SUSE)
• Resource limits are built in — MemoryMax=, CPUQuota= in the unit file
• Journal is structured — you can query by time, service, severity, not just grep

The case against systemd

• Scope creep. systemd absorbed init, cron (timers), syslog (journald), DNS (resolved), network config (networkd), login management (logind), device management (udevd), and a bootloader (systemd-boot). Critics say this violates the Unix philosophy of "do one thing well."

• Binary logs. journald stores logs in binary format. You can't grep raw journal files — you must use journalctl. Traditionalists who built decades of tooling around text logs (grep, awk, sed) found their workflows broken.

• Complexity. SysV init was simple enough to understand completely in an afternoon. systemd has hundreds of configuration options, a dozen unit types, and behavior that's difficult to predict without reading the documentation carefully.

• Hard to avoid. Because systemd provides so many services (DNS, logging, network, etc.), replacing just the init system is nearly impossible. It's all or nothing.

Trivia: The Debian vote on init systems (2014) was one of the most divisive events in open source history. The vote was close, the arguments were heated, and it led to the Devuan fork — a Debian variant specifically without systemd, maintained by "Veteran Unix Admins." Poettering had previously created PulseAudio, which was also controversial. The running joke is that systemd has absorbed so many system functions that it's "an operating system that happens to contain Linux."

---

What systemd Replaced — A Complete Map

Old tool	What it did	systemd replacement
SysV init scripts	Service startup/shutdown	.service unit files
cron / anacron	Scheduled tasks	.timer unit files
syslog / rsyslog	Text logging	journald (binary journal)
xinetd / inetd	Socket-activated services	Socket activation (.socket units)
ConsoleKit	Login/session tracking	logind
udev	Device management	systemd-udevd (absorbed)
NTP clients	Time synchronization	systemd-timesyncd
DHCP client	Network configuration	systemd-networkd
DNS stub resolver	Local DNS	systemd-resolved
shutdown / reboot	System power management	systemctl poweroff / systemctl reboot
/etc/fstab (partially)	Filesystem mounting	.mount units (generated from fstab)
tmpwatch	Temp file cleanup	systemd-tmpfiles
hostname	Hostname management	hostnamectl

---

Living with systemd — The Practical Takeaways

Whether you love or hate systemd, you'll use it every day. Here's what matters:

Unit file > shell script

If you're writing a bash loop to restart a crashed service, you're doing it wrong. systemd's Restart=on-failure does it better, with rate limiting, logging, and cgroup cleanup.

Targets > runlevels

Runlevels were 0-6. Targets are named and can have complex dependencies:

Old runlevel	systemd target	Meaning
0	poweroff.target	Halt
1	rescue.target	Single user
3	multi-user.target	Full system, no GUI
5	graphical.target	Full system with GUI
6	reboot.target	Reboot

# Switch targets (like changing runlevel)
systemctl isolate rescue.target    # Drop to rescue mode
systemctl isolate multi-user.target # Back to multi-user

# Set default boot target
systemctl set-default multi-user.target

Timers > cron

systemd timers give you everything cron does, plus persistent scheduling (run missed jobs after boot), randomized delays (prevent thundering herd), and journal integration:

# /etc/systemd/system/backup.timer
[Unit]
Description=Daily backup timer

[Timer]
OnCalendar=daily
Persistent=true           # Run missed backups after boot
RandomizedDelaySec=1h     # Spread load across the hour

[Install]
WantedBy=timers.target

# List all timers
systemctl list-timers --all

# See when a timer last ran and when it'll run next
systemctl status backup.timer

Journal > log files (for querying)

# Structured queries that are impossible with grep
journalctl -u myapp --since "1 hour ago" -p err
journalctl -u myapp --output json-pretty | jq '.MESSAGE'
journalctl _TRANSPORT=kernel --since today

---

Flashcard Check

Q1: What does the S in S01networking mean in SysV init?

Start. S scripts run at boot in numeric order. K scripts run at shutdown. The number controls ordering — lower numbers run first.

Q2: Why couldn't SysV init track double-forking daemons?

It used PID files. A double-forking daemon writes the final PID to a file, but if the daemon crashes and the PID gets reused, init kills the wrong process. There's no connection between init and the process beyond a text file.

Q3: What did Upstart use instead of numbered scripts?

Events. Services declared what event they needed (start on filesystem and net-device-up) and Upstart started them when the event fired.

Q4: How does systemd track all processes in a service, including forks?

Cgroups. Every service runs in its own cgroup. All forked children are automatically in the same cgroup. systemctl stop kills the entire cgroup, catching all descendants.

Q5: What is socket activation?

systemd listens on a socket before starting the service. When a connection arrives, systemd starts the service and hands over the socket. Services can start in any order because the socket is always ready.

Q6: What's the Devuan fork?

A Debian fork that removes systemd, created after the 2014 Debian init system vote. Maintained by "Veteran Unix Admins" who argued systemd's coupling violated choice.

---

Exercises

Exercise 1: Explore your init system (hands-on)

# What is PID 1?
ps -p 1 -o comm=
# → systemd (probably)

# How many units are loaded?
systemctl list-units --type=service | wc -l

# How many are failed?
systemctl --failed

# What target did we boot to?
systemctl get-default

# How long did boot take, and what was slowest?
systemd-analyze
systemd-analyze blame | head -10

---

Exercise 2: Compare init script vs unit file (think)

Here's a SysV init script. What does the equivalent systemd unit file look like?

#!/bin/bash
# /etc/init.d/myapp
case "$1" in
    start)
        su - appuser -c "/usr/bin/myapp -c /etc/myapp.conf &"
        echo $! > /var/run/myapp.pid
        ;;
    stop)
        kill $(cat /var/run/myapp.pid) 2>/dev/null
        rm -f /var/run/myapp.pid
        ;;
esac

Solution

[Unit]
Description=My Application

[Service]
Type=simple
User=appuser
ExecStart=/usr/bin/myapp -c /etc/myapp.conf
PIDFile=/var/run/myapp.pid

[Install]
WantedBy=multi-user.target

The unit file is shorter, doesn't need explicit PID tracking (systemd uses cgroups), and automatically gets `systemctl status`, journal logging, and dependency management for free. Adding `Restart=on-failure` gives automatic crash recovery — something the init script completely lacks.

---

Cheat Sheet

SysV Init (Legacy)

Task	Command
Start service	/etc/init.d/myapp start or service myapp start
Stop service	/etc/init.d/myapp stop
Check status	/etc/init.d/myapp status
Enable at boot	update-rc.d myapp defaults or chkconfig myapp on

systemd

Task	Command
Start/stop/restart	systemctl {start,stop,restart} myapp
Status + logs	systemctl status myapp
Enable at boot + start now	systemctl enable --now myapp
Reload after editing unit	systemctl daemon-reload
Follow logs	journalctl -u myapp -f
Boot time analysis	systemd-analyze blame
List timers	systemctl list-timers

---

Takeaways

1. Each generation solved the previous generation's problem. SysV was simple but sequential. Upstart was parallel but limited. systemd is comprehensive but complex.

2. Cgroups solved process tracking. PID files were always a hack. Cgroups give systemd reliable tracking of every process a service spawns, including forks and children.

3. The controversy is real and technical. Scope creep, binary logs, complexity, and coupling are legitimate concerns. But so is the 30-year backlog of problems that systemd solved.

4. Unit files beat shell scripts. Declarative, testable, parseable, and they get restart logic, resource limits, and dependency management for free.

5. Timers beat cron. Persistent scheduling, randomized delays, journal integration. Use systemctl list-timers to see what's scheduled.

---

Related Lessons

• What Happens When You Press Power — the boot sequence from firmware to login
• The Hanging Deploy — systemd services, signals, and process lifecycle in practice

Pages that link here

• Cgroups And Namespaces Containers Are A Lie
• Cross-Domain Lessons
• Linux The Complete Guide
• Systemd The Init System You Cant Avoid