Skip to content

Portal | Level: L2: Operations | Topics: systemd, Linux Fundamentals | Domain: Linux

systemd Architecture

Scope

This document explains systemd as the init system and service manager that brings a modern Linux system to life after the kernel hands off to PID 1. It covers:

  • boot targets and ordering
  • units and dependencies
  • service types
  • sockets, timers, mounts, automounts
  • cgroup integration
  • journald
  • common debugging patterns
  • boot performance analysis

Big picture

systemd is not just "the thing that starts services." When running as PID 1 it is the central orchestrator of userspace boot and service lifecycle.

High-level boot handoff

firmware
  -> bootloader
  -> kernel
  -> initramfs / switch_root
  -> PID 1 = systemd
  -> generators run
  -> unit graph is built
  -> default target and its dependencies are queued
  -> services, sockets, mounts, devices, timers, and other units activate
  -> system reaches operational target

What systemd actually does

  • tracks desired and actual unit state
  • builds dependency graphs
  • orders startup and shutdown
  • supervises services
  • manages cgroups
  • integrates logging and activation mechanisms
  • provides standardized interfaces for service lifecycle

It is closer to a boot-time and runtime dependency scheduler plus service supervisor than to a dumb init script runner.

What PID 1 really means

The PID 1 process is special on Linux: it reaps orphans/zombies, defines boot and shutdown policy, and acts as the root of service supervision.

systemd takes that role and adds dependency resolution, structured logging, cgroup integration, socket activation, on-demand startup, transient units, resource control, and isolation features.

If PID 1 is broken, the machine is broken in a very literal sense.

flowchart TD
    SYS["sysinit.target\nMounts, swap, udev\njournald, tmpfiles"] --> BAS["basic.target\nSockets, timers, paths\nOS plumbing ready"]
    BAS --> MU["multi-user.target\nNetwork, SSH, services\nNon-graphical login"]
    MU --> GR["graphical.target\nDisplay manager\nDesktop session"]

    SYS --> |"early"| journald["journald.service"]
    SYS --> |"early"| udev["udev.service"]
    BAS --> |"plumbing"| sockets["sockets.target"]
    BAS --> |"plumbing"| timers["timers.target"]
    MU --> |"services"| sshd["sshd.service"]
    MU --> |"services"| net["network.target"]
    GR --> |"desktop"| dm["display-manager.service"]

    style SYS fill:#4a6fa5,color:#fff
    style BAS fill:#6b8cae,color:#fff
    style MU fill:#f0ad4e,color:#333
    style GR fill:#5cb85c,color:#fff

Unit types

A unit is a configuration object representing something systemd can manage.

Common unit types:

  • .service - daemon or one-shot work
  • .socket - socket activation
  • .target - synchronization / grouping point
  • .mount - filesystem mount
  • .automount - on-demand mount trigger
  • .timer - scheduled activation
  • .path - activate on filesystem events
  • .device - kernel device exposure
  • .slice - cgroup hierarchy partition
  • .scope - externally created processes tracked by systemd
  • .swap - swap device/file
  • .tmpfiles / generators / other helpers exist around the ecosystem

Key point

A target is not a runlevel clone in the simplistic sense. It is a dependency grouping point in a graph.

Dependencies vs ordering

This distinction causes endless confusion.

Requirement dependency

Examples:

  • Requires=
  • Wants=
  • BindsTo=
  • PartOf=

These define relationship strength: what should come along or what failing units affect others.

Ordering dependency

Examples:

  • Before=
  • After=

These define sequence, not requirement.

Important truth

After=network.target does not mean "make sure networking is really usable." It only means order after that target's activation event. Wrong mental model here creates a lot of boot bugs.

Boot transaction

At boot, systemd computes a transaction: the set of units to activate to reach the desired target while respecting dependencies and ordering rules.

It then starts as much as possible in parallel, constrained by those rules.

This is why modern boot is faster than old serial init script chains. systemd is graph-based, not a giant linear shell script.

When you request a unit start, systemd builds a transaction: pulls in requirements, validates conflicts, orders work, merges with current state, and rejects impossible operations. The graph, not the command line, is the truth.

Boot stages and special targets

Important targets and milestones include:

  • sysinit.target - early system initialization
  • basic.target - basic system initialization complete enough for broader startup
  • multi-user.target - non-graphical multi-user mode
  • graphical.target - graphical login / desktop environment path
  • rescue.target
  • emergency.target
  • shutdown.target

Simplified boot flow

local-fs.target
  -> swap.target
  -> basic.target
  -> multi-user.target
  -> graphical.target (if used)

Real graphs are more complex, but this is the useful admin mental model.

Generators

Generators are small programs run early by systemd to dynamically produce unit files or symlinks based on current system state.

Examples of what generators handle:

  • fstab conversion into mount units
  • cryptsetup-derived units
  • system update redirection
  • initrd-specific logic

Why generators matter

The final runtime unit graph is not always just what sits statically in /etc/systemd/system and /usr/lib/systemd/system. Dynamic generation changes the graph.

Service unit types

A .service unit can represent different startup semantics.

Common Type= values:

  • simple
  • exec
  • forking
  • oneshot
  • notify
  • dbus

Why type matters

systemd needs to know when to consider startup successful and how to supervise the process.

Examples:

  • Type=simple assumes the main process is the launched command
  • Type=forking is for old daemons that background themselves
  • Type=notify waits for explicit readiness signal from the service
  • Type=oneshot is for finite tasks

Misusing Type= is a classic cause of fake-success startups or broken dependency timing.

Socket activation

systemd can create listening sockets first and only start the service when traffic arrives.

Benefits:

  • on-demand startup
  • reduced boot-time work
  • service restart without losing listen socket in some models
  • compatibility with superserver-like patterns

This is one of the reasons systemd is more than a service runner.

Timer units vs cron

Timers are systemd's native scheduled activation mechanism.

Advantages over classic cron for many use cases:

  • dependency-aware activation
  • journal integration
  • missed-run persistence with Persistent=
  • standard unit semantics
  • easy introspection with systemctl list-timers

Cron still exists, but timers are usually cleaner on systemd-managed hosts.

Mount and automount units

Mounts are first-class managed units. This allows:

  • boot ordering around filesystems
  • failure propagation
  • automount-on-access behavior
  • consistent dependency handling

This matters a lot for network filesystems and slow mounts that would otherwise stall boot.

cgroups and resource management

Modern systemd uses cgroups as its process tracking and resource control foundation.

Why this matters

systemd can reliably know which processes belong to a service because they live in that service's cgroup.

This avoids many old init-script problems where child processes escaped supervision like greased raccoons.

Resource controls

systemd can set limits such as:

  • CPU weight/quota
  • memory ceilings
  • IO weight/throttling
  • task limits

It effectively acts as a cgroup manager for services and slices.

Slices

Slices organize cgroup hierarchy, for example:

  • system.slice
  • user.slice
  • custom slices for workload grouping

Journald

systemd-journald provides structured log collection for local services and kernel/user logs.

Why it matters

  • service logs correlate to unit identity
  • boot logs can be queried by boot ID
  • metadata-rich searching is possible
  • journal and classic syslog can coexist depending on setup

Useful commands:

  • journalctl -b
  • journalctl -u sshd
  • journalctl -xe
  • journalctl -k

Operational caution

The journal is great for investigation, but retention, forwarding, and durability choices still matter. Local logs alone are not your observability strategy.

Activation states

Units can be in states like:

  • inactive
  • activating
  • active
  • deactivating
  • failed

There are also substate details that matter for services.

Key point: "loaded" only means the unit definition is known, not that the service is running.

Enable vs start

Another evergreen confusion.

  • systemctl start foo -> activate now
  • systemctl enable foo -> make it start automatically via symlink-based dependency wiring
  • systemctl enable --now foo -> both

If someone says a service is "enabled" that does not prove it is currently running.

Boot performance analysis

systemd-analyze provides boot performance introspection.

Useful commands:

  • systemd-analyze
  • systemd-analyze blame
  • systemd-analyze critical-chain

blame

Shows units sorted by reported initialization time. Useful, but can mislead because waiting time and real boot criticality are not identical.

critical-chain

Shows the ordering chain that actually determined when a target was reached. This is often more useful than blaming the longest individual unit.

Service supervision and restart logic

A service can specify policies such as:

  • Restart=
  • RestartSec=
  • watchdog integration
  • start rate limits

This is powerful but also dangerous if abused. A misconfigured restart loop can thrash boot or flood logs.

Shutdown flow

systemd also orchestrates shutdown:

  • reverse dependency order
  • stop units
  • unmount filesystems
  • tear down swap
  • reach shutdown/reboot/poweroff targets

Clean shutdown is not just "kill everything and pray."

Common failure patterns

1. Service starts manually but not at boot

Possible reasons:

  • not enabled
  • wrong target wiring
  • ordering issue
  • dependency failed at boot
  • environment difference between boot and interactive shell

2. Service marked active too early

Usually:

  • wrong Type=
  • app daemonizes unexpectedly
  • missing sd_notify() usage for Type=notify

3. Boot slow because of a mount or network wait

Common causes:

  • remote mount blocking boot
  • network-online.target misunderstanding
  • slow device timeout
  • dependency chain accidentally serialized

4. Service restarts forever

Could be:

  • Restart=always
  • failing dependency
  • bad config
  • permissions issue
  • missing secret or socket

5. Logs missing or fragmented

Could be:

  • journal volatility / retention behavior
  • service writes elsewhere
  • stdout/stderr redirection assumptions wrong
  • chroot/containerized execution differences

Debugging workflow

Step 1 - inspect unit status

systemctl status <unit>

This gives:

  • active state
  • recent logs
  • main PID
  • result reason
  • unit file path

Step 2 - inspect journal for that unit

journalctl -u <unit> -b

Use boot scoping aggressively.

Step 3 - inspect unit definition and overrides

  • systemctl cat <unit>
  • systemctl show <unit>
  • look for drop-ins under /etc/systemd/system/<unit>.d/

Step 4 - inspect dependencies

  • systemctl list-dependencies <unit>
  • systemd-analyze critical-chain <unit>

Step 5 - inspect cgroup/process behavior

  • systemctl status
  • systemd-cgls
  • systemd-cgtop

Interview angles

Questions often hidden here:

  • difference between Requires= and After=
  • what systemd gains from cgroups
  • difference between start and enable
  • what Type=notify solves
  • socket activation purpose
  • why systemd-analyze blame can mislead
  • what a target is
  • how timers differ from cron
  • why network.target is not "network fully ready"

Strong answers focus on dependency graph + supervision + cgroups.

Mental model to keep

systemd is a dependency graph executor and service supervisor for userspace.

It:

  • receives control after kernel/initramfs handoff
  • builds a unit graph
  • activates targets and dependencies in parallel where possible
  • tracks services by cgroups
  • supervises lifecycle
  • integrates logging and activation mechanisms
  • manages orderly shutdown

If boot or service management is broken, the question is usually:

  • what unit relationship, readiness signal, or resource policy was modeled wrong?

References

Fast mental model

systemd is a dependency-driven state transition engine for Linux userspace.

Or simpler: kernel gets you to PID 1; systemd gets you to a working operating system.

Wiki Navigation

Prerequisites

Related Content