Skip to content

Portal | Level: L1: Foundations | Topics: systemctl & journalctl Deep Dive, systemd, Linux Fundamentals | Domain: Linux

systemctl & journalctl Primer

This primer goes beyond systemctl start and journalctl -f. It covers the full unit type taxonomy, unit file anatomy, service types, the complete systemctl command surface, journalctl deep querying, timer units, socket activation, template units, drop-in overrides, boot analysis, resource control, and sandboxing directives.

Unit Types

systemd manages more than services. Every managed object is a unit, and the unit type determines how systemd treats it.

Type Suffix Purpose
Service .service Long-running daemons or one-shot tasks
Socket .socket IPC/network socket for on-demand activation
Timer .timer Scheduled activation (replaces cron)
Mount .mount Filesystem mount point (mirrors /etc/fstab)
Automount .automount On-access mount triggering
Path .path Filesystem path monitoring (triggers on change)
Slice .slice cgroup resource partitioning
Scope .scope Externally-created process grouping (e.g., user sessions)
Target .target Synchronization point / grouping
Device .device Kernel device exposure
Swap .swap Swap device or file
Snapshot .snapshot Runtime-only saved state (legacy, rarely used)

Path Units

Path units watch the filesystem and trigger a paired service when a condition is met:

# /etc/systemd/system/invoice-watcher.path
[Unit]
Description=Watch for new invoice files

[Path]
PathExistsGlob=/var/spool/invoices/*.pdf
MakeDirectory=yes

[Install]
WantedBy=multi-user.target

When a PDF lands in /var/spool/invoices/, systemd starts invoice-watcher.service automatically.

Slice Units

Slices partition the cgroup tree. Every service runs inside a slice.

-.slice (root)
  +- system.slice     (system services)
  +- user.slice       (user sessions)
  |    +- user-1000.slice
  +- machine.slice    (VMs and containers)

Custom slices let you cap resource usage for groups of services:

# /etc/systemd/system/batch-jobs.slice
[Slice]
CPUQuota=200%
MemoryMax=8G
IOWeight=50

Scope Units

Scopes wrap processes that systemd did not start itself. You cannot create scope units from unit files -- they are created at runtime via the systemd API. Common example: systemd-run --scope wraps a command in a transient scope.

systemd-run --scope -p MemoryMax=1G --unit=my-migration \
  /usr/local/bin/db-migrate --full

Unit File Anatomy

Every unit file has up to three main sections. The section names match the unit type: [Service] for services, [Timer] for timers, [Socket] for sockets, etc. [Unit] and [Install] are universal.

[Unit] Section

Metadata and dependency declarations. Present in all unit types.

[Unit]
Description=Application API Server
Documentation=https://internal.wiki/api-server
After=network-online.target postgresql.service redis.service
Requires=postgresql.service
Wants=redis.service
BindsTo=container-runtime.service
Conflicts=legacy-api.service
ConditionPathExists=/etc/api-server/config.yaml
AssertFileIsExecutable=/usr/local/bin/api-server

Key directives:

Directive Meaning
After= / Before= Ordering only -- does not create a dependency
Requires= Hard dependency: if it fails, we fail too
Wants= Soft dependency: try to start it, but proceed if it fails
BindsTo= Like Requires, plus stop us if it stops
Conflicts= Stop the conflicting unit when we start
ConditionPathExists= Skip activation silently if path missing
AssertFileIsExecutable= Fail activation loudly if binary not executable

After= and Requires= are independent. Requires=postgresql.service without After=postgresql.service starts both in parallel -- the app may start before the database is ready. You almost always need both.

[Service] Section

Defines how the process is started, stopped, and supervised.

[Service]
Type=notify
ExecStartPre=/usr/local/bin/api-server --validate-config
ExecStartPre=+/usr/bin/mkdir -p /var/run/api-server
ExecStart=/usr/local/bin/api-server \
  --config /etc/api-server/config.yaml \
  --listen 0.0.0.0:8080
ExecStartPost=/usr/local/bin/register-consul.sh
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/usr/local/bin/api-server --graceful-shutdown
ExecStopPost=/usr/local/bin/deregister-consul.sh
Restart=on-failure
RestartSec=5
TimeoutStartSec=30
TimeoutStopSec=60
WatchdogSec=30
User=apiserver
Group=apiserver
WorkingDirectory=/var/lib/api-server
EnvironmentFile=/etc/api-server/env
Environment="LOG_LEVEL=info" "GOMAXPROCS=4"
StandardOutput=journal
StandardError=journal
SyslogIdentifier=api-server

ExecStart, ExecStartPre, ExecStop

  • ExecStartPre= runs before the main process. Use it for config validation, directory creation, or certificate renewal checks. Multiple ExecStartPre= lines execute in order.
  • The + prefix (ExecStartPre=+/usr/bin/mkdir ...) runs that specific command as root even if User= is set to something else.
  • ExecStart= is the main process. Only one ExecStart= per service (except Type=oneshot, which allows multiple).
  • ExecStartPost= runs after the main process is considered started.
  • ExecReload= defines what happens on systemctl reload. Conventionally sends SIGHUP.
  • ExecStop= defines graceful shutdown. If omitted, systemd sends SIGTERM then SIGKILL after TimeoutStopSec.
  • ExecStopPost= runs after the service stops, regardless of how it stopped. Use it for cleanup.

The - Prefix

A - prefix means "do not fail the unit if this command fails":

ExecStartPre=-/usr/local/bin/optional-setup.sh

If optional-setup.sh exits non-zero, startup continues.

[Install] Section

Controls what happens when you run systemctl enable.

[Install]
WantedBy=multi-user.target
RequiredBy=critical-app.target
Alias=my-api.service
Also=api-server-metrics.service api-server-healthcheck.timer
Directive Meaning
WantedBy= Create a .wants/ symlink in this target's directory
RequiredBy= Create a .requires/ symlink (hard dependency)
Alias= Create an alias symlink for this unit
Also= Enable these units when this unit is enabled

If WantedBy= is missing, systemctl enable creates no symlinks and the service never starts at boot. This is one of the most common silent failures.

Service Type= Deep Dive

The Type= directive tells systemd how to determine when the service is "ready."

Type=simple (default)

systemd considers the service started immediately after fork(). The process must stay in the foreground.

[Service]
Type=simple
ExecStart=/usr/bin/python3 /opt/app/server.py

Use for: most modern services that do not fork.

Caveat: systemd considers the unit "active" before the process has even called exec(). If the binary does not exist, dependent services may have already started. Prefer Type=exec on systemd 240+.

Type=exec

Like simple, but systemd waits until exec() succeeds. If the binary is missing, the unit fails immediately instead of appearing "active" briefly.

[Service]
Type=exec
ExecStart=/usr/local/bin/myapp

Use for: new services on modern systemd. Strictly better than simple.

Type=forking

For legacy daemons that fork into the background. systemd waits for the parent process to exit, then considers the service started. The child process must be identifiable.

[Service]
Type=forking
PIDFile=/var/run/myapp.pid
ExecStart=/usr/sbin/myapp -d

Use for: traditional C daemons that daemonize themselves. Many provide a -D or --no-daemon flag -- prefer that with Type=simple or Type=exec instead.

Type=oneshot

For tasks that run and exit. systemd waits for the process to finish before considering the unit "active."

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/bin/setup-iptables.sh
ExecStop=/usr/local/bin/teardown-iptables.sh

RemainAfterExit=yes keeps the unit in "active" state after the process exits. Without it, the unit goes to "inactive" immediately.

Use for: initialization scripts, one-time setup, database migrations.

Type=notify

The process signals readiness to systemd using sd_notify(). systemd considers the service started only after receiving the notification.

[Service]
Type=notify
NotifyAccess=main
ExecStart=/usr/local/bin/myapp

Use for: services that perform significant initialization before they are ready to serve (loading data, warming caches, binding ports). The service must call sd_notify("READY=1") in its code.

For shell scripts, use systemd-notify --ready:

#!/bin/bash
# Do initialization
setup_database
load_cache
# Signal ready
systemd-notify --ready
# Continue running
exec main_loop

Type=dbus

Like notify, but readiness is signaled by acquiring a D-Bus bus name.

[Service]
Type=dbus
BusName=org.freedesktop.NetworkManager
ExecStart=/usr/sbin/NetworkManager

Use for: desktop/system services that register on D-Bus.

Type=idle

Like simple, but systemd delays starting the process until all active jobs are finished. Used for console output ordering (e.g., getty).

systemctl Command Reference

Lifecycle Commands

# Start, stop, restart
systemctl start nginx.service
systemctl stop nginx.service
systemctl restart nginx.service      # stop + start

# Reload configuration (sends SIGHUP or runs ExecReload)
systemctl reload nginx.service

# Reload if supported, restart otherwise
systemctl reload-or-restart nginx.service

# Reload unit file definitions (not the service)
systemctl daemon-reload

Enable / Disable / Mask

# Enable at boot (creates symlinks per [Install])
systemctl enable nginx.service

# Enable and start in one command
systemctl enable --now nginx.service

# Disable at boot (removes symlinks)
systemctl disable nginx.service

# Mask: symlink to /dev/null, preventing start even as dependency
systemctl mask nginx.service

# Unmask: remove the /dev/null symlink
systemctl unmask nginx.service

Status and Querying

# Detailed status with recent logs
systemctl status nginx.service

# Quick boolean checks (good for scripts)
systemctl is-active nginx.service    # active or inactive
systemctl is-enabled nginx.service   # enabled, disabled, masked, static
systemctl is-failed nginx.service    # failed or not

# Show all loaded units
systemctl list-units

# Show all unit files (including unloaded)
systemctl list-unit-files

# Show only failed units
systemctl list-units --failed

# Show unit file content
systemctl cat nginx.service

# Show all properties
systemctl show nginx.service

# Show specific property
systemctl show nginx.service -p MainPID -p MemoryCurrent

# Show dependencies
systemctl list-dependencies nginx.service
systemctl list-dependencies nginx.service --reverse

daemon-reload

After any change to unit files on disk, run:

systemctl daemon-reload

This reloads all unit file definitions. Without it, systemd uses stale cached versions. This is the single most forgotten systemd command.

journalctl Deep Usage

Basic Filtering

# Logs for a specific unit
journalctl -u nginx.service

# Follow mode (like tail -f)
journalctl -u nginx -f

# Last N lines
journalctl -u nginx -n 200

# Current boot only
journalctl -b

# Previous boot
journalctl -b -1

# List stored boots
journalctl --list-boots

Time-Based Queries

# Since a specific time
journalctl --since "2025-03-15 09:00:00"

# Until a specific time
journalctl --until "2025-03-15 10:30:00"

# Combined range
journalctl -u nginx --since "2025-03-15 09:00" --until "2025-03-15 10:00"

# Relative time
journalctl --since "1 hour ago"
journalctl --since "30 minutes ago" -u myapp

Priority Filtering

syslog priorities, from most to least severe:

Priority Keyword Meaning
0 emerg System is unusable
1 alert Action must be taken immediately
2 crit Critical conditions
3 err Error conditions
4 warning Warning conditions
5 notice Normal but significant
6 info Informational
7 debug Debug-level messages 
# Errors and above
journalctl -p err

# Errors and above for a specific unit
journalctl -p err -u nginx

# Range of priorities
journalctl -p warning..err

Kernel Messages

# Kernel ring buffer (like dmesg, but indexed)
journalctl -k

# Kernel messages from previous boot
journalctl -k -b -1

# Kernel errors only
journalctl -k -p err

Output Formats

# JSON output (one JSON object per line)
journalctl -u nginx -o json --no-pager

# Pretty-printed JSON
journalctl -u nginx -o json-pretty -n 5

# Short with precise timestamps
journalctl -u nginx -o short-precise

# Full structured output (all fields)
journalctl -u nginx -o verbose

# Export format (for import to another system)
journalctl -u nginx -o export > nginx-logs.export

Disk Management

# Check journal disk usage
journalctl --disk-usage

# Remove old entries to fit within size limit
journalctl --vacuum-size=500M

# Remove entries older than a time limit
journalctl --vacuum-time=30d

# Remove entries to keep only N files
journalctl --vacuum-files=5

# Verify journal integrity
journalctl --verify

Persistent Journal Configuration

Edit /etc/systemd/journald.conf:

[Journal]
Storage=persistent
SystemMaxUse=2G
SystemKeepFree=1G
SystemMaxFileSize=128M
MaxRetentionSec=3month
ForwardToSyslog=no
Compress=yes

After editing, restart journald:

systemctl restart systemd-journald

Advanced Queries

# Messages from a specific PID
journalctl _PID=1234

# Messages from a specific binary
journalctl _COMM=sshd

# Messages from a specific UID
journalctl _UID=1000

# Multiple conditions (AND within one field, OR across fields)
journalctl _SYSTEMD_UNIT=nginx.service _SYSTEMD_UNIT=php-fpm.service

# Combine with grep for complex patterns
journalctl -u myapp --no-pager | grep -i "connection refused"

# Show entries with specific syslog facility
journalctl SYSLOG_FACILITY=10

Timer Units

Timer units replace cron with better logging, dependency management, and missed-run handling.

OnCalendar Syntax

OnCalendar= uses a calendar expression format:

DayOfWeek Year-Month-Day Hour:Minute:Second

Examples:

OnCalendar=*-*-* 02:00:00         # Daily at 2:00 AM
OnCalendar=Mon *-*-* 09:00:00     # Every Monday at 9 AM
OnCalendar=*-*-01 00:00:00        # First of every month at midnight
OnCalendar=*-01,04,07,10-01       # Quarterly on the 1st
OnCalendar=hourly                 # Shorthand for *-*-* *:00:00
OnCalendar=daily                  # Shorthand for *-*-* 00:00:00
OnCalendar=weekly                 # Mon *-*-* 00:00:00
OnCalendar=*:0/15                 # Every 15 minutes
OnCalendar=*-*-* 08..17:00:00     # Every hour from 8 AM to 5 PM

Validate calendar expressions before deploying:

systemd-analyze calendar "Mon *-*-* 02:00:00"
systemd-analyze calendar "*:0/15"
systemd-analyze calendar "daily" --iterations=5

Monotonic Timers

[Timer]
OnBootSec=5min          # 5 minutes after boot
OnUnitActiveSec=1h      # 1 hour after the service last activated
OnStartupSec=10min      # 10 minutes after systemd started

Persistent Timer

[Timer]
OnCalendar=daily
Persistent=true

Persistent=true means: if the timer was supposed to fire while the system was off, fire it immediately on next boot. Without this, missed runs are silently lost.

RandomizedDelaySec

[Timer]
OnCalendar=*-*-* 02:00:00
RandomizedDelaySec=600

Adds a random delay up to 10 minutes. Prevents thundering herd when many machines have the same timer.

Complete Timer Example

# /etc/systemd/system/db-backup.timer
[Unit]
Description=Database backup timer

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=300
AccuracySec=1min

[Install]
WantedBy=timers.target

# /etc/systemd/system/db-backup.service
[Unit]
Description=Database backup
After=postgresql.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup-db.sh
User=backup
Nice=19
IOSchedulingClass=idle

systemctl enable --now db-backup.timer
systemctl list-timers --all

Socket Activation

Socket activation lets systemd listen on a socket and start the service only when a connection arrives. Benefits: faster boot (services start on demand), zero-downtime restarts (systemd holds the socket during restart).

# /etc/systemd/system/myapp.socket
[Unit]
Description=MyApp Socket

[Socket]
ListenStream=8080
Accept=no

# /etc/systemd/system/myapp.service
[Unit]
Description=MyApp Server
Requires=myapp.socket

[Service]
Type=notify
ExecStart=/usr/local/bin/myapp

The service receives the socket as file descriptor 3. In the application:

import socket
import os

# systemd passes the socket as FD 3
fd = 3
sock = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)
sock.listen()

Accept=no means the service gets the listening socket and handles connections itself (most common). Accept=yes means systemd accepts each connection and spawns a new service instance per connection.

Enable the socket, not the service:

systemctl enable --now myapp.socket
# myapp.service starts automatically on first connection

Template Units

Template units use an @ in the filename. They create parameterized instances.

# /etc/systemd/system/container@.service
[Unit]
Description=Container %i
After=docker.service

[Service]
ExecStart=/usr/bin/docker start -a %i
ExecStop=/usr/bin/docker stop %i
Restart=on-failure

[Install]
WantedBy=multi-user.target

%i is replaced by the instance name:

systemctl start container@redis.service
systemctl start container@postgres.service
systemctl enable container@redis.service

Common specifiers:

Specifier Meaning
%i Instance name (unescaped)
%I Instance name (escaped)
%n Full unit name
%N Full unit name (escaped)
%p Prefix (unit name without type suffix and instance)
%H Hostname

Template units are essential for managing multiple instances of the same service (containers, per-tenant workers, per-port listeners).

Drop-in Directories

Drop-in overrides let you modify vendor unit files without replacing them.

Using systemctl edit

# Create a drop-in override (opens editor)
systemctl edit nginx.service
# Creates: /etc/systemd/system/nginx.service.d/override.conf

# Replace the entire unit file
systemctl edit --full nginx.service
# Creates: /etc/systemd/system/nginx.service

Manual Drop-in Structure

/etc/systemd/system/nginx.service.d/
  +- 10-limits.conf
  +- 20-security.conf
  +- 30-logging.conf

Files are applied in lexicographic order. Numbering prefixes control order.

Override Rules

Most directives replace the vendor value. But some are additive:

Behavior Directives
Replace ExecStart, ExecStop, User, Type, Restart
Additive Environment, ExecStartPre, ExecStartPost

To replace ExecStart= in a drop-in, you must first clear it:

# /etc/systemd/system/nginx.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/sbin/nginx -g 'daemon off;' -c /etc/nginx/custom.conf

The empty ExecStart= clears the vendor value. The second line sets the new value. Without the clearing line, you get an error because service units allow only one ExecStart=.

systemd-analyze

Boot Performance

# Total boot time
systemd-analyze

# Time per unit, sorted slowest first
systemd-analyze blame

# Dependency chain with timing
systemd-analyze critical-chain

# Critical chain for a specific unit
systemd-analyze critical-chain nginx.service

# SVG boot chart
systemd-analyze plot > boot.svg

Unit Verification

# Check unit file syntax and dependencies
systemd-analyze verify myapp.service

# Check all loaded units
systemd-analyze verify --man=no /etc/systemd/system/*.service

Security Auditing

# Score a service's sandboxing (0 = fully sandboxed, 10 = fully exposed)
systemd-analyze security nginx.service

# Score all services
systemd-analyze security

# Detailed breakdown
systemd-analyze security nginx.service --no-pager

This command scores each service on a 0-10 exposure scale. A score above 7 means the service has significant access to the system. Use it to identify services that need sandboxing.

Calendar Expression Testing

# Parse and normalize a calendar expression
systemd-analyze calendar "Mon..Fri *-*-* 09:00"

# Show next N trigger times
systemd-analyze calendar "daily" --iterations=10

Resource Control (cgroups v2)

systemd uses cgroups to enforce resource limits per service.

CPU

[Service]
CPUQuota=150%           # 1.5 cores max
CPUWeight=100           # Relative weight (default 100)
AllowedCPUs=0-3         # Pin to specific CPUs

Memory

[Service]
MemoryMax=2G            # Hard limit (OOM kill if exceeded)
MemoryHigh=1536M        # Soft limit (throttling, reclaim pressure)
MemorySwapMax=0         # Disable swap for this service
MemoryMin=256M          # Guaranteed minimum (protect from reclaim)

I/O

[Service]
IOWeight=50             # Relative I/O weight (default 100)
IOReadBandwidthMax=/dev/sda 50M    # Per-device read limit
IOWriteBandwidthMax=/dev/sda 20M   # Per-device write limit
IODeviceLatencyTargetSec=/dev/sda 25ms

Tasks (Processes)

[Service]
TasksMax=512            # Max number of processes/threads

Monitoring Resource Usage

# Live cgroup resource monitor
systemd-cgtop

# Current memory usage of a service
systemctl show nginx -p MemoryCurrent

# Current CPU usage
systemctl show nginx -p CPUUsageNSec

# View the cgroup tree
systemd-cgls

# Detailed cgroup info
systemctl status nginx   # includes cgroup tree

Sandboxing Directives

systemd provides powerful sandboxing without containers. These directives restrict what a service can access.

Filesystem Protection

[Service]
ProtectSystem=strict        # Mount / as read-only except /dev, /proc, /sys
                            # Values: true (ro /usr, /boot), full (+ /etc), strict (everything)
ProtectHome=yes             # Hide /home, /root, /run/user
                            # Values: yes (inaccessible), read-only, tmpfs
PrivateTmp=yes              # Isolated /tmp and /var/tmp
ReadWritePaths=/var/lib/myapp /var/log/myapp
ReadOnlyPaths=/etc/myapp
InaccessiblePaths=/etc/shadow
TemporaryFileSystem=/var:ro
BindPaths=/data/shared:/mnt/data

Process Restrictions

[Service]
NoNewPrivileges=yes         # Cannot gain privileges via setuid/setgid/caps
PrivateDevices=yes          # No access to physical devices
PrivateUsers=yes            # User namespace isolation
ProtectKernelTunables=yes   # Read-only /proc/sys, /sys
ProtectKernelModules=yes    # Cannot load kernel modules
ProtectKernelLogs=yes       # Cannot read kernel logs
ProtectControlGroups=yes    # Read-only cgroup filesystem
ProtectClock=yes            # Cannot change system clock
ProtectHostname=yes         # Cannot change hostname
LockPersonality=yes         # Lock execution domain
RestrictRealtime=yes        # Cannot acquire realtime scheduling
RestrictSUIDSGID=yes        # Cannot create setuid/setgid files

Network Restrictions

[Service]
PrivateNetwork=yes          # Isolated network namespace (loopback only)
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
IPAddressDeny=any
IPAddressAllow=10.0.0.0/8 127.0.0.0/8

System Call Filtering

[Service]
SystemCallFilter=@system-service    # Allow only system-service calls
SystemCallFilter=~@mount @clock     # Deny mount and clock syscalls
SystemCallErrorNumber=EPERM         # Return EPERM instead of killing

Capability Restrictions

[Service]
CapabilityBoundingSet=CAP_NET_BIND_SERVICE CAP_DAC_READ_SEARCH
AmbientCapabilities=CAP_NET_BIND_SERVICE

Production-Ready Hardened Service

Combining the above into a real service:

[Unit]
Description=Production API Server
After=network-online.target postgresql.service
Wants=network-online.target
Requires=postgresql.service

[Service]
Type=notify
ExecStart=/usr/local/bin/api-server --config /etc/api-server/config.yaml
Restart=on-failure
RestartSec=5
User=apiserver
Group=apiserver

# Resource limits
MemoryMax=2G
MemoryHigh=1536M
CPUQuota=200%
TasksMax=256

# Sandboxing
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
NoNewPrivileges=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
SystemCallFilter=@system-service
ReadWritePaths=/var/lib/api-server /var/log/api-server
ReadOnlyPaths=/etc/api-server

[Install]
WantedBy=multi-user.target

Check the security score:

systemd-analyze security api-server.service

A well-hardened service should score below 4.0 on the exposure scale.

User Services

systemd supports per-user service managers. User units live in ~/.config/systemd/user/ and run without root privileges.

# ~/.config/systemd/user/dev-proxy.service
[Unit]
Description=Development proxy

[Service]
ExecStart=/usr/local/bin/dev-proxy --port 3000
Restart=on-failure

[Install]
WantedBy=default.target

# Manage user services (no sudo)
systemctl --user start dev-proxy
systemctl --user enable --now dev-proxy
systemctl --user status dev-proxy
journalctl --user -u dev-proxy

# Enable lingering (services run without login session)
loginctl enable-linger username

User services are ideal for development tools, personal cron replacements, and services that do not need system-wide scope.

Wiki Navigation

Prerequisites

Related Content