Systemd Cheat Sheet

Name origin: systemd = "system daemon." Created by Lennart Poettering and Kay Sievers (Red Hat) in 2010 to replace SysVinit. The controversial "d" follows Unix naming convention (httpd, sshd, crond). systemd is not just an init system — it manages services, logging (journald), networking (networkd), DNS (resolved), time sync (timesyncd), and login sessions (logind). It boots faster than SysVinit by starting services in parallel based on dependency graphs.

systemctl — Service Management

# Start / stop / restart
systemctl start nginx
systemctl stop nginx
systemctl restart nginx
systemctl reload nginx                 # Reload config without restart

# Enable / disable (auto-start on boot)
systemctl enable nginx
systemctl disable nginx
systemctl enable --now nginx           # Enable AND start immediately

# Status and inspection
systemctl status nginx                 # Status + recent logs
systemctl is-active nginx              # Just "active" or "inactive"
systemctl is-enabled nginx             # "enabled" or "disabled"
systemctl is-failed nginx              # "failed" or "active"
systemctl show nginx                   # All properties
systemctl show nginx -p MainPID        # Specific property
systemctl cat nginx                    # Show unit file contents

# List services
systemctl list-units --type=service                    # Running services
systemctl list-units --type=service --state=failed     # Failed services
systemctl list-unit-files --type=service               # All installed services

journalctl — Log Queries

# View logs for a service
journalctl -u nginx                    # All logs for unit
journalctl -u nginx -e                 # Jump to end (most recent)
journalctl -u nginx -f                 # Follow (like tail -f)
journalctl -u nginx --since "1 hour ago"
journalctl -u nginx --since "2024-01-15 09:00" --until "2024-01-15 10:00"

# Filter by priority
journalctl -p err                      # Errors and above
journalctl -p warning -u nginx         # Warnings and above for nginx
# Priority levels: emerg, alert, crit, err, warning, notice, info, debug

# Kernel messages
journalctl -k                          # Kernel messages (like dmesg)
journalctl -k -b -1                    # Kernel messages from previous boot

# Boot logs
journalctl -b                          # Current boot
journalctl -b -1                       # Previous boot
journalctl --list-boots                # List all recorded boots

# Output formats
journalctl -u nginx -o json-pretty     # JSON format
journalctl -u nginx -o short-precise   # Precise timestamps
journalctl -u nginx --no-pager         # Disable paging (for scripts)
journalctl -u nginx -n 50              # Last 50 lines

# Disk usage
journalctl --disk-usage                # How much space logs consume
journalctl --vacuum-size=500M          # Shrink to 500MB
journalctl --vacuum-time=7d            # Keep only last 7 days

Gotcha: journalctl -u <service> -f follows logs but only shows the journal — if the service writes to a file (e.g., /var/log/myapp.log), journalctl will not show it. Check the service's StandardOutput= and StandardError= settings with systemctl cat <service>. For container-style services, stdout/stderr logging via journald is preferred.

Unit File Syntax

# Location: /etc/systemd/system/myapp.service
# (custom units go in /etc/systemd/system/, not /usr/lib/systemd/system/)

[Unit]
Description=My Application Server
Documentation=https://docs.example.com
After=network.target postgresql.service
Requires=postgresql.service
Wants=redis.service

[Service]
Type=simple
User=appuser
Group=appgroup
WorkingDirectory=/opt/myapp
Environment=NODE_ENV=production
EnvironmentFile=/etc/myapp/env
ExecStartPre=/opt/myapp/pre-start.sh
ExecStart=/usr/bin/node /opt/myapp/server.js
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
StartLimitBurst=5
StartLimitIntervalSec=60

# Security hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/myapp /var/log/myapp
PrivateTmp=true

# Resource limits
LimitNOFILE=65536
MemoryMax=2G
CPUQuota=200%

[Install]
WantedBy=multi-user.target

Service Types

Type	Behavior
simple	Default. Process in ExecStart is the main process.
exec	Like simple, but systemd waits for exec() to complete.
forking	Process forks and parent exits. Set PIDFile=.
oneshot	Process exits after doing its job. Good for scripts.
notify	Process sends sd_notify() when ready.
idle	Like simple, but waits for other jobs to finish.

Restart Policies

Policy	Restarts On
no	Never (default)
on-failure	Non-zero exit, signal, timeout, watchdog
on-abnormal	Signal, timeout, watchdog
on-abort	Signal only
always	Any exit, including clean (exit 0)

Dependency Directives

Directive	Meaning
After=	Start after the listed unit (ordering only)
Before=	Start before the listed unit (ordering only)
Requires=	Hard dependency — if it fails, this unit fails
Wants=	Soft dependency — if it fails, this unit still starts
BindsTo=	Like Requires, but also stops when dependency stops
Conflicts=	Cannot run at the same time as listed unit

Remember: After editing a unit file, you MUST run systemctl daemon-reload before the changes take effect. Without it, systemd uses the cached (old) version. This is the #1 "why isn't my change working?" mistake. Mnemonic: "Edit, reload, restart" — three steps, always in that order.

Default trap: Restart=on-failure does NOT restart on clean exit (exit code 0). If your app exits cleanly but should always be running, use Restart=always. The difference matters: a graceful shutdown (SIGTERM handled, exit 0) will not trigger on-failure restart.

Timer Units (systemd cron alternative)

# /etc/systemd/system/backup.timer
[Unit]
Description=Daily backup timer

[Timer]
OnCalendar=*-*-* 02:00:00          # Daily at 2 AM
# OnCalendar=hourly                # Every hour
# OnCalendar=Mon *-*-* 09:00:00    # Every Monday at 9 AM
# OnBootSec=5min                   # 5 minutes after boot
# OnUnitActiveSec=1h               # 1 hour after last activation
Persistent=true                     # Run missed events after downtime
RandomizedDelaySec=300              # Jitter up to 5 minutes

[Install]
WantedBy=timers.target

# /etc/systemd/system/backup.service (companion service)
[Unit]
Description=Daily backup job

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
User=backup

# Timer management
systemctl enable --now backup.timer     # Enable and start timer
systemctl list-timers --all             # List all timers
systemctl status backup.timer           # Timer status
journalctl -u backup.service           # Logs from timer-triggered runs

OnCalendar Syntax

Expression	Meaning
minutely	Every minute
hourly	Every hour
daily	Every day at midnight
weekly	Every Monday at midnight
monthly	First day of month at midnight
*-*-* *:00:00	Every hour (explicit)
*-*-* *:*:00	Every minute (explicit)
Mon..Fri *-*-* 09:00:00	Weekdays at 9 AM
*-*-01 00:00:00	First of every month

# Validate calendar expression
systemd-analyze calendar "Mon *-*-* 09:00:00"
systemd-analyze calendar "daily" --iterations=5

Common Operations

# Reload after editing unit files
systemctl daemon-reload

# Mask a service (prevent start, even by dependencies)
systemctl mask nginx
systemctl unmask nginx

# Check boot time
systemd-analyze                        # Total boot time
systemd-analyze blame                  # Time per unit
systemd-analyze critical-chain         # Critical path

# View dependencies
systemctl list-dependencies nginx
systemctl list-dependencies --reverse nginx   # What depends on nginx

# Kill a service's processes
systemctl kill nginx
systemctl kill -s SIGKILL nginx        # Force kill

# Reset failed state
systemctl reset-failed nginx

# Edit a unit (creates override)
systemctl edit nginx                   # Creates drop-in override
systemctl edit --full nginx            # Edit the full unit file
# Overrides go to /etc/systemd/system/nginx.service.d/override.conf

Targets (Runlevels)

Target	Old Runlevel	Description
poweroff.target	0	Shut down
rescue.target	1	Single user
multi-user.target	3	Multi-user, no GUI
graphical.target	5	Multi-user with GUI
reboot.target	6	Reboot

# Check current target
systemctl get-default

# Set default target
systemctl set-default multi-user.target

# Switch target now
systemctl isolate rescue.target

Quick Recipes

# Find why a service failed
systemctl status myapp && journalctl -u myapp -e --no-pager -n 50

# Watch service restarts in real time
journalctl -u myapp -f -o short-precise

# Create a simple oneshot service
cat > /etc/systemd/system/cleanup.service << 'EOF'
[Unit]
Description=Cleanup temp files
[Service]
Type=oneshot
ExecStart=/usr/bin/find /tmp -mtime +7 -delete
EOF
systemctl daemon-reload && systemctl start cleanup

# Check resource usage of a service
systemctl show myapp -p MemoryCurrent,CPUUsageNSec,TasksCurrent