Skip to content

Portal | Level: L1: Foundations | Topics: Linux Logging, Linux Fundamentals, systemd, Logging | Domain: Linux

Linux Logging — Primer

Why This Matters

Logs are the black box recorder of your systems. When a service crashes, when a security incident occurs, when performance degrades, when a deployment fails — logs are usually the first (and sometimes only) place you can find out what happened. If you don't understand how Linux logging works, you're debugging with your eyes closed.

This primer covers the complete Linux logging stack: the kernel ring buffer, syslog infrastructure, systemd's journal, log rotation, audit logging, and practical patterns for operating log infrastructure at scale.

The Linux Logging Architecture

Modern Linux systems have multiple overlapping logging systems:

┌─────────────────────────────────────────────────────┐
│                    Applications                      │
│  (write to stdout, syslog, or log files directly)    │
└──────────────┬──────────────┬───────────────────────┘
               │              │
       ┌───────▼───────┐     │
       │   journald     │     │    Direct file writes
       │  (systemd)     │     │    (/var/log/myapp/)
       │                │     │
       └───────┬───────┘     │
               │              │
       ┌───────▼───────┐     │
       │    rsyslog     │◀────┘
       │  (or syslog-ng)│
       └───────┬───────┘
               │
       ┌───────▼───────┐
       │  /var/log/*    │
       │  (log files)   │
       └───────────────┘
               │
       ┌───────▼───────┐
       │  logrotate     │
       │  (rotation)    │
       └───────────────┘

Additionally, the kernel has its own ring buffer (accessible via dmesg), and the audit subsystem (auditd) provides security-focused logging.

Kernel Logging: dmesg and the Ring Buffer

The kernel maintains a circular buffer of messages. These include hardware detection, driver loading, filesystem errors, OOM kills, and other kernel-level events.

# View kernel ring buffer
$ dmesg
[    0.000000] Linux version 5.15.0-91-generic ...
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-5.15.0-91-generic ...
[    3.456789] EXT4-fs (sda2): mounted filesystem with ordered data mode
[  345.678901] e1000e 0000:00:19.0 eth0: Link is Up 1000 Mbps Full Duplex
[12345.678901] Out of memory: Killed process 1234 (java)

# Human-readable timestamps
$ dmesg -T
[Fri Mar 19 10:45:22 2026] e1000e 0000:00:19.0 eth0: Link is Up

# Filter by severity
$ dmesg --level=err
$ dmesg --level=warn,err,crit

# Follow (watch for new messages)
$ dmesg -w

# Clear the ring buffer (rarely needed)
$ sudo dmesg -c

The ring buffer is fixed-size (default 256 KB, adjustable via log_buf_len kernel parameter). Once full, old messages are overwritten. This is why critical kernel messages should also flow through journald/syslog to persistent storage.

Kernel Log Levels

Level Name Example
0 EMERG System is unusable
1 ALERT Action must be taken immediately
2 CRIT Critical conditions (hardware failures)
3 ERR Error conditions
4 WARNING Warning conditions
5 NOTICE Normal but significant
6 INFO Informational
7 DEBUG Debug-level messages 
# View/set console log level
$ cat /proc/sys/kernel/printk
4    4    1    7
# Current | Default | Minimum | Boot-time default
# Only messages with severity < 4 (WARNING) go to console

The Syslog Protocol

Syslog is both a protocol (RFC 5424) and the traditional Unix logging infrastructure. Messages have two key properties:

Facility

The facility indicates what type of program generated the message:

Code Facility Typical use
0 kern Kernel messages
1 user User-level messages
2 mail Mail system
3 daemon System daemons
4 auth Security/authorization
10 authpriv Security/authorization (private)
16-23 local0-local7 Custom/local use

Severity (Priority)

Same as kernel log levels (0=EMERG through 7=DEBUG). The combination of facility and severity is the priority: priority = facility × 8 + severity.

The logger Command

logger sends messages to syslog from the command line — invaluable for testing and scripting:

# Basic message
$ logger "Deploy completed successfully"

# With facility and severity
$ logger -p local0.info "Application started on port 8080"

# With a tag (identifies the program)
$ logger -t myapp "Connection pool initialized"

# In scripts
#!/bin/bash
logger -t deploy-script "Starting deployment of $APP_VERSION"
# ... deploy steps ...
if [ $? -eq 0 ]; then
    logger -t deploy-script -p local0.info "Deployment successful"
else
    logger -t deploy-script -p local0.err "Deployment FAILED"
fi

rsyslog

rsyslog is the standard syslog daemon on most Linux distributions. It receives syslog messages (from local processes, journald, and remote sources), processes them through rules, and writes them to files, databases, or remote servers.

Configuration

Main config: /etc/rsyslog.conf with drop-in files in /etc/rsyslog.d/.

# /etc/rsyslog.conf structure:

#### MODULES ####
module(load="imuxsock")    # Local system socket (/dev/log)
module(load="imklog")      # Kernel log messages
module(load="imjournal")   # systemd journal input
module(load="imtcp")       # TCP syslog receiver (for remote logs)

#### RULES ####
# Facility.Severity    Action
auth,authpriv.*         /var/log/auth.log
*.*;auth,authpriv.none  /var/log/syslog
kern.*                  /var/log/kern.log
mail.*                  /var/log/mail.log
cron.*                  /var/log/cron.log

# Emergency messages to all logged-in users
*.emerg                 :omusrmsg:*

# Custom application logging
local0.*                /var/log/myapp.log

# Forward to remote server
*.* @@logserver.example.com:514    # @@ = TCP, @ = UDP

rsyslog Templates

Templates control the format of log output:

# Traditional syslog format
template(name="TraditionalFormat" type="string"
    string="%timegenerated% %HOSTNAME% %syslogtag%%msg%\n")

# JSON format for log aggregation
template(name="JSONFormat" type="string"
    string="{\"timestamp\":\"%timegenerated:::date-rfc3339%\",\"host\":\"%HOSTNAME%\",\"program\":\"%programname%\",\"severity\":\"%syslogseverity-text%\",\"message\":\"%msg:::json%\"}\n")

# Use a template
local0.* action(type="omfile" file="/var/log/myapp.json" template="JSONFormat")

Rate Limiting

rsyslog has built-in rate limiting that can silently drop messages:

# /etc/rsyslog.conf
# Default: 200 messages per 5 seconds per source
$SystemLogRateLimitInterval 5
$SystemLogRateLimitBurst 200

# Disable rate limiting entirely (high-volume systems)
$SystemLogRateLimitInterval 0

# Or per-module
module(load="imjournal" ratelimit.interval="0")

syslog-ng

syslog-ng is an alternative to rsyslog, popular in some environments. Key differences:

  • More powerful filtering syntax
  • Better structured data handling
  • Simpler config for complex routing
# /etc/syslog-ng/syslog-ng.conf
source s_local {
    system();
    internal();
};

destination d_auth { file("/var/log/auth.log"); };
destination d_syslog { file("/var/log/syslog"); };
destination d_remote { tcp("logserver.example.com" port(514)); };

filter f_auth { facility(auth, authpriv); };
filter f_not_auth { not facility(auth, authpriv); };

log { source(s_local); filter(f_auth); destination(d_auth); };
log { source(s_local); filter(f_not_auth); destination(d_syslog); };
log { source(s_local); destination(d_remote); };

systemd Journal (journald)

systemd's journal is a structured, binary logging system that captures: - Service stdout/stderr - Syslog messages - Kernel messages - Audit messages

journalctl Basics

# View all logs (newest last)
$ journalctl

# Follow in real time (like tail -f)
$ journalctl -f

# Show logs from current boot
$ journalctl -b

# Show logs from previous boot
$ journalctl -b -1

# Show only errors and above
$ journalctl -p err

# Show warnings and above
$ journalctl -p warning

Filtering by Service

# Logs for a specific service
$ journalctl -u nginx.service

# Logs for a service, follow mode
$ journalctl -u nginx.service -f

# Logs for multiple services
$ journalctl -u nginx.service -u php-fpm.service

# Since last service restart
$ journalctl -u nginx.service --since "$(systemctl show -p ActiveEnterTimestamp nginx.service | cut -d= -f2)"

Filtering by Time

# Logs since a specific time
$ journalctl --since "2026-03-19 10:00:00"

# Logs in a time range
$ journalctl --since "2026-03-19 10:00:00" --until "2026-03-19 11:00:00"

# Relative time
$ journalctl --since "1 hour ago"
$ journalctl --since "30 minutes ago"
$ journalctl --since yesterday

Output Formats

# Short format (default, like syslog)
$ journalctl -u nginx -o short

# Verbose (all metadata fields)
$ journalctl -u nginx -o verbose

# JSON (one object per line — great for piping to jq)
$ journalctl -u nginx -o json | jq .

# JSON pretty-printed
$ journalctl -u nginx -o json-pretty

# Cat (message only, no timestamps)
$ journalctl -u nginx -o cat

# Export (binary, for transport to another system)
$ journalctl -u nginx -o export > nginx-logs.journal

Structured Fields

The journal stores structured metadata with every message:

# Show all fields for a message
$ journalctl -u nginx -o verbose -n 1
Thu 2026-03-19 10:45:22.123456 UTC [s=abc123;i=1234;b=def456...]
    _SYSTEMD_UNIT=nginx.service
    _PID=12345
    _UID=0
    _GID=0
    _COMM=nginx
    _EXE=/usr/sbin/nginx
    _HOSTNAME=webserver01
    PRIORITY=6
    SYSLOG_FACILITY=3
    MESSAGE=10.0.0.1 - - [19/Mar/2026:10:45:22 +0000] "GET / HTTP/1.1" 200 612
    _TRANSPORT=stdout
    _STREAM_ID=xyz789

# Filter by arbitrary fields
$ journalctl _PID=12345
$ journalctl _COMM=sshd
$ journalctl _UID=1000
$ journalctl CONTAINER_NAME=myapp

Journal Storage Configuration

Journal configuration lives in /etc/systemd/journald.conf:

[Journal]
# Storage: volatile (memory only), persistent (/var/log/journal), auto (persistent if dir exists)
Storage=persistent

# Size limits
SystemMaxUse=2G          # Max total disk usage
SystemKeepFree=4G        # Keep this much free on the filesystem
SystemMaxFileSize=128M   # Max size per journal file
RuntimeMaxUse=256M       # Max memory usage (volatile mode)

# Rate limiting
RateLimitIntervalSec=30s
RateLimitBurst=10000

# Forwarding
ForwardToSyslog=yes      # Forward to rsyslog
ForwardToKMsg=no
ForwardToConsole=no

# Compression
Compress=yes

# Sealing (for tamper evidence)
Seal=yes

After changing journal config:

$ sudo systemctl restart systemd-journald

Persistent Journal Storage

By default on many distros, journal data is stored in /run/log/journal/ (volatile — lost on reboot). To persist:

$ sudo mkdir -p /var/log/journal
$ sudo systemd-tmpfiles --create --prefix /var/log/journal
$ sudo systemctl restart systemd-journald

# Verify
$ journalctl --disk-usage
Archived and active journals take up 456.0M in /var/log/journal.

/var/log Directory Structure

The traditional log directory:

File Source Content
/var/log/syslog (Debian) or /var/log/messages (RHEL) rsyslog General system messages
/var/log/auth.log (Debian) or /var/log/secure (RHEL) rsyslog Authentication, sudo, SSH
/var/log/kern.log rsyslog Kernel messages
/var/log/cron.log rsyslog Cron job execution
/var/log/mail.log rsyslog Mail system
/var/log/boot.log systemd Boot messages
/var/log/dmesg init scripts Kernel ring buffer at boot
/var/log/dpkg.log or /var/log/yum.log Package manager Package install/remove history
/var/log/lastlog login Last login per user (binary)
/var/log/wtmp login Login history (binary, use last to read)
/var/log/btmp login Failed login attempts (binary, use lastb)
/var/log/faillog login Failed login counter (binary, use faillog) 
# View login history
$ last -10
deploy   pts/0    10.0.0.50    Thu Mar 19 10:30   still logged in
deploy   pts/0    10.0.0.50    Wed Mar 18 14:15 - 17:30 (03:15)

# View failed logins
$ sudo lastb -10
baduser  ssh:notty  10.99.0.1    Thu Mar 19 10:29 - 10:29 (00:00)

logrotate

logrotate prevents log files from filling the disk by rotating, compressing, and deleting old logs.

Configuration

Main config: /etc/logrotate.conf. Per-application configs in /etc/logrotate.d/.

# /etc/logrotate.d/nginx
/var/log/nginx/*.log {
    daily               # Rotate daily (also: weekly, monthly)
    missingok           # Don't error if log file is missing
    rotate 14           # Keep 14 rotated files
    compress            # Gzip old logs
    delaycompress       # Compress previous rotation (not current)
    notifempty          # Don't rotate empty files
    create 0640 www-data adm    # Permissions for new log file
    sharedscripts       # Run postrotate once (not per file)
    postrotate
        # Signal nginx to reopen log files
        if [ -f /var/run/nginx.pid ]; then
            kill -USR1 $(cat /var/run/nginx.pid)
        fi
    endscript
}

Common logrotate Options

Option Meaning
daily/weekly/monthly Rotation frequency
rotate N Keep N rotated files
compress Gzip rotated files
delaycompress Wait one cycle before compressing
copytruncate Copy log then truncate original (for apps that don't reopen)
create MODE OWNER GROUP Permissions for new log file
maxsize SIZE Rotate when file exceeds SIZE
minsize SIZE Don't rotate if smaller than SIZE
maxage N Delete rotated files older than N days
postrotate/endscript Script to run after rotation

Testing logrotate

# Dry run (show what would happen)
$ sudo logrotate -d /etc/logrotate.d/nginx

# Force rotation (useful for testing)
$ sudo logrotate -f /etc/logrotate.d/nginx

# Verbose
$ sudo logrotate -v /etc/logrotate.conf

# View logrotate state (when each file was last rotated)
$ cat /var/lib/logrotate/status
"/var/log/nginx/access.log" 2026-3-19-10:0:0
"/var/log/nginx/error.log" 2026-3-19-10:0:0

copytruncate vs create

create (default): rename the old log, create a new empty file. Requires the application to reopen the file (via signal like SIGHUP or USR1).

copytruncate: copy the log file, then truncate the original. The application never notices. Risk: messages written between copy and truncate can be lost. Use only when the application cannot be signaled to reopen.

Audit Logging (auditd)

The Linux Audit System provides security-focused logging at the kernel level. It can log: - File access (reads, writes, attribute changes) - System calls - User commands - Authentication events - Network connections

Basic auditd Setup

# Check if auditd is running
$ sudo systemctl status auditd

# View audit rules
$ sudo auditctl -l

# View audit log
$ sudo ausearch -ts recent    # Recent events
$ sudo ausearch -m LOGIN      # Login events
$ sudo aureport --summary     # Summary report

Common Audit Rules

Rules in /etc/audit/rules.d/audit.rules:

# Monitor /etc/passwd for changes
-w /etc/passwd -p wa -k identity

# Monitor /etc/shadow for any access
-w /etc/shadow -p rwa -k shadow_access

# Monitor sudo configuration
-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d/ -p wa -k sudoers

# Monitor SSH configuration
-w /etc/ssh/sshd_config -p wa -k sshd_config

# Log all commands run by root
-a always,exit -F arch=b64 -F euid=0 -S execve -k root_commands

# Monitor Docker socket access
-w /var/run/docker.sock -p rwxa -k docker_socket

Flags: -w = watch path, -p = permissions filter (r=read, w=write, x=execute, a=attribute change), -k = key tag for searching.

# Search by key
$ sudo ausearch -k shadow_access

# Load new rules
$ sudo auditctl -R /etc/audit/rules.d/audit.rules

# Or restart auditd
$ sudo systemctl restart auditd

Log Forwarding

In production, logs should flow to a central aggregation system. Common patterns:

rsyslog Forwarding

# /etc/rsyslog.d/50-remote.conf

# Forward all logs via TCP
*.* @@logserver.example.com:514

# Forward only auth logs
auth,authpriv.* @@logserver.example.com:514

# Forward with TLS
$DefaultNetstreamDriverCAFile /etc/ssl/certs/ca.pem
$ActionSendStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode x509/name
*.* @@logserver.example.com:6514

Forwarding to Modern Stacks

For ELK, Loki, or Datadog: - Filebeat watches log files and ships to Elasticsearch - Promtail watches log files and ships to Loki - Fluentd/Fluent Bit watches files, journal, and syslog - Vector watches files and journal, ships to many destinations

# Example: Promtail config for Loki
# /etc/promtail/config.yml
scrape_configs:
  - job_name: journal
    journal:
      json: false
      max_age: 12h
      labels:
        job: systemd-journal
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit'

  - job_name: varlog
    static_configs:
      - targets: [localhost]
        labels:
          job: varlogs
          __path__: /var/log/*.log

Structured Logging

Modern applications should emit structured logs (JSON) rather than unstructured text:

{"timestamp":"2026-03-19T10:45:22.123Z","level":"error","service":"api","request_id":"abc123","user_id":"u-456","message":"database connection timeout","latency_ms":30000,"db_host":"db-primary.internal"}

Benefits: - Machine-parseable without fragile regex - Queryable in log aggregation systems (field-level filtering) - Consistent schema across services - Correlation via request IDs, trace IDs

# Query structured JSON logs with jq
$ cat /var/log/myapp/app.json | jq 'select(.level == "error" and .latency_ms > 5000)'

# From journalctl JSON output
$ journalctl -u myapp -o json | jq 'select(.PRIORITY == "3")'

Container Logging (stdout/stderr)

Kubernetes captures anything written to stdout/stderr from containers:

# View current logs
kubectl logs my-pod -n production

# Follow logs (tail -f equivalent)
kubectl logs -f my-pod -n production

# Previous container instance (after a crash/restart)
kubectl logs my-pod -n production --previous

# All containers in a pod
kubectl logs my-pod -n production --all-containers

# Last 100 lines from all pods with a label
kubectl logs -l app=api-server -n production --tail=100

Container runtime stores logs at /var/log/containers/ on the node:

# On the node itself
ls /var/log/containers/
# my-pod_production_app-abc123.log -> /var/log/pods/...

# These are JSON-wrapped log lines
cat /var/log/containers/my-pod_production_app-abc123.log | head -1
# {"log":"2024-01-15T03:22:41Z INFO starting\n","stream":"stdout","time":"2024-01-15T03:22:41.123Z"}

For container logs, configure the container runtime:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "3"
  }
}

Centralized Logging — ELK and EFK Stacks

ELK: Elasticsearch + Logstash + Kibana EFK: Elasticsearch + Fluentd/Fluent Bit + Kibana

+---------+     +--------------+     +---------------+     +---------+
| App Logs |---->| Fluent Bit   |---->| Elasticsearch |---->| Kibana  |
| (stdout) |     | (collector)  |     | (storage)     |     | (UI)    |
+---------+     +--------------+     +---------------+     +---------+

Fluent Bit DaemonSet (runs on every node, ships container logs):

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:2.2
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: containers
          mountPath: /var/log/containers
          readOnly: true
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: containers
        hostPath:
          path: /var/log/containers

Log Aggregation Patterns

Sidecar pattern — a logging container alongside your app container:

containers:
- name: app
  image: myapp:latest
  volumeMounts:
  - name: shared-logs
    mountPath: /var/log/app
- name: log-shipper
  image: fluent/fluent-bit:2.2
  volumeMounts:
  - name: shared-logs
    mountPath: /var/log/app
    readOnly: true

DaemonSet pattern — one collector per node (most common for Kubernetes): - Fluent Bit or Fluentd as DaemonSet - Reads from /var/log/containers/ - Lower resource overhead than sidecars

Aggregator pattern — two-tier shipping: - DaemonSet collectors forward to a central aggregator (Fluentd, Logstash, Vector) - Aggregator handles filtering, enrichment, and routing - Reduces connections to Elasticsearch

Log Retention and Compliance

Retention Guidelines:
  - Application logs: 30-90 days hot, 1 year cold storage
  - Security/audit logs: 1-7 years (regulatory dependent)
  - Debug logs: 7-14 days maximum
  - Access logs: 90 days hot, 1 year archive

Cost control:
  - Ship only WARN+ to expensive storage (Elasticsearch)
  - Archive DEBUG/INFO to cheap storage (S3, GCS)
  - Use index lifecycle management (ILM) in Elasticsearch
  - Sample high-volume logs (keep 10% of DEBUG in production)

Summary

Linux logging is a layered system. The kernel ring buffer captures hardware and driver events. journald captures everything from systemd-managed services with rich metadata. rsyslog writes traditional log files and forwards to remote systems. logrotate keeps disk usage under control. auditd provides security-focused audit trails. Understanding all these layers — and how they interact — is essential for debugging production issues, investigating security incidents, and maintaining system health. The most important practical skills: journalctl -u <service> for quick service debugging, dmesg -T for kernel issues, and proper logrotate configuration to prevent disk-full emergencies.

Wiki Navigation

Prerequisites

Related Content