Skip to content

Audit Logging - Street-Level Ops

Real-world workflows for configuring, querying, and maintaining audit logs on Linux and Kubernetes.

Quick auditd Status Check

# Is auditd running?
systemctl status auditd

# How many rules are loaded?
auditctl -l | wc -l

# Check audit log size and rotation
ls -lh /var/log/audit/audit.log*

# Check if audit backlog is filling up (dropped events = bad)
auditctl -s

# Output:
# enabled 1
# failure 1
# pid 1234
# backlog_limit 8192
# lost 0            <-- non-zero means events were dropped
# backlog 3

Default trap: The default backlog_limit of 64 is far too low for busy servers — any burst of syscalls will overflow the buffer and drop audit events. Set it to at least 8192: auditctl -b 8192. A non-zero lost counter means you have gaps in your audit trail — compliance auditors will flag this.

Setting Up File Watches

# Watch critical config files for changes
auditctl -w /etc/passwd -p wa -k identity
auditctl -w /etc/shadow -p wa -k identity
auditctl -w /etc/sudoers -p wa -k sudo_changes
auditctl -w /etc/ssh/sshd_config -p wa -k sshd_config

# Watch for unauthorized cron modifications
auditctl -w /etc/crontab -p wa -k cron_changes
auditctl -w /var/spool/cron/ -p wa -k cron_changes

# Watch audit logs themselves (detect tampering)
auditctl -w /var/log/audit/ -p wa -k audit_log_access

# Verify rules are active
auditctl -l

# Output:
# -w /etc/passwd -p wa -k identity
# -w /etc/shadow -p wa -k identity
# -w /etc/sudoers -p wa -k sudo_changes

Tracking Command Execution

# Log all commands run via execve (comprehensive but noisy)
auditctl -a always,exit -F arch=b64 -S execve -k commands

# Log only privileged command usage
auditctl -a always,exit -F path=/usr/bin/sudo -F perm=x -k privileged_sudo
auditctl -a always,exit -F path=/usr/bin/passwd -F perm=x -k privileged_passwd
auditctl -a always,exit -F path=/usr/bin/chage -F perm=x -k privileged_chage

# Log privilege escalation syscalls
auditctl -a always,exit -F arch=b64 -S setuid -S setgid -k privilege_escalation

Searching Audit Logs

# Search by key (fastest way to find relevant events)
ausearch -k identity -i

# Output:
# type=SYSCALL ... comm="passwd" exe="/usr/bin/passwd" key="identity"
# type=PATH ... name="/etc/passwd" ...

# Search by time range
ausearch --start "03/14/2024" "09:00:00" --end "03/14/2024" "17:00:00" -k sudo_changes

# Search for failed login attempts
ausearch -m USER_LOGIN --success no -i

# Search by user
ausearch -ua 1001 -i

# Search for specific command execution
ausearch -k commands -i | grep "rm -rf"

# Pipe to aureport for summary
ausearch -k privileged_sudo -i | aureport --summary

Remember: ausearch key flags mnemonic: K-T-U-M — -k key (search by rule tag), -ts/--start time (start time), -ua user (search by UID), -m message type (USER_LOGIN, SYSCALL, etc.). The -i flag interprets numeric IDs into human-readable names — always use it for manual investigation.

Generating Reports

# Summary of all audit events
aureport --summary

# Output:
# Summary Report
# ======================
# Range of time in logs: 03/01/2024 00:00:01 - 03/15/2024 23:59:59
# Number of changes in configuration: 12
# Number of accounts, groups, roles changed: 3
# Number of logins: 245
# Number of failed logins: 18

# Authentication report
aureport --auth

# Failed events (attempted policy violations)
aureport --failed

# Login report
aureport --login --summary

# File access events
aureport --file --summary

# Command execution report
aureport --comm --summary | head -20

Making Rules Persistent

# Generate rules file from running config
auditctl -l > /etc/audit/rules.d/custom.rules

# Or edit rules file directly
cat > /etc/audit/rules.d/90-custom.rules << 'EOF'
# Identity and access
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/sudoers -p wa -k sudo_changes
-w /etc/sudoers.d/ -p wa -k sudo_changes

# SSH config
-w /etc/ssh/sshd_config -p wa -k sshd_config

# Privileged commands
-a always,exit -F path=/usr/bin/sudo -F perm=x -k privileged_sudo

# Make config immutable (requires reboot to change)
-e 2
EOF

> **Under the hood:** The `-e 2` flag makes audit rules immutable until reboot — even root cannot modify or disable them. This prevents an attacker who gains root from silently turning off auditing. Place `-e 2` as the very last line in your rules file because once it is loaded, no further rule changes are accepted. This is a hard requirement for PCI-DSS and many SOC 2 audits.

# Load new rules
augenrules --load

# Verify
auditctl -l

Kubernetes Audit Log Queries

# Check API server audit log for secret access
jq 'select(.objectRef.resource == "secrets")' /var/log/kubernetes/audit.log

# Who deleted a pod?
jq 'select(.verb == "delete" and .objectRef.resource == "pods")' \
  /var/log/kubernetes/audit.log | jq '{user: .user.username, pod: .objectRef.name, time: .requestReceivedTimestamp}'

# All RBAC changes
jq 'select(.objectRef.resource | test("roles|rolebindings|clusterroles|clusterrolebindings"))' \
  /var/log/kubernetes/audit.log

# Failed API requests (403 Forbidden)
jq 'select(.responseStatus.code == 403)' /var/log/kubernetes/audit.log | \
  jq '{user: .user.username, verb: .verb, resource: .objectRef.resource, reason: .responseStatus.reason}'

Centralized Log Shipping

# Configure audisp-remote for forwarding
cat /etc/audisp/plugins.d/au-remote.conf
# active = yes
# direction = out
# path = /sbin/audisp-remote

# Verify logs are arriving at central syslog
ssh syslog-server 'tail -5 /var/log/remote/audit.log'

# Check for forwarding backlogs
wc -l /var/spool/audit/*.log 2>/dev/null

Emergency: Investigating a Breach

# 1. Check for unauthorized user additions
ausearch -k identity -i --start recent

# 2. Check for privilege escalation
ausearch -k privilege_escalation -i --start today

# 3. Check for SSH key changes
ausearch -k sshd_config -i --start today

# 4. Check for cron modifications (persistence mechanism)
ausearch -k cron_changes -i --start today

# 5. Full timeline of a specific user's actions
ausearch -ua <UID> -i --start "03/14/2024" | aureport --file

Gotcha: Audit logs on the compromised host itself may have been tampered with. During a breach investigation, always cross-reference local audit logs with the centralized copy (syslog server, SIEM). If you do not have centralized audit log shipping (audisp-remote), you cannot trust the local logs on a compromised system.