Thinking Out Loud: Linux Hardening¶

A senior SRE's internal monologue while working through a real Linux hardening task. This isn't a tutorial — it's a window into how experienced engineers actually think.

The Situation¶

A new batch of bare-metal servers has been racked for a PCI-compliant payment processing workload. The servers have a base Ubuntu 22.04 install. I need to harden them to pass the PCI DSS audit and our internal security baseline before the payment team deploys their application.

The Monologue¶

PCI compliance hardening. This isn't just "make it more secure" — it's a specific checklist with auditable requirements. Let me work through the CIS benchmark systematically, starting with the highest-risk items.

First, let me baseline what we're starting with.

cat /etc/os-release | grep VERSION
uname -r

Ubuntu 22.04 LTS, kernel 5.15. Good — LTS release with long-term support. Let me start with filesystem hardening, then move to network, then services, then audit logging.

Filesystem first. Separate partitions for key directories prevent one area from filling up and crashing the system. PCI requires it.

df -h
mount | grep -E "/(tmp|var|var/log|var/log/audit|home)"

Everything is on one big root partition. For PCI, we need separate mount points for /tmp, /var, /var/log, and /var/log/audit with appropriate mount options. But re-partitioning a live system is disruptive. Since these are fresh servers, let me set this up properly now.

Actually, let me check if LVM is in use — that makes adding partitions easier.

lvs
pvs

LVM is set up with some free space. Good. Let me create the required logical volumes.

sudo lvcreate -L 2G -n tmp vg0
sudo mkfs.ext4 /dev/vg0/tmp
echo '/dev/vg0/tmp /tmp ext4 nodev,nosuid,noexec 0 0' | sudo tee -a /etc/fstab
sudo mount -a

Mental Model: Mount Options as Security Layers¶

Mount options are cheap, effective security controls. noexec prevents execution of binaries (stops attackers from running downloaded malware in /tmp). nosuid prevents SUID elevation. nodev prevents device file creation. Apply these to every mountpoint that doesn't need those capabilities. The cost is zero performance impact; the benefit is significant attack surface reduction.

Now let me set up /tmp properly and then tackle /var/log/audit for auditd.

# Remount /tmp with security options
sudo mount -o remount,nodev,nosuid,noexec /tmp

# Create audit log partition
sudo lvcreate -L 5G -n audit vg0
sudo mkfs.ext4 /dev/vg0/audit
sudo mkdir -p /var/log/audit
echo '/dev/vg0/audit /var/log/audit ext4 defaults 0 0' | sudo tee -a /etc/fstab
sudo mount -a

Now kernel hardening. There are sysctl parameters that PCI requires.

cat <<'EOF' | sudo tee /etc/sysctl.d/90-pci-hardening.conf
# Network hardening
net.ipv4.ip_forward = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.tcp_syncookies = 1
net.ipv6.conf.all.accept_ra = 0
net.ipv6.conf.default.accept_ra = 0

# Kernel hardening
kernel.randomize_va_space = 2
kernel.dmesg_restrict = 1
kernel.kptr_restrict = 2
fs.suid_dumpable = 0
EOF

sudo sysctl -p /etc/sysctl.d/90-pci-hardening.conf

Let me verify a few of the critical ones took effect.

sysctl net.ipv4.ip_forward net.ipv4.tcp_syncookies kernel.randomize_va_space

Mental Model: Sysctl as Kernel Firewall¶

Sysctl parameters are the kernel's configuration interface. Many are security-relevant but often left at insecure defaults. The most important ones: disable IP forwarding (unless it's a router), enable SYN cookies (SYN flood protection), enable ASLR (randomize_va_space=2), disable source routing (prevents packet routing manipulation), and enable reverse path filtering (anti-spoofing). Apply these before anything else.

Next: auditd. PCI requires comprehensive audit logging of privileged actions, file access, and authentication events.

sudo apt-get install auditd audispd-plugins -y

Configure the audit rules.

cat <<'EOF' | sudo tee /etc/audit/rules.d/pci-audit.rules
# Record all commands run as root
-a always,exit -F arch=b64 -F euid=0 -S execve -k root-commands

# Monitor authentication files
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/gshadow -p wa -k identity
-w /etc/sudoers -p wa -k identity
-w /etc/sudoers.d/ -p wa -k identity

# Monitor SSH configuration
-w /etc/ssh/sshd_config -p wa -k sshd-config

# Monitor cron
-w /etc/crontab -p wa -k cron
-w /etc/cron.d/ -p wa -k cron

# Login/logout events
-w /var/log/lastlog -p wa -k logins
-w /var/run/faillock/ -p wa -k logins

# Time changes (PCI requirement)
-a always,exit -F arch=b64 -S adjtimex -S settimeofday -k time-change
-a always,exit -F arch=b64 -S clock_settime -k time-change
-w /etc/localtime -p wa -k time-change

# Make the audit configuration immutable (requires reboot to change)
-e 2
EOF

sudo augenrules --load
sudo systemctl enable auditd
sudo systemctl restart auditd

The -e 2 at the end makes the audit rules immutable — they can't be changed without a reboot. This prevents an attacker from disabling audit logging after gaining access. But it also means I can't change audit rules without rebooting. I always put this last and double-check my rules before loading.

Let me verify auditd is working.

sudo auditctl -l | head -10
sudo ausearch -k identity --start today | head -5

Now let me handle user security. PCI requires password complexity, account lockout, and session timeouts.

# Password complexity (even though we use key-based auth, PCI still requires it)
sudo apt-get install libpam-pwquality -y
sudo sed -i 's/^# minlen.*/minlen = 14/' /etc/security/pwquality.conf
sudo sed -i 's/^# dcredit.*/dcredit = -1/' /etc/security/pwquality.conf
sudo sed -i 's/^# ucredit.*/ucredit = -1/' /etc/security/pwquality.conf
sudo sed -i 's/^# lcredit.*/lcredit = -1/' /etc/security/pwquality.conf
sudo sed -i 's/^# ocredit.*/ocredit = -1/' /etc/security/pwquality.conf

# Account lockout after 5 failed attempts
sudo cat <<'EOF' > /etc/security/faillock.conf
deny = 5
unlock_time = 900
fail_interval = 900
EOF

# Session timeout
echo 'TMOUT=600' | sudo tee -a /etc/profile.d/timeout.sh
echo 'readonly TMOUT' | sudo tee -a /etc/profile.d/timeout.sh
echo 'export TMOUT' | sudo tee -a /etc/profile.d/timeout.sh

Let me also disable unused services and remove unnecessary packages.

# Find and disable unnecessary services
sudo systemctl list-unit-files --type=service --state=enabled | grep -v -E 'ssh|rsyslog|auditd|cron|networkd|resolved|ufw|systemd'

Anything here that the payment app doesn't need should be disabled. Let me check...

# Disable services not needed for this workload
sudo systemctl disable --now cups 2>/dev/null
sudo systemctl disable --now avahi-daemon 2>/dev/null
sudo systemctl disable --now bluetooth 2>/dev/null
sudo systemctl mask ctrl-alt-del.target

Finally, let me run a CIS benchmark scan to verify my work.

# If lynis is available
sudo apt-get install lynis -y
sudo lynis audit system --quick 2>&1 | tail -30

Lynis gives a hardening index. Let me check the score... 78. Pre-hardening was probably around 50. The biggest remaining gaps are usually around AIDE (file integrity monitoring) and log forwarding, which we'll set up when the SIEM is configured.

What Made This Senior-Level¶

Junior Would...	Senior Does...	Why
Start patching packages first	Start with filesystem mount options and kernel sysctl parameters	These are the cheapest, most effective security controls with zero performance impact
Apply audit rules without the immutable flag	Make audit rules immutable with `-e 2` to prevent tampering	An attacker who gains root can disable audit logging unless rules are immutable
Harden one server manually and hope it's right	Run a CIS benchmark tool (lynis) to verify hardening and find gaps	Automated benchmarks catch things humans miss and provide an auditable score
Fix things in whatever order seems easiest	Work systematically: filesystem -> kernel -> network -> services -> audit -> users	Systematic hardening ensures nothing is missed and creates a repeatable process

Key Heuristics Used¶

Mount Options Are Free Security: noexec, nosuid, nodev on every mountpoint that doesn't need those capabilities. Zero performance cost, significant attack surface reduction.
Sysctl Before Services: Kernel parameters (IP forwarding, SYN cookies, ASLR) are the foundation. Harden the kernel before hardening services on top of it.
Immutable Audit Rules: Use -e 2 to prevent audit rule tampering. Accept the reboot requirement for rule changes as a security tradeoff.

Cross-References¶

Primer — Linux security architecture, PAM, and the CIS benchmark framework
Street Ops — Hardening checklists, sysctl references, and auditd configuration
Footguns — Forgetting mount options, not making audit rules immutable, and disabling password auth before verifying key auth