SELinux & Linux Hardening - Street-Level Ops¶

What experienced ops engineers know about running hardened Linux systems without losing their minds.

Quick Diagnosis Commands¶

# SELinux status and mode
getenforce
sestatus

# Recent SELinux denials
ausearch -m avc -ts recent
ausearch -m avc -ts today --interpret

# Human-readable denial explanations
sealert -a /var/log/audit/audit.log | head -80

# Check SELinux context on files
ls -Z /var/www/html/
ls -Z /etc/nginx/

# Check SELinux context on processes
ps -eZ | grep -E 'httpd|nginx|sshd'

# Check SELinux booleans for a service
getsebool -a | grep httpd
getsebool -a | grep nginx

# Check listening ports and their SELinux labels
semanage port -l | grep http
semanage port -l | grep ssh

# Audit log summary
aureport --avc --summary

# Check for failed logins
lastb | head -20
faillock --user <username>

# Check sysctl values
sysctl -a | grep ip_forward
sysctl -a | grep syncookies

# Check for world-writable files
find /etc -type f -perm -0002 2>/dev/null

# Check SUID binaries
find /usr -type f -perm -4000 -exec ls -la {} \; 2>/dev/null

Gotcha: Application Breaks After Enabling SELinux¶

You enabled SELinux enforcing mode on a server that has been running permissive for months. Half the services stop working. The application cannot read its own files, bind to its port, or connect to the database.

Fix:

# Step 1: Switch to permissive first (log but don't block)
setenforce 0

# Step 2: Relabel the entire filesystem
fixfiles -F onboot
# Or for RHEL 9+:
touch /.autorelabel
reboot

# Step 3: After relabel, check for denials in permissive mode
ausearch -m avc -ts boot

# Step 4: Fix the most common issues
restorecon -Rv /var/www/
restorecon -Rv /opt/myapp/
semanage port -a -t http_port_t -p tcp 8080
setsebool -P httpd_can_network_connect on

# Step 5: Only switch to enforcing when denials are clean
setenforce 1

Never go from disabled to enforcing without a relabel. The filesystem has no labels and everything will be denied.

One-liner: The relabel-on-boot (touch /.autorelabel) walks every file on every filesystem and sets the correct SELinux context based on the policy's file context rules. On a server with millions of files, this can take over an hour. Schedule it during a maintenance window, not during a production incident.

Gotcha: Custom App Files Have Wrong SELinux Context¶

You deployed your application to /opt/myapp/ and files have default_t context. SELinux blocks the web server from reading them. restorecon does not help because there is no policy for that path.

Fix:

# Define a custom file context rule
semanage fcontext -a -t httpd_sys_content_t "/opt/myapp/static(/.*)?"
semanage fcontext -a -t httpd_sys_rw_content_t "/opt/myapp/uploads(/.*)?"
semanage fcontext -a -t httpd_log_t "/opt/myapp/logs(/.*)?"

# Apply the context rules
restorecon -Rv /opt/myapp/

# Verify
ls -Z /opt/myapp/

Gotcha: audit2allow Generates Overly Broad Policies¶

You run ausearch -m avc | audit2allow -M fix and install it. It works. But the generated policy allows far more than the specific action that was denied. You just punched a hole in your security policy.

Fix:

# Always review the generated policy first
ausearch -m avc -ts recent | audit2allow -m mypolicy > mypolicy.te
cat mypolicy.te

# Look for overly broad rules like:
#   allow httpd_t file_type:file { read open getattr };
# This allows httpd to read ALL file types, not just web content

# Instead, write targeted rules manually
cat > myapp.te << 'EOF'
module myapp 1.0;
require {
    type httpd_t;
    type myapp_data_t;
    class file { read open getattr };
}
allow httpd_t myapp_data_t:file { read open getattr };
EOF

checkmodule -M -m -o myapp.mod myapp.te
semodule_package -o myapp.pp -m myapp.mod
semodule -i myapp.pp

Gotcha: SSH Hardening Locks You Out¶

You pushed SSH hardening via Ansible: PasswordAuthentication no, AllowUsers deploy. But the deploy user's SSH key is not on the target server. You just locked yourself out of every server in the fleet.

Fix:

# Prevention: Always test SSH config before restarting
sshd -t  # Syntax check
sshd -T  # Show effective config

# In Ansible, validate before applying:
# - name: Validate sshd config
#   command: sshd -t -f /etc/ssh/sshd_config.new
#   register: sshd_check
#   changed_when: false
#
# - name: Deploy sshd config
#   copy:
#     src: sshd_config
#     dest: /etc/ssh/sshd_config
#   when: sshd_check.rc == 0

# Recovery: If locked out
# 1. Console access (IPMI/iLO/iDRAC/cloud console)
# 2. Boot to single-user mode
# 3. Fix /etc/ssh/sshd_config
# 4. systemctl restart sshd

Always maintain out-of-band access (console, IPMI) on hardened servers. SSH is not your only door.

Gotcha: When using Ansible to harden SSH, always use a two-step deploy: first push the new config to a temporary file and validate with sshd -t, then atomically move it into place. If you push a broken config directly to /etc/ssh/sshd_config and restart sshd, existing sessions may survive but no new connections will work — and you may not notice until your current session ends.

Gotcha: faillock Locks Out Legitimate Users¶

You configured pam_faillock with deny=3 unlock_time=900. An attacker brute-forces your admin account. The account locks. Your legitimate admin cannot log in for 15 minutes during an incident.

Fix:

# Unlock a specific user
faillock --user admin --reset

# Check lock status
faillock --user admin

# Better config: longer lockout but allow console bypass
# In /etc/pam.d/system-auth, add before faillock:
# auth [success=1 default=ignore] pam_succeed_if.so service = login
# This skips faillock for local console logins

Pattern: Hardened AMI Build Pipeline¶

# Packer + Ansible pipeline for hardened AMIs
# 1. Start from official vendor AMI
# 2. Run CIS hardening playbook
# 3. Run SELinux policy setup
# 4. Run sysctl hardening
# 5. Scan with OpenSCAP
# 6. Publish AMI

# OpenSCAP scan for CIS compliance
oscap xccdf eval \
    --profile xccdf_org.ssgproject.content_profile_cis \
    --results scan-results.xml \
    --report scan-report.html \
    /usr/share/xml/scap/ssg/content/ssg-rhel9-ds.xml

# Parse results
oscap xccdf generate report scan-results.xml > report.html

Pattern: Ongoing Compliance Monitoring¶

#!/bin/bash
# compliance-check.sh - run via cron weekly
set -euo pipefail

REPORT_DIR=/var/log/compliance
mkdir -p "$REPORT_DIR"
DATE=$(date +%Y-%m-%d)

# Check SELinux is enforcing
MODE=$(getenforce)
[ "$MODE" != "Enforcing" ] && echo "FAIL: SELinux is $MODE" >> "$REPORT_DIR/$DATE.log"

# Check for world-writable files in /etc
WORLD_WRITABLE=$(find /etc -type f -perm -0002 2>/dev/null | wc -l)
[ "$WORLD_WRITABLE" -gt 0 ] && echo "FAIL: $WORLD_WRITABLE world-writable files in /etc" >> "$REPORT_DIR/$DATE.log"

# Check SSH config
grep -q "^PermitRootLogin no" /etc/ssh/sshd_config || \
    echo "FAIL: root login not disabled" >> "$REPORT_DIR/$DATE.log"
grep -q "^PasswordAuthentication no" /etc/ssh/sshd_config || \
    echo "FAIL: password auth not disabled" >> "$REPORT_DIR/$DATE.log"

# Check sysctl values
[ "$(sysctl -n net.ipv4.ip_forward)" != "0" ] && \
    echo "FAIL: IP forwarding enabled" >> "$REPORT_DIR/$DATE.log"

# Check for new SUID binaries since last scan
find /usr /bin /sbin -type f -perm -4000 2>/dev/null | sort > "$REPORT_DIR/suid-$DATE.list"
if [ -f "$REPORT_DIR/suid-baseline.list" ]; then
    diff "$REPORT_DIR/suid-baseline.list" "$REPORT_DIR/suid-$DATE.list" || \
        echo "WARN: SUID binaries changed" >> "$REPORT_DIR/$DATE.log"
fi

echo "Compliance scan complete: $REPORT_DIR/$DATE.log"

# 1. At GRUB menu, add to kernel command line:
#    enforcing=0
# This boots in permissive mode without changing config

# 2. Once booted, check what's wrong
ausearch -m avc -ts boot

# 3. Fix the issue (usually a relabel is needed)
fixfiles -F onboot
reboot

# 4. If relabel fixes it, switch back to enforcing
setenforce 1
# Verify in /etc/selinux/config: SELINUX=enforcing

# Nuclear option (only if system is completely broken):
# Add selinux=0 to kernel command line
# This disables SELinux entirely — use only for recovery
# You MUST relabel before re-enabling

Emergency: Auditd Filling Disk¶

# 1. Check audit log size
du -sh /var/log/audit/

# 2. Rotate immediately
service auditd rotate

# 3. Check what's generating the most events
aureport --summary

# 4. Reduce noisy rules
auditctl -l | grep -c ""  # Count current rules

# 5. Temporarily reduce logging
auditctl -e 0  # Disable auditing (TEMPORARY)

# 6. Fix rules, then re-enable
auditctl -e 1

# 7. Set proper log rotation in /etc/audit/auditd.conf
# max_log_file = 50
# num_logs = 5
# max_log_file_action = ROTATE

SELinux & Linux Hardening - Street-Level Ops¶

Quick Diagnosis Commands¶

Gotcha: Application Breaks After Enabling SELinux¶

Gotcha: Custom App Files Have Wrong SELinux Context¶

Gotcha: audit2allow Generates Overly Broad Policies¶

Gotcha: SSH Hardening Locks You Out¶

Gotcha: faillock Locks Out Legitimate Users¶

Pattern: Hardened AMI Build Pipeline¶

Pattern: Ongoing Compliance Monitoring¶

Emergency: SELinux Preventing Boot or Login¶

Emergency: Auditd Filling Disk¶

Pages that link here¶