Linux Users and Permissions — Street Ops

Real-world operational scenarios for user and permission issues. These are the situations you'll actually face in production, not textbook exercises.

---

Emergency Root Access Recovery

Scenario: You're locked out of root on a physical/VM server

No one knows the root password, and the only sudo user's account is locked or deleted.

Recovery via GRUB (single-user mode):

1. Reboot the machine
2. At the GRUB menu, press e to edit the boot entry
3. Find the line starting with linux or linuxefi
4. Append init=/bin/bash to the end of the line
5. Press Ctrl+X or F10 to boot

# You now have a root shell with root filesystem mounted read-only
# Remount root read-write
$ mount -o remount,rw /

# Reset root password
$ passwd root

# Or unlock a locked account
$ passwd -u deploy
$ usermod -U deploy

# If you need to fix sudoers
$ visudo

# Sync and reboot
$ sync
$ exec /sbin/init
# Or: reboot -f

Recovery via rescue/live USB:

# Boot from USB, mount the system root
$ mount /dev/sda2 /mnt
$ mount --bind /dev /mnt/dev
$ mount --bind /proc /mnt/proc
$ mount --bind /sys /mnt/sys

# Chroot into the system
$ chroot /mnt /bin/bash

# Now you're effectively root on the installed system
$ passwd root
$ exit
$ umount -R /mnt
$ reboot

For cloud instances (AWS/GCP/Azure):

You typically can't access GRUB. Instead: 1. Stop the instance 2. Detach the root volume 3. Attach it to a working instance as a secondary volume 4. Mount it, chroot, fix the issue 5. Reattach to the original instance

---

Locked Account Troubleshooting

Scenario: A user reports "Permission denied" when trying to SSH

Systematic diagnosis:

# 1. Check if account exists
$ id baduser
id: 'baduser': no such user

# 2. Check if account is locked (! or * prefix in shadow)
$ sudo grep baduser /etc/shadow
baduser:!$6$salt$hash...:19500:0:99999:7:::
# The ! prefix means the account is LOCKED

# 3. Check if account is expired
$ sudo chage -l baduser
Account expires         : Jan 01, 2026  # <-- expired!

# 4. Check if password is expired
$ sudo chage -l baduser
Password expires        : Dec 01, 2025  # <-- expired!
Password inactive       : Jan 01, 2026  # <-- inactive period expired too

# 5. Check if shell is nologin
$ grep baduser /etc/passwd
baduser:x:1001:1001::/home/baduser:/sbin/nologin

# 6. Check PAM restrictions
$ sudo grep -r baduser /etc/security/access.conf
- : baduser : ALL  # <-- explicitly denied

# 7. Check SSH-specific restrictions
$ sudo grep -E "^(AllowUsers|DenyUsers|AllowGroups|DenyGroups)" /etc/ssh/sshd_config
AllowGroups ssh-users   # <-- user might not be in this group

# 8. Check faillock (too many failed attempts)
$ sudo faillock --user baduser
baduser:
When                Type  Source                      Valid
2026-03-19 10:30:00 RHOST 10.0.0.99                  V
2026-03-19 10:30:05 RHOST 10.0.0.99                  V
2026-03-19 10:30:10 RHOST 10.0.0.99                  V
# Three failures — might be locked by pam_faillock

Fixes:

# Unlock account
$ sudo usermod -U baduser

# Reset failed login counter
$ sudo faillock --user baduser --reset

# Extend account expiry
$ sudo chage -E -1 baduser    # Remove expiry
$ sudo chage -E 2027-12-31 baduser  # Set new expiry

# Force password reset on next login
$ sudo chage -d 0 baduser

# Change shell from nologin
$ sudo usermod -s /bin/bash baduser

# Add to SSH allowed group
$ sudo usermod -aG ssh-users baduser

---

Permission Auditing

Scenario: Security team wants a report of SUID binaries and world-writable files

# Find all SUID binaries
$ sudo find / -perm -4000 -type f 2>/dev/null | sort
/usr/bin/chage
/usr/bin/gpasswd
/usr/bin/mount
/usr/bin/newgrp
/usr/bin/passwd
/usr/bin/su
/usr/bin/sudo
/usr/bin/umount
# Review this list — anything unexpected is a red flag

# Find all SGID binaries
$ sudo find / -perm -2000 -type f 2>/dev/null | sort

# Find world-writable files (excluding /proc, /sys, /tmp)
$ sudo find / -path /proc -prune -o -path /sys -prune -o -path /tmp -prune -o \
    -perm -0002 -type f -print 2>/dev/null

# Find world-writable directories (without sticky bit — dangerous)
$ sudo find / -path /proc -prune -o -path /sys -prune -o \
    -perm -0002 ! -perm -1000 -type d -print 2>/dev/null

# Find files with no owner
$ sudo find / -path /proc -prune -o -nouser -print 2>/dev/null

# Find files with no group
$ sudo find / -path /proc -prune -o -nogroup -print 2>/dev/null

# Audit home directory permissions
$ for dir in /home/*/; do
    user=$(basename "$dir")
    perms=$(stat -c '%a' "$dir")
    echo "$user: $dir ($perms)"
    if [ "$perms" != "700" ] && [ "$perms" != "750" ]; then
        echo "  WARNING: Home directory permissions too open"
    fi
done

# Check for .ssh directories with wrong permissions
$ for dir in /home/*/.ssh; do
    [ -d "$dir" ] || continue
    perms=$(stat -c '%a' "$dir")
    echo "$dir: $perms"
    if [ "$perms" != "700" ]; then
        echo "  WARNING: .ssh directory should be 700"
    fi
    for f in "$dir"/authorized_keys "$dir"/id_rsa "$dir"/id_ed25519; do
        [ -f "$f" ] || continue
        fperms=$(stat -c '%a' "$f")
        echo "  $f: $fperms"
    done
done

---

ACL Debugging

Scenario: A user has group access but still gets "Permission denied"

# Check standard permissions
$ ls -la /opt/project/data.csv
-rw-rw---- 1 app developers 1024 Mar 19 10:00 /opt/project/data.csv

# User is in the group...
$ id analyst
uid=1002(analyst) gid=1002(analyst) groups=1002(analyst),2000(developers)

# But wait — are there ACLs overriding?
$ getfacl /opt/project/data.csv
# file: opt/project/data.csv
# owner: app
# group: developers
user::rw-
user:analyst:---         # <-- EXPLICIT DENY for this user via ACL!
group::rw-
mask::rw-
other::---

# The named user ACL entry overrides the group permission.
# Fix: remove the restrictive ACL
$ sudo setfacl -x u:analyst /opt/project/data.csv

# Check the mask too — it can silently reduce effective permissions
$ getfacl /opt/project/data.csv
mask::r--     # <-- mask limits effective group/named-user perms to read-only!

# Fix mask
$ sudo setfacl -m m::rw- /opt/project/data.csv

Debugging the full permission chain

# Use namei to trace every directory in the path
$ namei -l /opt/project/subdir/data.csv
f: /opt/project/subdir/data.csv
drwxr-xr-x root    root    /
drwxr-xr-x root    root    opt
drwxr-x--- root    devteam project      # <-- analyst not in devteam!
drwxrwxr-x app     developers subdir
-rw-rw---- app     developers data.csv

# The user can't traverse /opt/project/ because they're not in devteam group
# Fix: either add analyst to devteam, or add an ACL on the directory
$ sudo setfacl -m u:analyst:x /opt/project/

---

sudo Log Analysis

Scenario: Investigate who ran what as root last night

# Search auth log for sudo usage
$ sudo grep "sudo:" /var/log/auth.log | grep "COMMAND" | tail -20
Mar 19 02:15:33 server01 sudo: deploy : TTY=pts/0 ; PWD=/home/deploy ; USER=root ; COMMAND=/usr/bin/systemctl restart nginx
Mar 19 02:16:01 server01 sudo: deploy : TTY=pts/0 ; PWD=/home/deploy ; USER=root ; COMMAND=/usr/bin/vim /etc/nginx/nginx.conf
Mar 19 03:45:22 server01 sudo: unknown : TTY=pts/1 ; PWD=/tmp ; USER=root ; COMMAND=/bin/bash

# Failed sudo attempts (potential security concern)
$ sudo grep "NOT in sudoers" /var/log/auth.log
Mar 19 03:44:55 server01 sudo: intruder : user NOT in sudoers ; TTY=pts/1 ; PWD=/tmp ; USER=root ; COMMAND=/bin/bash

# Using journalctl (if journal is persistent)
$ sudo journalctl _COMM=sudo --since "yesterday" --until "today" | grep COMMAND

# Who sudoed to root and when
$ sudo grep "session opened" /var/log/auth.log | grep sudo
Mar 19 02:15:33 server01 sudo: pam_unix(sudo:session): session opened for user root(uid=0) by deploy(uid=1000)

# Audit trail (if auditd is running)
$ sudo ausearch -m USER_CMD --start yesterday -i

---

Mass Permission Fix After Bad chmod -R

Scenario: Someone ran chmod -R 777 / or chmod -R 644 /var

This is an emergency. The system may become unstable quickly.

If chmod -R 777 / was run (worst case):

# SUID/SGID bits are destroyed. Critical binaries stop working.
# su, sudo, passwd, ping — all broken.

# If you still have a root shell:

# Fix the most critical files first
$ chmod 4755 /usr/bin/sudo
$ chmod 4755 /usr/bin/su
$ chmod 4755 /usr/bin/passwd
$ chmod 4755 /usr/bin/newgrp
$ chmod 0600 /etc/shadow
$ chmod 0600 /etc/gshadow
$ chmod 0644 /etc/passwd
$ chmod 0644 /etc/group
$ chmod 0440 /etc/sudoers
$ chmod 1777 /tmp
$ chmod 700 /root

# Fix SSH (or SSH will refuse to work)
$ chmod 700 /root/.ssh 2>/dev/null
$ chmod 600 /root/.ssh/authorized_keys 2>/dev/null
$ chmod 600 /etc/ssh/ssh_host_*_key
$ chmod 644 /etc/ssh/ssh_host_*_key.pub
$ chmod 644 /etc/ssh/sshd_config

# Restore from package manager (best approach for full recovery)
# Debian/Ubuntu:
$ dpkg --verify 2>/dev/null | grep "^..5" | awk '{print $2}' | while read pkg; do
    apt-get install --reinstall "$pkg"
done

# RHEL/CentOS:
$ rpm -Va 2>/dev/null | grep "^.M" | awk '{print $NF}' | while read file; do
    rpm -qf "$file" | head -1
done | sort -u | while read pkg; do
    yum reinstall -y "$pkg"
done

If chmod -R 644 /var was run:

# Directories lost execute bit — nothing inside /var is accessible
# Fix: restore execute on directories only
$ sudo find /var -type d -exec chmod +x {} \;

# Then fix specific known permissions
$ sudo chmod 1777 /var/tmp
$ sudo chmod 755 /var/log /var/run /var/lib
$ sudo chmod 750 /var/log/journal 2>/dev/null

# Let package manager verify and fix
$ sudo dpkg --verify 2>&1 | grep "/var" | awk '{print $2}' | \
    xargs -I{} rpm -qf {} 2>/dev/null | sort -u

---

NFS UID Mapping Issues

Scenario: Files created on NFS share show wrong ownership

# On the NFS client, files show as "nobody:nogroup"
$ ls -la /mnt/nfs/
-rw-r--r-- 1 nobody nogroup 1024 Mar 19 10:00 data.csv

# Or files show numeric UIDs instead of usernames
$ ls -la /mnt/nfs/
-rw-r--r-- 1 1000 1000 1024 Mar 19 10:00 data.csv

Diagnosis:

# Check if the UID exists on the client
$ id 1000
uid=1000(deploy) gid=1000(deploy)  # Good — mapping exists
# If id fails, the UID doesn't have a local mapping

# Check NFS mount options
$ mount | grep nfs
192.168.1.10:/export on /mnt/nfs type nfs4 (rw,sec=sys,...)

# Check if idmapd is running (NFSv4)
$ systemctl status nfs-idmapd

# Check idmapd config
$ cat /etc/idmapd.conf
[General]
Domain = example.com    # Must match on client AND server!

# Check if root_squash is causing issues
# On NFS server:
$ cat /etc/exports
/export  10.0.0.0/24(rw,sync,root_squash)
# root_squash maps UID 0 to nobody — this is normal and desired

# all_squash maps ALL users to nobody — might be the problem
/export  10.0.0.0/24(rw,sync,all_squash)

Fixes:

# Ensure UIDs/GIDs match between client and server
# Or use LDAP/SSSD for centralized identity

# For NFSv4, ensure idmapd domain matches
# Client AND server /etc/idmapd.conf:
[General]
Domain = example.com

# Restart idmapd after changes
$ sudo systemctl restart nfs-idmapd

# If using all_squash with anonuid/anongid for a shared directory:
# Server /etc/exports:
/export/shared  10.0.0.0/24(rw,sync,all_squash,anonuid=2000,anongid=2000)

# Clear the NFS client's idmap cache
$ sudo nfsidmap -c

---

Investigating "Permission Denied" for a Running Service

Scenario: A systemd service fails to start with permission errors

# Check the service logs
$ journalctl -u myapp.service --since "5 minutes ago"
Mar 19 10:45:22 server01 myapp[1234]: Error: EACCES: permission denied, open '/var/lib/myapp/data.db'

# Check what user the service runs as
$ systemctl show myapp.service -p User -p Group
User=myapp
Group=myapp

# Verify the user can access the file
$ sudo -u myapp test -r /var/lib/myapp/data.db && echo "OK" || echo "DENIED"
DENIED

# Check the full path chain
$ namei -l /var/lib/myapp/data.db
f: /var/lib/myapp/data.db
drwxr-xr-x root  root  /
drwxr-xr-x root  root  var
drwxr-xr-x root  root  lib
drwx------ root  root  myapp      # <-- myapp user can't enter this directory!
-rw-r--r-- myapp myapp data.db

# Fix
$ sudo chown myapp:myapp /var/lib/myapp
$ sudo chmod 750 /var/lib/myapp

# Also check SELinux context if applicable
$ ls -laZ /var/lib/myapp/
$ sudo restorecon -Rv /var/lib/myapp/

# Check systemd hardening that might restrict access
$ systemctl show myapp.service | grep -E "(ProtectSystem|ProtectHome|ReadWrite|ReadOnly|Inaccessible)"
ProtectSystem=strict    # <-- only /proc, /dev, /sys are writable
ReadWritePaths=/var/lib/myapp   # <-- must be listed here!

---

Bulk User Management

Scenario: Onboard 50 new developers with consistent permissions

#!/bin/bash
# bulk-create-users.sh

GROUP="developers"
SHELL="/bin/bash"
EXPIRY="2027-12-31"

# Ensure group exists
getent group "$GROUP" > /dev/null || groupadd "$GROUP"

while IFS=',' read -r username fullname; do
    # Skip header or empty lines
    [[ "$username" =~ ^#|^$ ]] && continue

    if id "$username" &>/dev/null; then
        echo "SKIP: $username already exists"
        continue
    fi

    useradd -m -s "$SHELL" -G "$GROUP,docker" -c "$fullname" -e "$EXPIRY" "$username"

    # Set temporary password (force change on first login)
    echo "${username}:TempPass123!" | chpasswd
    chage -d 0 "$username"

    # Set proper home directory permissions
    chmod 750 "/home/$username"

    # Create standard directories
    mkdir -p "/home/$username/.ssh"
    chmod 700 "/home/$username/.ssh"
    chown -R "${username}:${username}" "/home/$username"

    echo "CREATED: $username ($fullname)"
done < users.csv

echo "Done. All users must change password on first login."

# users.csv
jsmith,Jane Smith
bwilson,Bob Wilson
agarcia,Ana Garcia

---

Emergency: sudoers Syntax Error Recovery

Scenario: Someone edited /etc/sudoers directly (not visudo) and introduced a syntax error. Now nobody can sudo.

# Attempting sudo gives:
$ sudo -l
>>> /etc/sudoers: syntax error near line 25 <<<
sudo: parse error in /etc/sudoers near line 25
sudo: no valid sudoers sources found, quitting

# Options:
# 1. If you have an open root shell: just fix it
$ visudo  # Will show you the error and let you fix it

# 2. If you have pkexec (PolicyKit) available:
$ pkexec visudo

# 3. Boot into single-user mode (see Emergency Root Access Recovery above)

# 4. If the error is in a drop-in file:
$ pkexec rm /etc/sudoers.d/broken-file
# Or rename it:
$ pkexec mv /etc/sudoers.d/broken-file /etc/sudoers.d/broken-file.bak

Prevention: Always use visudo. If you must automate sudoers changes:

# Validate syntax before installing
$ visudo -cf /tmp/new-sudoers-fragment
/tmp/new-sudoers-fragment: parsed OK

# Only then install
$ cp /tmp/new-sudoers-fragment /etc/sudoers.d/99-myconfig
$ chmod 440 /etc/sudoers.d/99-myconfig

Pages that link here

• Anti-Primer: Linux Users And Permissions
• Linux Users and Permissions