Linux Recovery

← Back to all decks

10 cards — 🟡 5 medium | 🔴 5 hard

🟡 Medium (5)

1. What is the difference between /dev/random and /dev/urandom?

Show answer

/dev/random blocks when the entropy pool is exhausted; /dev/urandom never blocks.

Historically:
- /dev/random: 'true' randomness, blocks when entropy low — used for long-lived crypto keys
- /dev/urandom: pseudo-random, never blocks — used for everything else

Modern reality (Linux 5.6+): Both use the same CSPRNG. /dev/random only blocks until the pool is initially seeded (at boot). After that, urandom and random are equivalent.

Recommendation: Use /dev/urandom for virtually everything. The only exception is generating long-lived keys on a freshly booted system with no entropy (rare). Use getrandom() syscall in code — it blocks only until initial seeding, then never blocks.

2. Explain Access Control Lists (ACLs) with getfacl and setfacl

Show answer

ACLs extend traditional Unix permissions (owner/group/other) with fine-grained per-user and per-group rules.

# View ACLs:
getfacl /path/to/file

# Grant user read access:
setfacl -m u:alice:r /path/to/file

# Grant group write access:
setfacl -m g:devs:rw /path/to/file

# Set default ACL for new files in directory:
setfacl -d -m u:alice:rw /path/to/dir/

# Remove specific ACL:
setfacl -x u:alice /path/to/file

# Remove all ACLs:
setfacl -b /path/to/file

The + in ls -l output (drwxr-xr-x+) indicates ACLs present. Filesystem must be mounted with acl option. Effective permissions = ACL mask AND granted permissions.

3. Explain AppArmor profiles and enforcement modes

Show answer

AppArmor is a Mandatory Access Control (MAC) system using path-based profiles to restrict program capabilities.

Modes:
- enforce: violations blocked and logged
- complain: violations logged but allowed (for testing)
- unconfined: no restrictions

# Check status:
aa-status

# Put profile in complain mode:
aa-complain /etc/apparmor.d/usr.bin.firefox

# Put profile in enforce mode:
aa-enforce /etc/apparmor.d/usr.bin.firefox

# Generate profile interactively:
aa-genprof /path/to/binary

Profile syntax:
/usr/bin/app {
/etc/app.conf r,
/var/log/app.log w,
/tmp/ rw,
network inet stream,
deny /etc/shadow r,
}

AppArmor is simpler than SELinux (path-based vs label-based). Used by default in Ubuntu, SUSE. Kubernetes supports AppArmor annotations for pod security.

4. How do you use SysRq keys for emergency recovery?

Show answer

Magic SysRq keys send commands directly to the kernel, bypassing userspace — useful when the system is unresponsive.

Enable: echo 1 > /proc/sys/kernel/sysrq

Common keys (Alt + SysRq + key):
R — Raw keyboard mode (take back keyboard from X)
E — tErminate all processes (SIGTERM)
I — kIll all processes (SIGKILL)
S — Sync all filesystems
U — Unmount and remount read-only
B — reBoot immediately

Safe reboot sequence: REISUB (mnemonic: Raising Elephants Is So Utterly Boring)
Wait a few seconds between each key.

Useful for: fork bomb recovery (Alt+SysRq+F triggers OOM killer), hung system safe reboot, filesystem sync before hard power-off.

5. How do you diagnose and recover from a full filesystem?

Show answer

When / or another critical filesystem fills up:

1. Identify the culprit:
df -h # which filesystem is full
du -sh /* | sort -rh | head -10 # largest dirs
find / -xdev -size +100M -type f # large files
lsof +L1 # deleted-but-open files holding space

2. Quick relief:
- Truncate large logs: > /var/log/bigfile.log
- Clean package cache: apt clean / yum clean all
- Remove old kernels: apt autoremove
- Clear /tmp: find /tmp -mtime +7 -delete

3. If /tmp is full and you can't create temp files:
Use /dev/shm (tmpfs in RAM) as temporary workspace

4. For deleted-but-open files consuming space:
Restart the process holding the fd, or truncate via /proc//fd/

Prevention: set up monitoring alerts at 80% capacity, use logrotate, separate /var and /tmp.

🔴 Hard (5)

1. chmod -x /bin/chmod was accidentally run. How do you fix it?

Show answer

Several approaches since chmod itself is now non-executable:

1. Use a language interpreter:
perl -e 'chmod 0755, \"/bin/chmod\"'
python3 -c 'import os; os.chmod(\"/bin/chmod\", 0o755)'

2. Use /lib/ld-linux to run the binary directly:
/lib64/ld-linux-x86-64.so.2 /bin/chmod +x /bin/chmod

3. Copy from another system or package:
cp /bin/ls /tmp/fix && cat /bin/chmod > /tmp/fix && /tmp/fix +x /bin/chmod

4. Install from package manager: apt install --reinstall coreutils

The perl approach is the most portable and commonly cited in interviews.

2. How do you recover a deleted file still held open by a process?

Show answer

If a process still has the file open, the data exists in /proc.

1. Find the process:
lsof | grep deleted_file
# or: lsof +L1 (list all deleted-but-open files)

2. Find the file descriptor:
ls -la /proc//fd/
# Look for the symlink pointing to '(deleted)'

3. Recover the content:
cp /proc//fd/ /path/to/recovered_file

This works because Linux doesn't actually free disk blocks until all file descriptors are closed. The inode remains valid as long as any process holds a reference. For databases, use the process's own recovery mechanisms instead of raw fd copying.

3. What does the fork bomb :(){ :|:& };: do and how do you stop it?

Show answer

:(){ :|:& };: defines a function ':' that calls itself twice, piped, in the background — exponentially spawning processes.

Breakdown:
:() — define function named ':'
{ :|:& } — body: call ':' piped to ':' in background
;: — end definition and invoke

Mitigation:
1. Prevention: ulimit -u 100 (limit max processes per user)
Add to /etc/security/limits.conf:
* hard nproc 500

2. Recovery (if you can get a shell):
killall -9 -u
Or from another terminal/SSH session.

3. If system is unresponsive:
Use SysRq keys: Alt+SysRq+F (OOM killer)
Or Alt+SysRq+REISUB for safe reboot.

PAM limits and cgroups are the proper defenses.

4. How do LVM snapshots work and when would you use them?

Show answer

LVM snapshots create a point-in-time copy of a logical volume using copy-on-write.

# Create snapshot (allocate space for changes):
lvcreate -s -n snap_data -L 5G /dev/vg0/data

# Mount and access snapshot:
mount /dev/vg0/snap_data /mnt/snap

# Restore from snapshot:
lvconvert --merge /dev/vg0/snap_data
# Requires unmount + reactivation or reboot

# Remove snapshot:
lvremove /dev/vg0/snap_data

COW means only changed blocks consume snapshot space. Size the snapshot for expected change volume — if it fills up, it becomes invalid. Use cases: consistent backups of active databases, safe upgrade rollback, testing changes. For thin snapshots, use thin provisioning for more efficient multi-snapshot scenarios.

5. How do you recover GRUB when the system won't boot?

Show answer

Common GRUB recovery scenarios:

1. From GRUB rescue prompt:
ls # list partitions
set root=(hd0,gpt2) # set root partition
set prefix=(hd0,gpt2)/boot/grub
insmod normal
normal # boot normally

2. From a live USB:
mount /dev/sda2 /mnt
mount /dev/sda1 /mnt/boot/efi # if UEFI
mount --bind /dev /mnt/dev
mount --bind /proc /mnt/proc
mount --bind /sys /mnt/sys
chroot /mnt
grub-install /dev/sda # reinstall GRUB
update-grub # regenerate config

3. Reinstall from running system:
sudo grub-install /dev/sda
sudo update-grub

For UEFI: ensure EFI partition is mounted, use grub-install --target=x86_64-efi. Check /etc/default/grub for configuration.