Interview Cheatsheet Linux

lesson ---# Linux Interview Cheatsheet

Mental model: Kernel (CPU/mem/devices/net) + userspace (systemd, shells, daemons, tools). Syscall boundary = apps ask kernel via open/read/write/fork/exec/socket.

Boot chain: UEFI/BIOS → GRUB → kernel (decompresses itself) → initramfs (finds root, loads drivers, unlocks LUKS) → PID 1 (systemd) → services - PID 1 is special: can't be killed, kernel panics if it exits, reaps orphans - systemd-analyze blame shows slowest services | cat /proc/cmdline shows kernel params - Initramfs fails? Wrong root UUID, missing driver, stale initramfs → rebuild: update-initramfs -u (Debian) / dracut --force (RHEL) - Recovery: add systemd.unit=rescue.target or init=/bin/bash to kernel cmdline in GRUB

systemd¶

systemctl status/start/stop/restart/enable/disable/mask | systemctl list-units --failed | systemctl daemon-reload

Logs: journalctl -u svc -f (follow) | --since "1 hour ago" | -p err -b (errors since boot) | -xe (recent errors+context)

Unit file directives:

Directive	Meaning
`After=`	Ordering only (start after X)
`Wants=`	Soft dep — start X, don't fail if X fails
`Requires=`	Hard dep — fail if X fails
`Type=simple`	Process stays foreground (most common)
`Type=forking`	Process forks, parent exits (legacy daemons)
`Restart=on-failure`	Auto-restart on non-zero exit
`RestartSec=5`	Wait between restarts
`MemoryMax=512M`	cgroup memory limit
`CPUQuota=200%`	cgroup CPU limit (200% = 2 cores)

Security directives: NoNewPrivileges=yes PrivateTmp=yes ProtectSystem=strict ProtectHome=read-only

Key traps: - network.target != "network ready" — use network-online.target only when service truly needs connectivity - "Works in shell but not systemd" → different env/PATH/working dir/user → systemctl show svc -p Environment - Drop-in overrides: systemctl edit nginx (don't edit packaged unit files) - enable = on boot | start = now | mask = prevent ANY start (symlink to /dev/null)

Timers (cron replacement): OnCalendar=*-*-* 02:00:00 + Persistent=true | systemctl list-timers

Processes and Signals¶

Lifecycle: fork() → exec() → wait() → exit()

State	Symbol	Meaning
Running	R	On CPU or in run queue
Sleeping	S	Waiting for event (interruptible)
Disk sleep	D	Waiting for I/O (cannot be killed — fix the I/O)
Zombie	Z	Exited, parent hasn't called wait()
Stopped	T	Ctrl+Z or SIGSTOP

Permissions¶

-rwxr-xr-- = user(rwx) group(r-x) other(r--)

- Files: r=read, w=write, x=execute | Dirs: r=list names, x=traverse/enter, w=create/delete entries - chmod 755 (rwxr-xr-x) | chmod 644 (rw-r--r--) | chown user:group file - SUID(4000)=run as owner | SGID(2000)=inherit group on dirs | Sticky(1000)=only owner deletes (/tmp) - umask 0022 → new files 644, new dirs 755 | find / -perm -4000 -ls = audit SUID binaries - ACLs: getfacl file / setfacl -m u:deploy:rx file — fine-grained beyond user/group/other - Capabilities: setcap cap_net_bind_service=+ep binary — specific root powers without full root - SELinux: getenforce | ausearch -m avc -ts recent | restorecon -Rv /path | AppArmor: profile-based MAC

Filesystems and Storage¶

Filesystem concepts: Inode = metadata + block pointers (NOT the filename) | Hard link = same inode | Symlink = path pointer (can break)

FS	Strengths	Caveat
ext4	General purpose, mature	Can shrink offline
XFS	Fast at scale, default on RHEL	Cannot shrink — ever
Btrfs	Snapshots, checksums, compression	Newer, more complex
tmpfs	RAM-backed (/tmp, /run)	Lost on reboot

df -h (space) | df -i (inodes — 0% disk used + 100% inodes = cannot create files) | findmnt (mount tree)
lsblk -f (devices+FS) | blkid (UUIDs) | mount | column -t

LVM: PV → VG → LV → filesystem - Create: pvcreate /dev/sdb1 → vgcreate data /dev/sdb1 → lvcreate -L 50G -n app data → mkfs.ext4 /dev/data/app - Extend (online!): lvextend -L +20G /dev/data/app then resize2fs (ext4) or xfs_growfs (XFS) - Status: pvs vgs lvs

Other: LUKS = block encryption (unlocked in initramfs) | RAID: 0(stripe) 1(mirror) 5(parity) 10(mirror+stripe) - Disk health: smartctl -a /dev/sda | I/O stats: iostat -xz 1 3 | Per-process I/O: iotop - Deleted-but-open files still hold space: lsof +L1 | Mount security: noexec,nosuid,nodev on /tmp

Memory¶

MemAvailable matters, not MemFree — Linux uses free RAM for page cache (normal and good)
OOM killer fires when mem+swap exhausted: dmesg -T | grep -i oom | free -h | ps aux --sort=-%mem | head
vmstat 1 5 — si/so columns = swap in/out (any activity = memory pressure)
Virtual memory: each process gets own address space, mapped to physical via page tables
Swap on SSD is fine | sysctl vm.swappiness (0=only swap to avoid OOM, 60=default)
slabtop = kernel slab cache | /proc/meminfo = full breakdown | /proc/PID/smaps_rollup = per-process

Networking¶

Basics: ip addr (IPs) | ip route (routing) | ss -tlnp (listening TCP+PIDs) | ip neigh (ARP) | ip link (interfaces)

DNS path: Application → NSS (Name Service Switch) → resolver (systemd-resolved or direct) → DNS server - getent hosts name = what system actually resolves (includes /etc/hosts, NSS) - dig name +short = direct DNS query (bypasses NSS) | resolvectl status = systemd-resolved state

TCP states that matter:

State	Meaning	Concern
LISTEN	Waiting for connections	Normal for servers
ESTABLISHED	Active connection	Normal
TIME_WAIT	Connection closed, expiring	High count = many short connections
CLOSE_WAIT	Remote closed, local didn't	Application bug — not calling close()
SYN_SENT	Connecting	High count = upstream unreachable

ss -tn state close-wait = find connection leaks | ss -s = socket summary

Firewalls: nftables = modern (nft list ruleset) | iptables = legacy but Docker/K8s/fail2ban still emit iptables rules - Rule order: first match wins | Always put ESTABLISHED,RELATED + SSH ACCEPT before DROP - iptables -L -n -v --line-numbers | firewall-cmd --list-all (RHEL) | ufw status verbose (Ubuntu)

Debug: tcpdump -i eth0 port 80 -nn | curl -v telnet://host:port (TCP test) | traceroute host | mtr host

Troubleshooting — 60-second triage¶

uptime && nproc              # load avg vs CPU count (>2x = saturated)
dmesg -T | tail -20          # kernel messages: OOM? disk errors? panics?
free -h                      # memory pressure?
df -h && df -i               # disk space AND inodes
iostat -xz 1 3               # disk I/O: await=latency, %util=saturation
ss -s                        # socket summary: connection leaks?
ps aux --sort=-%cpu | head   # CPU hogs
systemctl list-units --failed # broken services

Decision patterns: - Stuck process → strace -p PID → shows blocked syscall → ls -la /proc/PID/fd/N → what it's waiting on - High load, low CPU → I/O wait: iostat -xz 1 → which process: iotop -a - High load, high CPU → ps aux --sort=-%cpu | head → known process or runaway? - Service won't start → systemctl status → journalctl -xeu svc → port conflict: ss -tlnp | grep PORT - Disk full → du -sh /* | sort -rh | head → logs? journalctl --vacuum-size=500M → deleted-open: lsof +L1

Advanced tools: strace -p PID -e trace=network | /proc/PID/environ (env vars, including secrets) | /proc/PID/fd/ (all open files) - eBPF/bpftrace for production tracing without strace overhead | perf top for CPU profiling

Containers: namespaces(isolation) + cgroups(resource limits) + filesystem layers = containers (not tiny VMs) - cgroup v2 = modern unified hierarchy | systemd-cgtop = live cgroup resource view

30-second answer¶

"I troubleshoot bottom-up: service state and logs first, then resources — CPU, memory, disk space AND inodes, I/O wait. If it's mysterious I go to strace and /proc. I think in systemd, cgroup v2, iproute2. For performance I use the USE method — utilization, saturation, errors for each resource. For any incident, I separate what I can observe from what I'm guessing, and I prefer reversible actions over irreversible ones."