Linux Kernel

← Back to all decks

69 cards — 🟢 18 easy | 🟡 30 medium | 🔴 15 hard

🟢 Easy (18)

1. What is Linux and what distinguishes it from other operating systems?

Show answer

[Wikipedia](https://en.wikipedia.org/wiki/Linux): "Linux is a family of open-source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991, by Linus Torvalds. Linux is typically packaged in a Linux distribution."

[Red Hat](https://www.redhat.com/en/topics/linux/what-is-linux): "Linux® is an open source operating system (OS).

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

2. What is a kernel, and what does it do?

Show answer

The kernel is part of the operating system and is responsible for tasks like:

* Allocating memory
* Schedule processes
* Control CPU

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

3. How do you pass kernel boot parameters?

Show answer

Temporarily: edit GRUB (`e`) and modify `linux` line.

Persistently: `/etc/default/grub` → `GRUB_CMDLINE_LINUX` → regenerate config.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

4. What are the differences between Unix, Linux, BSD, and GNU?

Show answer

**GNU** isn't really an OS. It's more of a set of rules or philosophies that govern free software, that at the same time gave birth to a bunch of tools while trying to create an OS. So **GNU** tools are basically open versions of tools that already existed, but were reimplemented to conform to principals of open software.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

5. What is a Linux kernel module and how do you load a new module?

Show answer

A Linux kernel module is a piece of code that can be dynamically loaded into the kernel to extend its functionality. These modules are typically used to add support for hardware devices, filesystems, or system calls. The kernel itself is monolithic, but with modules, its capabilities can be extended without having to reboot the system or recompile the entire kernel.

Remember: lsmod=list, modprobe=load+deps, rmmod=remove, modinfo=details.

Example: `modprobe br_netfilter` — needed for K8s networking.

6. What is User-mode Linux?

Show answer

In Linux, user mode is a restricted operating mode in which a user's application or process runs. User mode is a non-privileged mode that prevents user-level processes from accessing sensitive system resources directly.


Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

7. What is a Linux distribution?

Show answer

* A collection of packages - kernel, GNU, third party apps, ...
* Sometimes distributions store some information on the distribution in `/etc/*-release` file
* For example for Red Hat distribution it will be `/etc/redhat-release` and for Amazon it will be `/etc/os-release`
* `lsb_release` is a common command you can use in multiple different distributions

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

8. What is GRUB2 and what role does it play in the Linux boot process?

Show answer

GRUB2 (GRand Unified Bootloader 2) is the default Linux bootloader.

Functions:
- Loads the kernel and initramfs
- Provides boot menu for multiple OS/kernels
- Supports various filesystems
- Chain-loading other bootloaders

Key files:
- /boot/grub/grub.cfg - Generated config (don't edit)
- /etc/default/grub - User settings
- /etc/grub.d/ - Menu entry scripts

Commands:
- grub-install - Install bootloader
- grub-mkconfig - Regenerate config
- update-grub - Debian/Ubuntu wrapper

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

9. What is the Linux kernel?

Show answer

The Linux kernel is a free, open-source Unix-like kernel – the core of the operating system – which manages hardware and system resources.

Fun fact: Linus Torvalds released it on August 25, 1991: 'I'm doing a (free) operating system (just a hobby, won't be big and professional like gnu).'

Under the hood: the kernel manages CPU scheduling, memory, device drivers, filesystems, and networking. Runs in ring 0 (privileged mode).

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

10. Give some examples of Linux distribution. What is your favorite distro and why?

Show answer

- Red Hat Enterprise Linux
- Fedora
- CentOS
- Debian
- Ubuntu

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

11. What is a Linux distribution (distro)?

Show answer

A complete operating system built around the Linux kernel, bundling system libraries, tools, and often a package manager (examples: Debian, Red Hat).

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

12. What is Libvirt and how does it manage virtual machines?

Show answer

It's an open source collection of software used to manage virtual machines. It can be used with: KVM, Xen, LXC and others. It's also called Libvirt Virtualization API.

From the official [docs](https://libvirt.org/)
Hypervisor supported [docs](https://libvirt.org/drivers.html)

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

13. What is KVM (Kernel-based Virtual Machine) and how does it work?

Show answer

Is an open source virtualization technology used to operate on x86 hardware.

From the official [docs](https://www.linux-kvm.org/page/Main_Page)
Recommended read:
* [Red Hat Article - What is KVM?](https://www.redhat.com/en/topics/virtualization/what-is-KVM)

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

14. How do you view kernel messages with human-readable timestamps and filter for errors?

Show answer

Use dmesg -T for human-readable timestamps. Filter for errors: dmesg -l err,crit,alert,emerg. Combine: dmesg -T -l err,crit,alert,emerg. Use dmesg -w to follow new messages in real-time. This is your first diagnostic command for system-level issues.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

15. What patterns should you grep for in dmesg when troubleshooting hardware or system issues?

Show answer

Hardware: "error|fault|fail|warn". Memory: "oom|out of memory|page allocation failure". Disk: "i/o error|medium error|sector|ata|scsi". Network: "link down|link up|carrier|dropped|reset". CPU: "mce|machine check|thermal|throttl". Filesystem: "ext4|xfs|corrupt|mount|remount".

Remember: dmesg = kernel ring buffer. Hardware, drivers, errors. `dmesg -T` for timestamps.

Example: `dmesg | grep -i error` — quick kernel error scan.

16. What is the difference between a kernel oops and a kernel panic?

Show answer

An oops is a kernel bug that kills the offending process but the system usually continues running (degraded, marked tainted). A panic is fatal -- the kernel cannot continue and the system halts or reboots. An oops can escalate to a panic if panic_on_oops=1 is set.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

17. What is your favorite shell and why?

Show answer

**BASH** is my favorite. It’s really a preferential kind of thing, where I love the syntax and it just "clicks" for me. The input/output redirection syntax (`>>`, `<< 2>&1`, `2>`, `1>`, etc) is similar to C++ which makes it easier for me to recognize.

I also like the **ZSH** shell, because is much more customizable than **BASH**. It has the Oh-My-Zsh framework, powerful context based tab completion, pattern matching/globbing on steroids, loadable modules and more.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

18. What is stored in each of the following logs: /var/log/messages, /var/log/syslog, /var/log/auth.log?

Show answer

These are key system log files:

/var/log/messages (RHEL/CentOS):
- General system messages
- Kernel messages
- Service start/stop
- Hardware events

/var/log/syslog (Debian/Ubuntu):
- Similar to messages
- General system log
- Most daemon output

/var/log/auth.log (or secure):
- Authentication events
- SSH logins/failures
- sudo usage
- PAM messages

With systemd, also use:
- journalctl - All logs
- journalctl -u service - Specific service
- journalctl -p err - By priority

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

🟡 Medium (30)

1. Are wildcards implemented in user or kernel space?

Show answer

User space - specifically in the shell.

How it works:
1. You type: ls *.txt
2. Shell (bash) sees the wildcard
3. Shell expands *.txt to matching files
4. Shell calls exec with expanded list
5. ls receives: ls file1.txt file2.txt file3.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

2. Are the changes you make to kernel parameters in a container, affects also the kernel parameters of the host on which the container runs?

Show answer

No. Containers have their own /proc filesystem so any change to kernel parameters inside a container, are not affecting the host or other containers running on that host.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

3. In what phases of kernel lifecycle, can you change its configuration?

Show answer

* Build time (when it's compiled)
* Boot time (when it starts)
* Runtime (once it's already running)

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

4. What virtualization solutions are available for Linux?

Show answer

* [KVM](https://www.linux-kvm.org/page/Main_Page)
* [XEN](http://www.xen.org/)
* [VirtualBox](https://www.virtualbox.org/)
* [Linux-VServer](http://linux-vserver.org/Welcome_to_Linux-VServer.org)
* [User-mode Linux](http://user-mode-linux.sourceforge.net/)
* ...

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

5. Where can you find the file that contains the command passed to the boot loader to run the kernel?

Show answer

`/proc/cmdline`

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

6. Where can you find kernel's configuration?

Show answer

Usually it will reside in `/boot/config-..`

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

7. How changes to kernel runtime parameters persist? (applied even after reboot to the system for example)

Show answer

There is a service called `systemd-sysctl` that takes the content of /etc/sysctl.conf and applies it. This is how changes persist, even after reboot, when they are written in /etc/sysctl.conf

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

8. What are the components of a Linux distribution?

Show answer

* Kernel
* Utilities
* Services
* Software/Packages Management

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

9. How to list kernel's runtime parameters?

Show answer

`sysctl -a` — lists all kernel runtime parameters (from the /proc/sys tree) and their current values. Use `sysctl ` to read one parameter or `sysctl -w =` to change one at runtime.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

10. Under which license Linux is distributed?

Show answer

GPL v2 (GNU General Public License version 2). This means anyone can use, modify, and distribute the kernel, but modifications must also be released under GPL v2.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

11. How to change the kernel parameters? What kernel options might you need to tune?

Show answer

To set the kernel parameters in Unix-like, first edit the file `/etc/sysctl.conf` after making the changes save the file and run the command `sysctl -p`, this command will make the changes permanently without rebooting the machine.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

12. What Linux distributions are you familiar with?

Show answer

Major distribution families:

Red Hat family:
- RHEL (enterprise, paid support)
- CentOS/Rocky/Alma (RHEL clones)
- Fedora (cutting edge)
- Uses: rpm, dnf/yum, systemd

Debian family:
- Debian (stable, slow)

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

13. Explain user space vs. kernel space

Show answer

The operating system executes the kernel in protected memory to prevent anyone from changing (and risking it crashing). This is what is known as "Kernel space".
"User space" is where users executes their commands or applications. It's important to create this separation since we can't rely on user applications to not tamper with the kernel, causing it to crash.

Applications can access system resources and indirectly the kernel space by making what is called "system calls".

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

14. True or False? both /tmp and /var/tmp cleared upon system boot

Show answer

False. /tmp is cleared upon system boot while /var/tmp is cleared every a couple of days or not cleared at all (depends on distro).

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

15. How do you find out which Kernel version your system is using?

Show answer

`uname -a` command

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

16. What is a system call? What system calls are you familiar with?

Show answer

System calls are the interface between user programs and the kernel.

Purpose:
- Request kernel services
- Access hardware resources
- Perform privileged operations

Common system calls:
- File: open, read, write, close, stat
- Process: fork, exec, exit, wait, kill

Remember: Syscalls = user↔kernel interface. `strace` traces them. Common: open, read, write, fork.

17. How to permanently set $PATH on Linux/Unix? Why is this variable so important?

Show answer

Your console has two types of messages:

- **generated by the kernel** (via printk)
- **generated by userspace** (usually your init system)

Kernel messages are always stored in the **kmsg** buffer, visible via `dmesg` command. They're also often copied to your **syslog**. This also applies to userspace messages written to `/dev/kmsg`, but those are fairly rare.


Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

18. How sysctl applies the changes to kernel's runtime parameters the moment you run sysctl command?

Show answer

If you `strace` the sysctl command you can see it does it by changing the file under /proc/sys/...

In the past it was done with sysctl system call, but it was deprecated at some point.

Example: `sysctl -w net.ipv4.ip_forward=1` — runtime. Persist: `/etc/sysctl.d/99-custom.conf`.

Gotcha: `sysctl -w` is temporary. Always persist and run `sysctl --system`.

19. If I plug a new device into a Linux machine, where and how does the detection process start?

Show answer

Device detection involves kernel, udev, and hardware:

1. Hardware detection:
- Kernel driver detects device (USB, PCI, etc.)
- Creates device in kernel data structures

2. Kernel notification:
- Kernel creates kobject in /sys
- Sends uevent to userspace

3. udev processing:
- udevd receives uevent

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

20. Will running sysctl -a as a regular user vs. root, produce different result?

Show answer

Yes, you might notice that in most systems, when running `systctl -a` with root, you'll get more runtime parameters compared to executing the same command with a regular user.

Example: `sysctl -w net.ipv4.ip_forward=1` — runtime. Persist: `/etc/sysctl.d/99-custom.conf`.

Gotcha: `sysctl -w` is temporary. Always persist and run `sysctl --system`.

21. How to make sure a Service starts automatically after a reboot or crash?

Show answer

Depends on the init system.

Systemd: ` systemctl enable [service_name] `
System V: ` update-rc.d [service_name] ` and add this line ` id:5678:respawn:/bin/sh /path/to/app ` to /etc/inittab
Upstart: add Upstart init script at /etc/init/service.conf

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

22. How do you perform server OS installation and configuration?

Show answer

Server OS installation and configuration involve the following steps: **Prepare Installation Media:* • Create a bootable installation media, such as a USB drive or DVD, containing the server OS. **Boot from Installation Media:* • • Insert the installation media into the server. • Boot the server from the installation media. **Follow Installation Wizard:* • Follow the prompts of the installation wizard, which typically involves selecting language, time zone, and keyboard layout. **Partitioning and Disk Setup:* • • Choose the disk or partition where the OS will be installed.

23. What is kdump and how does it capture crash dumps during a kernel panic?

Show answer

kdump reserves a small amount of memory at boot for a second (crash) kernel. During a panic, the crash kernel activates and writes the contents of memory (vmcore) to disk at /var/crash/. Setup: install kexec-tools, enable kdump service, ensure crashkernel=256M is in the kernel command line. Without kdump, crash forensic evidence is lost.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

24. What is the SysRq REISUB sequence, and when would you use it?

Show answer

REISUB is a safe emergency reboot sequence when the system is hung: R (un-Raw keyboard), E (tErminate all, SIGTERM), I (kIll all, SIGKILL), S (Sync disks), U (Unmount/remount read-only), B (reBoot). Wait 2-5 seconds between each key. This is the cleanest reboot when nothing else works. Enable with kernel.sysrq=1 in /etc/sysctl.d/.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

25. What does it mean when a kernel is "tainted," and why does it matter?

Show answer

A tainted kernel has been modified from its pristine state. Common taint flags: P (proprietary module like nvidia), F (module force-loaded), W (warning/oops occurred), E (unsigned module). Check with cat /proc/sys/kernel/tainted (0=clean). Tainted kernels may affect vendor support and bug report acceptance.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

26. How does the OOM killer work, and how do you investigate OOM kill events?

Show answer

When the system runs out of memory, the OOM killer selects and kills a process to free memory. Check for OOM kills: dmesg | grep -i "out of memory\|oom-killer\|killed process". Adjust OOM priority: echo -1000 > /proc//oom_score_adj (never kill) or echo 1000 (kill first). The OOM killer is a symptom -- fix the memory pressure, do not just restart the killed process.

Remember: OOM killer picks by oom_score. Protect: `echo -1000 > /proc//oom_score_adj`.

Under the hood: Activates when RAM+swap exhausted. Kills highest score to free most memory.

27. What SysRq keys help debug hung systems without rebooting?

Show answer

t = dump task states (debug hung processes), m = dump memory info (debug memory issues), w = dump blocked D-state tasks (debug I/O hangs), s = sync all filesystems, e = send SIGTERM to all processes. Access via Alt+SysRq+ on console, or echo > /proc/sysrq-trigger remotely. These work even when the system is mostly unresponsive.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

28. How to generate a random string?

Show answer

One way is to run the following: `cat /proc/sys/kernel/random/uuid`

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

29. True or False? only root can create files in /proc

Show answer

False. No one can create file in /proc directly (certain operations can lead to files being created in /proc by the kernel).

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

30. Where are Linux system logs located, and what are the key log files?

Show answer

/var/log is the standard log directory. Key files: `/var/log/syslog` or `/var/log/messages` (general system logs), `/var/log/auth.log` or `/var/log/secure` (authentication events), `/var/log/kern.log` (kernel messages), `/var/log/dmesg` (boot messages). On systemd systems, use `journalctl` to query the journal. Log rotation is managed by `logrotate`.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

🔴 Hard (15)

1. What is the use of ulimit in Unix-like systems?

Show answer

Most Unix-like operating systems, including Linux and BSD, provide ways to limit and control the usage of system resources such as threads, files, and network connections on a per-process and per-user basis. These "**ulimits**" prevent single users from using too many system resources.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

2. If a server is "unresponsive" but still pings, what do you suspect?

Show answer

The kernel's network stack is alive, but userspace is frozen or resource-starved.

Likely causes:
- OOM (Out of Memory): Kernel killing processes, system thrashing
- CPU lockup: Runaway process or kernel bug consuming all CPU
- Kernel oops/soft lockup: Partial kernel failure, userspace frozen
- Resource exhaustion: PID limit, file descriptor limit, memory
- D state processes: Mass I/O wait

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

3. How do you debug boot failures remotely?

Show answer

Without physical access, you need out-of-band management:

1. **Serial/IPMI console**: Connect via iLO, iDRAC, IPMI for console access during boot

2. **Recovery options**:
- Boot to previous kernel via GRUB
- Drop to initramfs shell (`rd.break` kernel param)
- Boot to rescue mode (`systemd.unit=rescue.target`)

3.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

4. What is a "Kernel Panic" and how do you debug it post-mortem?

Show answer

A kernel panic is a fatal kernel error that forces the system to halt.

What causes kernel panics:
- Critical hardware failures
- Kernel bugs or driver issues
- Memory corruption
- Null pointer dereference in kernel code
- Unrecoverable filesystem errors

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

5. How do you safely update the kernel in production with minimal risk?

Show answer

Use a staged rollout approach with multiple safety mechanisms.

Deployment strategy:
1. Test on non-production environment first
2. Canary deployment to small subset of prod (1-5%)
3. Monitor for issues, gradually expand rollout
4.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

6. grub> vs grub-rescue>. Explain.

Show answer

- `grub>` - this is the mode to which it passes if you find everything you need to run the system in addition to the configuration file. With this mode, we have access to most (if not all) modules and commands.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

7. Describe start-up configuration files and directory in BSD systems.

Show answer

In BSD the primary start-up configuration file is `/etc/defaults/rc.conf`. System startup scripts such as `/etc/rc` and `/etc/rc.d` just include this file.

If you want to add other programs to system startup you need to change `/etc/rc.conf` file instead of `/etc/defaults/rc.conf`.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

8. A server suddenly becomes unresponsive but you can still ping it. How do you debug without rebooting?

Show answer

Use Magic SysRq keys - they communicate directly with the kernel, bypassing userspace.

Enable SysRq if not already:
- `echo 1 > /proc/sys/kernel/sysrq`

Useful SysRq triggers:
- `echo t > /proc/sysrq-trigger` - show task states (stack traces)
- `echo w > /proc/sysrq-trigger` - show blocked (D state) tasks
- `echo m > /proc/sysrq-trigger` - show memory info
- `echo l > /proc/sysrq-trigger

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

9. How does Linux boot, end to end?

Show answer

Firmware → bootloader (GRUB) → kernel → initramfs → kernel mounts root FS → `systemd` PID 1 → targets/services. Most boot issues live in initramfs, fstab, or broken units.

The detailed sequence:
1. **BIOS/UEFI**: Hardware initialization, POST, loads bootloader
2. **GRUB**: Loads kernel and initramfs into memory
3.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

10. What are the different types of kernels? Explain.

Show answer

**Monolithic Kernels**

Earlier in this type of kernel architecture, all the basic system services like a process and memory management, interrupt handling etc were packaged into a single module in kernel space.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

11. You need to debug a kernel panic that only happens under heavy load. What do you configure?

Show answer

Set up kdump to capture kernel crash dumps for post-mortem analysis.

Configuration steps:
1. Install kdump: `yum install kexec-tools` or `apt install kdump-tools`
2. Configure GRUB - add to kernel cmdline:
- `crashkernel=auto` (or specific size like `crashkernel=256M`)
- Edit /etc/default/grub, run grub2-mkconfig
3. Configure /etc/kdump.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

12. How do you analyze a kernel crash dump using the crash utility, and how do you read a backtrace?

Show answer

Install crash and kernel-debuginfo. Open dump: crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux /var/crash/*/vmcore. Key commands: bt (backtrace), log (kernel log at crash time), ps (process list), sys (system info). Read backtraces bottom-up: the lowest frame is where the problem started, the root cause is usually in the middle frames.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

13. What are Machine Check Exceptions (MCEs), and how do you diagnose them?

Show answer

MCEs are hardware errors reported by the CPU. Common causes: faulty RAM (diagnose with memtest86+), overheating CPU (check thermal sensors), or failing CPU (needs replacement). Check with dmesg | grep -i "machine check\|mce". Install mcelog for detailed analysis. MCEs indicate real hardware problems that cannot be fixed with software.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

14. How do you configure kdump for remote crash dump storage, and what does the dump level (-d flag) control?

Show answer

Configure in /etc/kdump.conf. For NFS: nfs server:/crash-dumps. For SSH: ssh user@server with sshkey path. The core_collector makedumpfile -d 31 flag controls what to exclude from the dump: 1=zero pages, 2=cache pages, 4=cache private, 8=user pages, 16=free pages. -d 31 excludes all (smallest dump). Lower values produce larger but more complete dumps.

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.

15. How to log all commands run by root on production servers?

Show answer

`auditd` is the correct tool for the job here:

1. Add these 2 lines to `/etc/audit/audit.rules`:

```bash\n-a exit,always -F arch=b64 -F euid=0 -S execve\n-a exit,always -F arch=b32 -F euid=0 -S execve\n```

These will track all commands run by root (euid=0). Why two rules? The execve syscall must be tracked in both 32 and 64 bit code.

2. To get rid of `auid=4294967295` messages in logs, add `audit=1` to the kernel's cmdline (by editing `/etc/default/grub`)

3. Place the line

Remember: Kernel manages hardware, memory, processes, I/O. User programs interact via syscalls.

Under the hood: Monolithic kernel with loadable modules. Extend without rebooting.