--- tags: - linux - l1 - flashcard-deck - linux-performance --- [Portal](../../../../library/portal/index.md) | **Level:** [L1: Foundations](../../../../library/portal/levels.md) | **Topics:** [Linux Performance Tuning](../../../../library/portal/topics.md) | **Domain:** Linux id category difficulty tags question answer source_path linux-performance/070ae4cdb319 linux-performance medium linux, kernel, limits, file-descriptors, tuning Explain the difference between ulimit -n and fs.file-max — how do they interact? These are two different layers of file descriptor limits.\n\nulimit -n (per-process limit):\n- Soft and hard limits per process\n- Configured in /etc/security/limits.conf\n- Syntax: ` nofile `\n- Example: `* hard nofile 65535`\n- Check current: `ulimit -n` (soft), `ulimit -Hn` (hard)\n\nRemember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).\n\nRemember: USE method: Utilization, Saturation, Errors for each resource. projects/knowledge/interview/linux/521-ulimit-vs-file-max.txt linux-performance/0853314c93c0 linux-performance hard cpu, io, linux, memory, system High load average but low CPU usage - why? Load average includes both runnable AND uninterruptible (D state) processes. Low CPU with high load means processes are blocked waiting on something:\n\nCommon causes:\n* **I/O wait**: Disk saturation, slow storage\n* **NFS latency**: Hung NFS mounts\n* **Blocked threads**: Mutex contention, lock waits\n* **Storage issues**: SAN latency, RAID rebuild\n\n\n\nRemember: Load avg = processes in run/IO-wait over 1/5/15 min. 4-core: load 4.0 = 100%.\n\nGotcha: Load includes I/O wait. High load + low CPU = disk bottleneck. projects/knowledge/interview/linux/451-high-load-but-low-cpu-usage-why.txt linux-performance/29cefd14d784 linux-performance easy benchmarking, cpu, linux, memory, system What is load average? Linux **load averages** are "system load averages" that show the running thread (task) demand on the system as an average number of running plus waiting threads. This measures demand, which can be greater than what the system is currently processing. Most tools show three averages, for 1, 5, and 15 minutes.\n\n\nRemember: Load avg = processes in run/IO-wait over 1/5/15 min. 4-core: load 4.0 = 100%.\n\nGotcha: Load includes I/O wait. High load + low CPU = disk bottleneck. projects/knowledge/interview/linux/012-what-is-load-average.txt linux-performance/2fba3801873b linux-performance medium linux, profiling, system You have added several aliases to `.profile`. How to reload shell without exit? The best way is `exec $SHELL -l` because `exec` replaces the current process with a new one. Also good (but other) solution is `. ~/.profile`.\n\nRemember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).\n\nRemember: USE method: Utilization, Saturation, Errors for each resource. projects/knowledge/interview/linux/076-you-have-added-several-aliases-to-codeprofilecode-.txt linux-performance/37db55e35f13 linux-performance medium benchmarking, cpu, io, linux, monitoring, performance What is the difference between CPU load and utilization? They measure different things:\n\nCPU Utilization:\n- Percentage of time CPU is busy (0-100%)\n- Measured by: top, mpstat\n\nCPU Load (Load Average):\n- Number of processes wanting CPU + waiting for I/O\n- Can exceed number of cores\n- Measured by: uptime, /proc/loadavg\n\nKey insight: Linux load includes D-state (I/O wait) processes.\n- High load + low CPU util = I/O bottleneck\n- High load + high CPU util = CPU bottleneck\n\nRemember: CPU: us(user), sy(system), wa(IO wait), st(stolen). High wa = disk problem. projects/knowledge/interview/linux/334-what-is-the-difference-between-cpu-load-and-utiliz.txt linux-performance/3d54922f1401 linux-performance medium cpu, io, linux, memory, system A Linux server is slow. Where do you start? Systematic approach - don't guess, validate bottlenecks:\n\n1. **Load**: `uptime` - is the system under pressure?\n2. **CPU**: `top`/`htop` - check steal time (VM), iowait, user vs system\n3. **Memory**: `free -h`, check for swapping (`vmstat 1`)\n4. **Disk I/O**: `iostat -x 1`, `iotop` - check await, %util\n5. **Network**: `ss -s`, `iftop` if network-bound\n6.\n\nRemember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).\n\nRemember: USE method: Utilization, Saturation, Errors for each resource. projects/knowledge/interview/linux/450-a-linux-server-is-slow-where-do-you-start.txt linux-performance/4d75b2372b04 linux-performance hard cpu, io, linux, system What does CPU jumps mean? An OS is a very busy thing, particularly so when you have it doing something (and even when you aren't). And when we are looking at an active enterprise environment, something is always going on.\n\nMost of this activity is "bursty", meaning processes are typically quiescent with short periods of intense activity. This is certainly true of any type of network-based activity (e.g.\n\nRemember: CPU: us(user), sy(system), wa(IO wait), st(stolen). High wa = disk problem. projects/knowledge/interview/linux/102-what-does-cpu-jumps-mean.txt linux-performance/53e7bfb6d855 linux-performance easy linux, debugging, strace, syscalls How do you trace system calls? `strace ` or `strace -f -p `.\n\nUse `-e` filters. For lower overhead: `perf trace`.\n\nRemember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).\n\nRemember: USE method: Utilization, Saturation, Errors for each resource. projects/knowledge/interview/linux/551-trace-system-calls.txt linux-performance/54e23c3b3b42 linux-performance hard linux, performance, troubleshooting, load-average, io-wait How would you debug high load average with almost no CPU usage? High load with low CPU indicates processes in uninterruptible sleep (D state), typically waiting on I/O.\n\nDiagnostic approach:\n1. Identify D state processes: `top` or `htop` - look for 'D' in state column\n2. Check I/O metrics: `iostat -x 1` - look at %util, await, avgqu-sz\n3. System overview: `vmstat 1` - check 'b' column (blocked processes)\n4.\n\nRemember: Load avg = processes in run/IO-wait over 1/5/15 min. 4-core: load 4.0 = 100%.\n\nGotcha: Load includes I/O wait. High load + low CPU = disk bottleneck. projects/knowledge/interview/linux/513-high-load-average-low-cpu-usage.txt linux-performance/6027b0951f7e linux-performance medium bottleneck, linux You know how to see the load average, great. but what each part of it means? for example 1.43, 2.34, 2.78 [This article](http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html) summarizes the load average topic in a great way\n\nRemember: Load avg = processes in run/IO-wait over 1/5/15 min. 4-core: load 4.0 = 100%.\n\nGotcha: Load includes I/O wait. High load + low CPU = disk bottleneck. projects/knowledge/interview/linux/219-you-know-how-to-see-the-load-average-great-but-wha.txt linux-performance/696b24925075 linux-performance medium benchmarking, cpu, io, linux, memory, performance How you measure time execution of a program? Several methods:\n\n1. time command: time ./program\n - real = wall clock, user = user CPU, sys = kernel CPU\n\n2. /usr/bin/time -v: Detailed stats including memory\n\n3. perf stat: CPU cycles, cache misses\n\n4. hyperfine: Benchmarking with statistics\n\nInterpretation:\n- real > user+sys = I/O or sleep\n- user+sys > real = multi-core\n\nRemember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).\n\nRemember: USE method: Utilization, Saturation, Errors for each resource. projects/knowledge/interview/linux/335-how-you-measure-time-execution-of-a-program.txt linux-performance/6a18a36ba579 linux-performance easy bottleneck, linux How to check what is the current load average? One can use `uptime` or `top`\n\nRemember: Load avg = processes in run/IO-wait over 1/5/15 min. 4-core: load 4.0 = 100%.\n\nGotcha: Load includes I/O wait. High load + low CPU = disk bottleneck. projects/knowledge/interview/linux/218-how-to-check-what-is-the-current-load-average.txt linux-performance/78f5e684ca0a linux-performance medium linux, performance, troubleshooting, cpu You found a server with high CPU load but it's not clear which process is causing it. How would you troubleshoot? Systematic approach to find CPU hogs:\n\n1. Real-time monitoring:\n - top (press 1 for per-CPU view)\n - htop (more user-friendly)\n - Look for high %CPU processes\n\n2. Sort by CPU:\n - ps aux --sort=-%cpu | head\n - ps -eo pid,ppid,%cpu,cmd --sort=-%cpu\n\n3. Check load average:\n\nRemember: CPU: us(user), sy(system), wa(IO wait), st(stolen). High wa = disk problem. projects/knowledge/interview/linux/412-you-found-there-is-a-server-with-high-cpu-load-but.txt linux-performance/8667e65aeddf linux-performance easy cpu, io, linux, memory, system Your first 5 commands on a *nix server after login. - `w` - a lot of great information in there with the server uptime\n- `top` - you can see all running processes, then order them by CPU, memory utilization and more\n- `netstat` - to know on what port and IP your server is listening on and what processes are using those\n- `df` - reports the amount of available disk space being used by file systems\n- `history` - tell you what was previously run by the user you are currently connected to projects/knowledge/interview/linux/006-your-first-5-commands-on-a-nix-server-after-login.txt linux-performance/9ae011fa83f9 linux-performance medium cpu, io, linux, memory, performance, troubleshooting How do you troubleshoot a Linux system that's acting slow? I start by checking CPU, I/O wait, memory pressure, and storage latency — top, iostat, vmstat, dstat, and logs. Then I look at runaway processes, misbehaving services, and disk space. From there I verify systemd units, network paths, and kernel messages. My approach is always layered: symptoms → resource limits → logs → root cause.\n\nRemember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).\n\nRemember: USE method: Utilization, Saturation, Errors for each resource. projects/knowledge/interview/linux/447-how-do-you-troubleshoot-a-linux-system-thats-acti.txt linux-performance/b0bdf20e8890 linux-performance easy linux, performance, load-average, cpu, monitoring Explain the difference between "Load Average" and "CPU Utilization." These metrics measure different aspects of system performance.\n\nCPU Utilization:\n- Percentage of time CPU was busy (not idle)\n- Ranges from 0% to 100% per CPU core\n- Measured via /proc/stat (user, system, idle, iowait, etc.)\n- High CPU = CPU is actively processing work\n- Tools: top, mpstat, sar\n\nLoad Average:\n\nRemember: Load avg = processes in run/IO-wait over 1/5/15 min. 4-core: load 4.0 = 100%.\n\nGotcha: Load includes I/O wait. High load + low CPU = disk bottleneck. projects/knowledge/interview/linux/525-load-average-vs-cpu-utilization.txt linux-performance/d81c1d9a71cc linux-performance medium bottleneck, cpu, monitoring, performance What do the three load average numbers represent in Linux (shown by `uptime` or `top`)? The three numbers represent the average number of processes in a runnable or\nuninterruptible state over 1, 5, and 15 minute intervals.\n\n- 1-minute average: Short-term load, shows recent activity\n- 5-minute average: Medium-term trend\n- 15-minute average: Long-term trend\n\nInterpretation on a single-core system:\n- 1.0 = CPU is exactly at capacity\n- Below 1.0 = CPU has idle time\n\nRemember: top: PID, PR, NI, VIRT, RES, %CPU, %MEM. htop = prettier with mouse. projects/knowledge/interview/linux/002-load-average.txt linux-performance/da51a7d20fa6 linux-performance hard cpu, io, linux, memory, system Load averages are above 30 on a server with 24 cores but CPU shows around 70 percent idle. One of the common causes of this condition is? How to debug and fixed? Requests which involve disk I/O can be slowed greatly if cpu(s) needs to wait on the disk to read or write data. I/O Wait, is the percentage of time the CPU has to wait on disk.\n\nLets looks at how we can confirm if disk I/O is slowing down application performance by using a few terminal command line tools (`top`, `atop` and `iotop`).\n\n\nRemember: Load avg = processes in run/IO-wait over 1/5/15 min. 4-core: load 4.0 = 100%.\n\nGotcha: Load includes I/O wait. High load + low CPU = disk bottleneck. projects/knowledge/interview/linux/108-load-averages-are-above-30-on-a-server-with-24-cor.txt linux-performance/dd16a04dd092 linux-performance medium benchmarking, cpu, io, iostat, linux, monitoring, performance Explain iostat output iostat shows CPU and I/O statistics. Key columns:\n\nCPU section:\n- %user, %system - User/kernel CPU time\n- %iowait - CPU waiting for I/O\n- %idle - Idle CPU time\n\nDevice section (iostat -x):\n- r/s, w/s - Reads/writes per second\n- rkB/s, wkB/s - KB read/written per second\n- await - Average I/O wait time (ms)\n- %util - Device utilization percentage\n\nKey insights:\n- High %iowait = I/O bottleneck\n- High await = slow storage\n- %util near 100% = device saturated (HDDs)\n\nExample: `iostat -xz 1` — %util(busy), await(latency), r/s & w/s (IOPS). projects/knowledge/interview/linux/332-explain-iostat-output.txt linux-performance/e04b5db3c8d1 linux-performance medium cpu, linux, profiling What are you using for debugging CPU related issues? `top` will show you how much CPU percentage each process consumes\n`perf` is a great choice for sampling profiler and in general, figuring out what your CPU cycles are "wasted" on\n`flamegraphs` is great for CPU consumption visualization (http://www.brendangregg.com/flamegraphs.html)\n\nRemember: CPU: us(user), sy(system), wa(IO wait), st(stolen). High wa = disk problem. projects/knowledge/interview/linux/181-what-are-you-using-for-debugging-cpu-related-issue.txt linux-performance/e78b0b925751 linux-performance medium bottleneck, io, linux You get a call from someone claiming "my system is SLOW". What do you do? * Check with `top` for anything unusual\n* Run `dstat -t` to check if it's related to disk or network.\n* Check if it's network related with `sar`\n* Check I/O stats with `iostat`\n\nRemember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).\n\nRemember: USE method: Utilization, Saturation, Errors for each resource. projects/knowledge/interview/linux/182-you-get-a-call-from-someone-claiming-my-system-is-.txt linux-performance/f555ef8531c8 linux-performance hard benchmarking, cpu, linux, memory, system Explain interrupts and interrupt handlers in Linux. Here's a high-level view of the low-level processing. I'm describing a simple typical architecture, real architectures can be more complex or differ in ways that don't matter at this level of detail.\n\nWhen an **interrupt** occurs, the processor looks if interrupts are masked. If they are, nothing happens until they are unmasked.\n\nRemember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).\n\nRemember: USE method: Utilization, Saturation, Errors for each resource. projects/knowledge/interview/linux/088-explain-interrupts-and-interrupt-handlers-in-linux.txt linux-performance/4ec1eb4e7397 linux-performance hard cyber-security, web-security, cors Is there a way to allow multiple cross-domains using the Access-Control-Allow-Origin header in Nginx? Yes. Use `if` blocks to match `$http_origin` against a regex of allowed domains, then set the header dynamically:\n\n```\nlocation / {\n if ($http_origin ~* (^https?://([^/]+\.)*(domain1|domain2)\.com$)) {\n add_header 'Access-Control-Allow-Origin' "$http_origin";\n add_header 'Access-Control-Allow-Credentials' 'true';\n add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';\n }\n}\n```\n\nKey point: you cannot list multiple origins in a single `Access-Control-Allow-Origin` header. Instead, dynamically echo back the matched origin.\n\nRemember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).\n\nRemember: USE method: Utilization, Saturation, Errors for each resource. projects/knowledge/interview/cyber-security/004-is-there-a-way-to-allow-multiple-cross-domains-usi.txt linux-performance/83e0b9ea9dce linux-performance hard cyber-security, linux, recovery How to recover deleted file held open e.g. by Apache? A deleted file that is still open retains its inode (hard link count = 0). Linux exposes open file descriptors via `/proc//fd/`. The symlink target shows the original path with `(deleted)` appended.\n\nTo recover: `cat /proc//fd/ > /path/to/recovered_file`\n\nTo find the fd: check `ls -l /proc//fd/` or use `lsof | grep deleted`. You can iterate all open fds for a process or scan all processes under `/proc/[1-9]*/fd/*`.\n\nRemember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).\n\nRemember: USE method: Utilization, Saturation, Errors for each resource. projects/knowledge/interview/cyber-security/006-how-to-recover-deleted-file-held-open-eg-by-apache.txt linux-performance/8d0491605992 linux-performance hard cyber-security, principles, access-control Write two golden rules for reducing the impact of hacked system. 1. **Principle of Least Privilege**: Run services with the minimum permissions needed. If Apache is compromised, the attacker is limited to what the `apache` user can access — not root.\n\n2. **Principle of Separation of Privileges**: Isolate components — e.g., give the web app a read-only database account. Use SELinux or AppArmor to enforce mandatory access controls. Whitelist allowed actions rather than blacklisting bad ones to reduce attack surface.\n\nRemember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).\n\nRemember: USE method: Utilization, Saturation, Errors for each resource. projects/knowledge/interview/cyber-security/003-write-two-golden-rules-for-reducing-the-impact-of-.txt linux-performance/ecc219e61152 linux-performance hard cyber-security, linux, process-control Explain `:(){ :|:& };:` and how stop this code if you are already logged into a system? It is a **fork bomb**. `:()` defines a function named `:`. The body `:|:&` calls itself, pipes output to another copy of itself, and backgrounds it. The final `:` executes it, causing exponential process creation.\n\nTo stop it if already logged in:\n- `killall -STOP -u ` to freeze all user processes\n- If the shell can't fork: `exec killall -STOP -u ` (replaces the shell process)\n\nPrevention: use PAM (`/etc/security/limits.conf`) to limit per-user process count (`nproc`).\n\nRemember: top: PID, PR, NI, VIRT, RES, %CPU, %MEM. htop = prettier with mouse. projects/knowledge/interview/cyber-security/005-explain-code-code-and-how-stop-this-code-if-you-ar.txt linux-performance/ee8860d38564 linux-performance hard cyber-security, linux, recovery The team of admins needs your support. You must remotely reinstall the system on one of the main servers. There is no access to the management console (e.g. iDRAC). How to install Linux on disk, from and where other Linux exist and running? Use `debootstrap` to install a minimal Linux into a working directory, chroot into it, then mount and wipe the old root filesystem, restore from backup, and reinstall GRUB.\n\nHigh-level steps:\n1. `debootstrap` a minimal system to `/mnt/system`\n2. Bind-mount `/proc`, `/sys`, `/dev` and chroot in\n3. Mount the old root (e.g., `/dev/sda1`), delete old files, extract backup tarball\n4. Chroot into restored system, run `grub-install` and `update-grub`\n5. Reboot with `sync; reboot -f` (normal shutdown commands won't work from chroot)\n\nRemember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).\n\nRemember: USE method: Utilization, Saturation, Errors for each resource. projects/knowledge/interview/cyber-security/007-the-team-of-admins-needs-your-support-you-must-rem.txt linux-performance/cs-owasp-top10 linux-performance medium cyber-security, owasp, web-security What is the OWASP Top 10 and why does it matter for DevOps? The OWASP Top 10 is a regularly updated list of the most critical web application security risks. Current top entries include: Broken Access Control, Cryptographic Failures, Injection, Insecure Design, Security Misconfiguration, Vulnerable Components, Authentication Failures, Software Integrity Failures, Logging Failures, and SSRF. DevOps teams use it to prioritize security testing in CI/CD pipelines and set security gates.\n\nRemember: top: PID, PR, NI, VIRT, RES, %CPU, %MEM. htop = prettier with mouse. training/interactive/knowledge/data/cards/cyber-security.tsv linux-performance/cs-incident-response linux-performance medium cyber-security, incident-response, process What are the phases of incident response? 1) Preparation: policies, tools, training, runbooks.\n2) Identification: detect and confirm the incident via monitoring, alerts, or reports.\n3) Containment: limit damage (short-term: isolate affected systems; long-term: apply temporary fixes).\n4) Eradication: remove the root cause (malware, compromised accounts, vulnerabilities).\n5) Recovery: restore systems to normal operation, verify integrity.\n6) Lessons Learned: post-incident review, update procedures, improve defenses.\n\nRemember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).\n\nRemember: USE method: Utilization, Saturation, Errors for each resource. training/interactive/knowledge/data/cards/cyber-security.tsv linux-performance/cs-threat-modeling linux-performance hard cyber-security, threat-modeling, design What is threat modeling and name a common framework for it. Threat modeling is the process of identifying potential threats to a system during design. It answers: What are we building? What can go wrong? What are we going to do about it?\nSTRIDE is a common framework: Spoofing (identity), Tampering (data), Repudiation (deniability), Information Disclosure (confidentiality), Denial of Service (availability), Elevation of Privilege (authorization). Each maps to a security property to protect.\n\nRemember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).\n\nRemember: USE method: Utilization, Saturation, Errors for each resource. training/interactive/knowledge/data/cards/cyber-security.tsv linux-performance/cs-pentest-vs-redteam linux-performance hard cyber-security, testing, offensive What is the difference between penetration testing and red teaming? Penetration testing: scoped, time-boxed assessment of specific systems or applications. Goal is to find as many vulnerabilities as possible. The target team usually knows it is happening.\nRed teaming: adversary simulation that tests the organization holistically (people, processes, technology). Goal is to test detection and response capabilities. Often covert, longer duration, uses social engineering and physical access. Red teams emulate real attackers; pen testers find bugs.\n\nRemember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).\n\nRemember: USE method: Utilization, Saturation, Errors for each resource. training/interactive/knowledge/data/cards/cyber-security.tsv linux-performance/cs-encryption-types linux-performance easy cyber-security, encryption, fundamentals Explain the difference between symmetric and asymmetric encryption. Symmetric: same key for encryption and decryption (e.g., AES). Fast, used for bulk data encryption. Challenge: secure key distribution.\nAsymmetric: uses a key pair (public key encrypts, private key decrypts, e.g., RSA, ECDSA). Slower, used for key exchange, digital signatures, and TLS handshakes. In practice, TLS uses asymmetric crypto to exchange a symmetric session key, then uses symmetric crypto for the data.\n\nRemember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).\n\nRemember: USE method: Utilization, Saturation, Errors for each resource. training/interactive/knowledge/data/cards/cyber-security.tsv linux-performance/cs-least-priv-iam linux-performance medium cyber-security, iam, least-privilege How do you implement least privilege in cloud IAM? 1) Start with zero permissions and add only what is needed; 2) Use managed policies scoped to specific services; 3) Avoid wildcard permissions (Resource: "*"); 4) Use conditions (IP range, MFA required, time-based); 5) Separate roles for different workloads; 6) Use IAM Access Analyzer to find unused permissions; 7) Regularly audit and remove stale permissions; 8) Prefer short-lived credentials (STS AssumeRole) over long-lived access keys.\n\nRemember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).\n\nRemember: USE method: Utilization, Saturation, Errors for each resource. training/interactive/knowledge/data/cards/cyber-security.tsv linux-performance/42ea9a2e2beb linux-performance medium linux, control-flow, permissions, scheduling "Running the command df you get ""command not found"". What could be wrong and how to fix it?" Most likely the default $PATH was modified or overridden, so `/bin/` (where df lives) is missing.\n\nFix:\n1. Manually reset PATH: `PATH=/bin:/sbin:/usr/bin:/usr/sbin`\n2. Check what broke it: review `~/.bashrc`, `~/.bash_profile`, `/etc/profile`\n3. Fix the offending file and source it: `source ~/.bashrc`\n\nTo schedule periodic tasks, use `cron`:\n`crontab -e` then add entries like: `*/30 * * * * bash myscript.sh`\nFormat: ` `. On systemd distros, consider systemd timers as an alternative.\n\nRemember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).\n\nRemember: USE method: Utilization, Saturation, Errors for each resource. projects/knowledge/interview/linux/145-running-the-command-df-you-get-command-not-found-w.txt linux-performance/96cf83d8397c linux-performance medium linux, bash, shell, environment You define x=2 in /etc/bashrc and x=6 in ~/.bashrc. You then log in. What is the value of x? x=6 (user's .bashrc overrides system bashrc)\n\nOrder of execution (login shell):\n1. /etc/profile\n2. ~/.bash_profile (or ~/.bash_login or ~/.profile)\n - Often sources ~/.bashrc\n3. /etc/bashrc (typically sourced by .bashrc)\n4. ~/.bashrc\n\nFor login shells:\n- System files first, user files after\n- Later definitions override earlier ones\n- x=2 set in /etc/bashrc\n- x=6 set in ~/.bashrc (wins)\n\nImportant notes:\n- Non-login shells may differ\n- Depends on how files source each other\n- .bashrc usually sources /etc/bashrc first\n\nResult: x=6 (user config takes precedence)\n\nRemember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).\n\nRemember: USE method: Utilization, Saturation, Errors for each resource. projects/knowledge/interview/linux/423-you-define-x2-in-etcbashrc-and-x6-bashrc-you-then-.txt linux-performance/a4c4e55d807c linux-performance medium linux, control-flow, piping Explain piping. How do you perform piping? Using a pipe in Linux, allows you to send the output of one command to the input of another command. For example: `cat /etc/services | wc -l`\n\nExample: `perf top` = live CPU hotspots. `perf record && perf report` for profiling. projects/knowledge/interview/linux/142-explain-piping-how-do-you-perform-piping.txt linux-performance/f93b3d002db3 linux-performance medium linux, system What does `LC_ALL=C` before command do? In what cases it will be useful? `LC_ALL` is the environment variable that overrides all the other localisation settings. This sets all `LC_` type variables at once to a specified locale.\n\nThe main reason to set `LC_ALL=C` before command is that fine to simply get English output (general change the locale used by the command).\n\nOn the other hand, also important is to increase the speed of command execution with `LC_ALL=C` e.g. `grep` or `fgrep`. Using the `LC_ALL=C` locale increased our performance and brought command execution time down.\n\nRemember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).\n\nRemember: USE method: Utilization, Saturation, Errors for each resource. projects/knowledge/interview/linux/047-what-does-codelcallccode-before-command-do-in-what.txt linux-performance/top-si-container linux-performance hard linux, containers, top, networking, performance What does high `si` (software interrupts) in `top` indicate on a Kubernetes node, and why is it invisible to kubectl top? High `si` means the kernel is spending significant time processing softirqs — typically network packet processing (NET_RX). On a container host, all pods share the host kernel's softirq handling. A pod receiving a flood of traffic drives up `si` on the host, degrading all pods. `kubectl top` only reports per-pod CPU usage and cannot see shared kernel overhead. Diagnose with `cat /proc/softirqs` and look for NET_RX growth.\n\nRemember: top: PID, PR, NI, VIRT, RES, %CPU, %MEM. htop = prettier with mouse. training/library/topics/linux-performance/street_ops.md linux-performance/top-st-cloud linux-performance medium linux, cloud, top, virtualization, performance A Kubernetes node on AWS shows 8% `st` (steal time) in `top` but all pods report normal CPU usage via `kubectl top`. What is happening and what can you do? Steal time means the underlying hypervisor is taking CPU cycles from the VM to serve other tenants. Pods report normal usage because cAdvisor measures CPU time consumed, not wall-clock time. The actual execution is slower because the VM is not getting all the CPU time it asks for. Solutions: resize to a dedicated/larger instance type, migrate the node, or use instances with dedicated tenancy. You cannot fix steal time from inside the VM.\n\nRemember: top: PID, PR, NI, VIRT, RES, %CPU, %MEM. htop = prettier with mouse. training/library/topics/linux-performance/street_ops.md linux-performance/top-wa-container-io linux-performance medium linux, containers, top, io, performance How does high `wa` (I/O wait) on a container host relate to container I/O throttling? `wa` on the host means CPUs are idle waiting for I/O to complete. For containers, this often maps to pods hitting their blkio cgroup limits or contending for shared node storage. A container writing heavily to an emptyDir on the node's disk drives up host `wa` and affects every container on that node. Check container I/O limits and consider using dedicated volumes or adjusting blkio cgroup settings.\n\nRemember: Toolkit: top/htop(overview), vmstat(memory), iostat(disk), sar(historical).\n\nRemember: USE method: Utilization, Saturation, Errors for each resource. training/library/topics/linux-performance/street_ops.md linux-performance/top-load-vs-cpu linux-performance medium linux, performance, load-average, diagnostics A server shows load average of 24 on a 4-core system but CPU utilization is only 15%. What does this indicate? High load average with low CPU utilization means most of the load is from processes in uninterruptible sleep (D state), not from CPU work. These processes are blocked on I/O — typically disk, NFS, or SAN. The load average counts both runnable and D-state processes. Confirm with `vmstat 1` (check the `b` column for blocked processes) and `iostat -xz 1` to identify the saturated device.\n\nRemember: Load avg = processes in run/IO-wait over 1/5/15 min. 4-core: load 4.0 = 100%.\n\nGotcha: Load includes I/O wait. High load + low CPU = disk bottleneck. training/library/topics/linux-performance/street_ops.md linux-performance/top-buffcache-trap linux-performance easy linux, memory, top, monitoring A junior engineer sees 128 MB free in `top` and panics that the server is out of memory. The server has 16 GB total. Is this a real problem? Almost certainly not. Linux aggressively uses free memory for page cache (buff/cache). The real question is what `avail Mem` shows — this includes reclaimable cache and buffers. If avail Mem is 11 GB, the server has plenty of memory. Only worry if avail Mem drops below ~10% of total AND swap is actively churning (check vmstat si/so columns).\n\nRemember: top: PID, PR, NI, VIRT, RES, %CPU, %MEM. htop = prettier with mouse. training/library/topics/linux-performance/street_ops.md linux-performance/top-dstate-processes linux-performance hard linux, performance, io, processes, troubleshooting Multiple processes show state D (uninterruptible sleep) in `top`. What does this mean and how do you investigate? D state means the process is waiting for the kernel to complete an I/O operation and cannot be interrupted — not even by kill -9. Multiple D-state processes signal an active I/O problem: disk failure, NFS hang, SAN timeout, or filesystem corruption.\n\nInvestigate with:\n1. `cat /proc//wchan` — kernel function it's blocked in\n2. `cat /proc//stack` — full kernel stack trace\n3. `iostat -xz 1` — identify saturated devices\n4. `dmesg -T` — check for hardware errors or NFS timeouts\n\nFix the underlying I/O problem; the processes will unblock on their own. training/library/topics/linux-performance/street_ops.md --- ## Wiki Navigation ### Related Content - [/proc Filesystem](../../../../library/topics/proc-filesystem/index.md) (Topic Pack, L2) — Linux Performance Tuning - [Linux Kernel Tuning](../../../../library/topics/linux-kernel-tuning/index.md) (Topic Pack, L2) — Linux Performance Tuning - [Linux Memory Management](../../../../library/topics/linux-memory-management/index.md) (Topic Pack, L1) — Linux Performance Tuning - [Linux Performance Tuning](../../../../library/topics/linux-performance/index.md) (Topic Pack, L2) — Linux Performance Tuning - [Runbook: High CPU (Runaway Process)](../../../../library/runbooks/linux/high-cpu.md) (Runbook, L1) — Linux Performance Tuning