Perf¶
20 cards — 🟢 4 easy | 🟡 7 medium | 🔴 3 hard
🟢 Easy (4)¶
1. What does perf top show?
Show answer
A live, continuously updating view of which functions are consuming the most sampled CPU time.Example: sudo perf top shows functions ranked by sample count in real time. Press arrow keys to navigate, Enter to annotate source.
Remember: "perf top = live htop for functions, not processes."
2. What does perf record do?
Show answer
Collects CPU profiling samples and writes them to a data file (perf.data) for later analysis with perf report.Example: sudo perf record -g -p 1234 sleep 30 profiles PID 1234 for 30 seconds with call graphs.
Remember: "Record now, Report later" — perf record writes perf.data, perf report reads it.
3. What does perf stat measure?
Show answer
Aggregate hardware and software performance counters — cycles, instructions, cache misses, context switches, page faults, and branch mispredictions for a command.Example: perf stat ls -R / shows counters for a recursive ls. Key counters: instructions per cycle (IPC) > 1.0 is good.
Remember: "Stat = Summary, Record = Samples" — stat gives totals, record gives per-function breakdowns.
4. What permissions does perf typically require?
Show answer
Root or appropriate perf_event settings. Most perf subcommands (top, record, trace) need sudo or kernel perf_event_paranoid tuning.Example: To lower the permission barrier: echo 1 > /proc/sys/kernel/perf_event_paranoid (allows non-root sampling). Set to -1 for full access.
Gotcha: In containers, you often need --privileged or SYS_ADMIN capability.
🟡 Medium (7)¶
1. What should you focus on in perf report output?
Show answer
Percentage of samples per function, the library/binary name, and the call graph. The highest-sample functions are where CPU time is being spent.Gotcha: A function with 30% of samples might be called from many places — always check the call graph to find the real caller.
Remember: "Samples show WHERE, call graphs show WHY."
2. How does a sampling profiler like perf work?
Show answer
It periodically interrupts execution and records the instruction pointer (and optionally the call stack). The statistical distribution of samples shows where time is spent, without recording every instruction.Fun fact: perf defaults to ~4000 samples/second. You can tune with -F flag, e.g., perf record -F 99 avoids lockstep with timer interrupts.
Under the hood: The kernel uses PMU (Performance Monitoring Unit) hardware counters to trigger interrupts at sample points.
3. Why are debug symbols important for perf output?
Show answer
Without symbols, perf shows raw memory addresses instead of function names. Debug symbols (or debug packages) map addresses to readable function and source locations.Example: Install debug symbols on Ubuntu: apt install linux-tools-$(uname -r) for perf itself, and
Gotcha: Stripped binaries show [unknown] in profiles — always keep debug packages on profiling hosts.
4. If a perf profile shows mostly kernel functions, what does that indicate?
Show answer
The workload is spending significant time crossing the kernel boundary — making many syscalls, doing heavy I/O, networking, or experiencing scheduler/memory pressure. It does not necessarily mean the kernel is broken.Example: If 80% of samples are in __GI___libc_read or do_sys_read, the app is I/O-bound, not CPU-bound.
Remember: "Kernel-heavy profile = look at I/O and syscalls, not algorithms."
5. What does perf trace do?
Show answer
Traces system calls and events for a process, similar to strace but often with lower overhead. Example: sudo perf trace -pExample: sudo perf trace -p $(pgrep nginx) shows every syscall nginx makes in real time. Useful for finding unexpected I/O.
perf trace vs strace: perf trace has lower overhead because it uses kernel tracepoints instead of ptrace.
6. Does sampling profiling record every instruction?
Show answer
No. It is statistical — it periodically samples where execution is. This keeps overhead low but means very short or rare functions may not appear in the profile.Under the hood: At 4000 Hz sampling, a 1-second function appears in ~4000 samples. A 1ms function appears in ~4 samples — possibly zero if unlucky.
Remember: "Sampling = statistics, not census. Rare events need longer runs."
7. What can high cache-miss counts in perf stat indicate?
Show answer
The workload has poor data locality — it is accessing memory in patterns that defeat CPU cache, causing frequent slow main-memory fetches instead of fast cache hits.Remember: "Cache miss = slow memory fetch." L1 cache hit ~1ns, L2 ~4ns, L3 ~10ns, main memory ~100ns — a 100x penalty.
Example: perf stat -e cache-misses,cache-references ./myapp shows the miss ratio.
🔴 Hard (3)¶
1. How do you capture call graphs with perf record?
Show answer
Use --call-graph dwarf (or --call-graph fp for frame pointers). Example: sudo perf record --call-graph dwarf -pGotcha: Frame-pointer unwinding (--call-graph fp) is faster but many compilers omit frame pointers with -O2. DWARF unwinding is more reliable but slower.
Remember: "DWARF = reliable stacks, FP = fast but fragile."
2. What are common pitfalls when using perf?
Show answer
Profiling too short a run (not enough samples), missing symbols, assuming the top function is the full story (ignoring call graphs), profiling the wrong process, and not distinguishing between CPU-busy and I/O-waiting problems.Remember: "SMIPC" — Short run, Missing symbols, Ignoring call graphs, Profiling wrong Process, Confusing CPU-busy with I/O-wait.
Gotcha: A process waiting on I/O shows low CPU in perf but high latency — use perf sched or off-CPU profiling for that.
3. Why can profiling in containers be tricky?
Show answer
Container namespace boundaries may hide process details, symbol resolution may fail if debug packages are in the host but not the container (or vice versa), and PID mapping across namespaces adds complexity.Gotcha: In Docker, perf inside the container may not see host symbols. Best practice: run perf on the host with the container's PID.
Example: perf record -p $(docker inspect --format '{{.State.Pid}}' mycontainer) -g sleep 30