--- tags: - linux - l1 - flashcard-deck - perf --- [Portal](../../../../library/portal/index.md) | **Level:** [L1: Foundations](../../../../library/portal/levels.md) | **Topics:** [perf Profiling](../../../../library/portal/topics.md) | **Domain:** Linux id category difficulty tags question answer source_path perf/a1c2d3e4f5b6 perf easy perf,profiling,basics What does perf top show? A live, continuously updating view of which functions are consuming the most sampled CPU time.\n\nExample: sudo perf top shows functions ranked by sample count in real time. Press arrow keys to navigate, Enter to annotate source.\n\nRemember: "perf top = live htop for functions, not processes." zines/profiling.and.tracing.with.perf.zine.cleaned.notes perf/b2d3e4f5a6c7 perf easy perf,record,profiling What does perf record do? Collects CPU profiling samples and writes them to a data file (perf.data) for later analysis with perf report.\n\nExample: sudo perf record -g -p 1234 sleep 30 profiles PID 1234 for 30 seconds with call graphs.\n\nRemember: "Record now, Report later" — perf record writes perf.data, perf report reads it. zines/profiling.and.tracing.with.perf.zine.cleaned.notes perf/c3e4f5a6b7d8 perf easy perf,stat,counters What does perf stat measure? Aggregate hardware and software performance counters — cycles, instructions, cache misses, context switches, page faults, and branch mispredictions for a command.\n\nExample: perf stat ls -R / shows counters for a recursive ls. Key counters: instructions per cycle (IPC) > 1.0 is good.\n\nRemember: "Stat = Summary, Record = Samples" — stat gives totals, record gives per-function breakdowns. zines/profiling.and.tracing.with.perf.zine.cleaned.notes perf/d4f5a6b7c8e9 perf medium perf,report,analysis What should you focus on in perf report output? Percentage of samples per function, the library/binary name, and the call graph. The highest-sample functions are where CPU time is being spent.\n\nGotcha: A function with 30% of samples might be called from many places — always check the call graph to find the real caller.\n\nRemember: "Samples show WHERE, call graphs show WHY." zines/profiling.and.tracing.with.perf.zine.cleaned.notes perf/e5a6b7c8d9f0 perf medium perf,sampling,methodology How does a sampling profiler like perf work? It periodically interrupts execution and records the instruction pointer (and optionally the call stack). The statistical distribution of samples shows where time is spent, without recording every instruction.\n\nFun fact: perf defaults to ~4000 samples/second. You can tune with -F flag, e.g., perf record -F 99 avoids lockstep with timer interrupts.\n\nUnder the hood: The kernel uses PMU (Performance Monitoring Unit) hardware counters to trigger interrupts at sample points. zines/profiling.and.tracing.with.perf.zine.cleaned.notes perf/f6b7c8d9e0a1 perf medium perf,symbols,debugging Why are debug symbols important for perf output? Without symbols, perf shows raw memory addresses instead of function names. Debug symbols (or debug packages) map addresses to readable function and source locations.\n\nExample: Install debug symbols on Ubuntu: apt install linux-tools-$(uname -r) for perf itself, and -dbgsym for app symbols.\n\nGotcha: Stripped binaries show [unknown] in profiles — always keep debug packages on profiling hosts. zines/profiling.and.tracing.with.perf.zine.cleaned.notes perf/a7c8d9e0f1b2 perf medium perf,kernel,userspace If a perf profile shows mostly kernel functions, what does that indicate? The workload is spending significant time crossing the kernel boundary — making many syscalls, doing heavy I/O, networking, or experiencing scheduler/memory pressure. It does not necessarily mean the kernel is broken.\n\nExample: If 80% of samples are in __GI___libc_read or do_sys_read, the app is I/O-bound, not CPU-bound.\n\nRemember: "Kernel-heavy profile = look at I/O and syscalls, not algorithms." zines/profiling.and.tracing.with.perf.zine.cleaned.notes perf/b8d9e0f1a2c3 perf hard perf,call-graph,record How do you capture call graphs with perf record? Use --call-graph dwarf (or --call-graph fp for frame pointers). Example: sudo perf record --call-graph dwarf -p . This records stack traces so perf report can show the full call chain.\n\nGotcha: Frame-pointer unwinding (--call-graph fp) is faster but many compilers omit frame pointers with -O2. DWARF unwinding is more reliable but slower.\n\nRemember: "DWARF = reliable stacks, FP = fast but fragile." zines/profiling.and.tracing.with.perf.zine.cleaned.notes perf/c9e0f1a2b3d4 perf medium perf,trace,syscalls What does perf trace do? Traces system calls and events for a process, similar to strace but often with lower overhead. Example: sudo perf trace -p \n\nExample: sudo perf trace -p $(pgrep nginx) shows every syscall nginx makes in real time. Useful for finding unexpected I/O.\n\nperf trace vs strace: perf trace has lower overhead because it uses kernel tracepoints instead of ptrace. zines/profiling.and.tracing.with.perf.zine.cleaned.notes perf/d0f1a2b3c4e5 perf hard perf,pitfalls,methodology What are common pitfalls when using perf? Profiling too short a run (not enough samples), missing symbols, assuming the top function is the full story (ignoring call graphs), profiling the wrong process, and not distinguishing between CPU-busy and I/O-waiting problems.\n\nRemember: "SMIPC" — Short run, Missing symbols, Ignoring call graphs, Profiling wrong Process, Confusing CPU-busy with I/O-wait.\n\nGotcha: A process waiting on I/O shows low CPU in perf but high latency — use perf sched or off-CPU profiling for that. zines/profiling.and.tracing.with.perf.zine.cleaned.notes perf/e1a2b3c4d5f6 perf easy perf,permissions,requirements What permissions does perf typically require? Root or appropriate perf_event settings. Most perf subcommands (top, record, trace) need sudo or kernel perf_event_paranoid tuning.\n\nExample: To lower the permission barrier: echo 1 > /proc/sys/kernel/perf_event_paranoid (allows non-root sampling). Set to -1 for full access.\n\nGotcha: In containers, you often need --privileged or SYS_ADMIN capability. zines/profiling.and.tracing.with.perf.zine.cleaned.notes perf/f2b3c4d5e6a7 perf medium perf,overhead,sampling Does sampling profiling record every instruction? No. It is statistical — it periodically samples where execution is. This keeps overhead low but means very short or rare functions may not appear in the profile.\n\nUnder the hood: At 4000 Hz sampling, a 1-second function appears in ~4000 samples. A 1ms function appears in ~4 samples — possibly zero if unlucky.\n\nRemember: "Sampling = statistics, not census. Rare events need longer runs." zines/profiling.and.tracing.with.perf.zine.cleaned.notes perf/a3c4d5e6f7b8 perf hard perf,containers,namespaces Why can profiling in containers be tricky? Container namespace boundaries may hide process details, symbol resolution may fail if debug packages are in the host but not the container (or vice versa), and PID mapping across namespaces adds complexity.\n\nGotcha: In Docker, perf inside the container may not see host symbols. Best practice: run perf on the host with the container's PID.\n\nExample: perf record -p $(docker inspect --format '{{.State.Pid}}' mycontainer) -g sleep 30 zines/profiling.and.tracing.with.perf.zine.cleaned.notes perf/b4d5e6f7a8c9 perf medium perf,cache-misses,counters What can high cache-miss counts in perf stat indicate? The workload has poor data locality — it is accessing memory in patterns that defeat CPU cache, causing frequent slow main-memory fetches instead of fast cache hits.\n\nRemember: "Cache miss = slow memory fetch." L1 cache hit ~1ns, L2 ~4ns, L3 ~10ns, main memory ~100ns — a 100x penalty.\n\nExample: perf stat -e cache-misses,cache-references ./myapp shows the miss ratio. zines/profiling.and.tracing.with.perf.zine.cleaned.notes --- ## Wiki Navigation ### Related Content - [perf Profiling](../../../../library/topics/perf-profiling/index.md) (Topic Pack, L2) — perf Profiling