--- tags: - linux - l1 - flashcard-deck - ebpf --- [Portal](../../../../library/portal/index.md) | **Level:** [L1: Foundations](../../../../library/portal/levels.md) | **Topics:** [eBPF](../../../../library/portal/topics.md) | **Domain:** Linux id category difficulty tags question answer source_path ebpf/a1b2c3d4e5f5 ebpf easy ebpf, overview What is eBPF and why is it a breakthrough for Linux observability? eBPF lets you run sandboxed programs inside the Linux kernel without writing kernel modules, without rebooting, and without risking a kernel panic. It provides function-level tracing, network packet inspection, and performance profiling in production with negligible overhead (typically <1%).\n\nName origin: eBPF = extended Berkeley Packet Filter. Original BPF (1993) was a packet filter for tcpdump. eBPF (2014, Linux 3.18+) extended it to general-purpose kernel programmability.\n\nWho made it: Alexei Starovoitov and Daniel Borkmann drove eBPF's evolution in the Linux kernel. Brendan Gregg popularized its use for observability. training/library/topics/ebpf-observability/primer.md ebpf/b2c3d4e5f5a6 ebpf easy ebpf, bcc, execsnoop What does the execsnoop BCC tool show, and when would you use it? execsnoop shows every new process execution system-wide (command name, PID, PPID, return code, and arguments). Use it when you need to answer "what processes are being spawned?" — for example, diagnosing unexpected process creation or identifying what a cron job actually runs.\n\nExample: execsnoop-bpfcc reveals that a cron job spawns 50 shell processes per minute, each forking grep and awk — explaining the CPU spikes.\n\nDebug clue: if you see unfamiliar processes in execsnoop, it could indicate a compromised host. Cross-reference with audit logs. training/library/topics/ebpf-observability/primer.md ebpf/c3d4e5f5a6b7 ebpf easy ebpf, safety What are three safety guarantees the eBPF verifier provides before a program can run in the kernel? 1) Memory safety: all memory accesses are checked. \n2) Guaranteed termination: programs must have bounded loops and provably terminate. \n3) Restricted API: programs can only call approved kernel helper functions. The verifier rejects any program that violates these guarantees before it is loaded. training/library/topics/ebpf-observability/primer.md ebpf/d4e5f5a6b7c8 ebpf medium ebpf, bcc, tcplife What does the tcplife BCC tool show, and how can it help detect connection leaks? tcplife shows TCP session lifecycles: PID, command, local/remote addresses, ports, bytes transmitted/received, and session duration in milliseconds. Connection leaks show as thousands of long-lived connections with zero bytes transferred. This pattern indicates connections being opened but never closed or returned to a pool. training/library/topics/ebpf-observability/primer.md ebpf/e5f5a6b7c8d9 ebpf medium ebpf, bcc, biolatency How do you use biolatency to determine if disk I/O is causing performance problems? Run biolatency-bpfcc to get a histogram of block I/O latency. If operations frequently exceed 10ms, your storage is struggling. Compare against a baseline captured during healthy operation. The histogram shows the distribution, so you can see if latency is uniformly slow or has outlier spikes. training/library/topics/ebpf-observability/primer.md ebpf/f5a6b7c8d9e0 ebpf medium ebpf, bpftrace What is bpftrace and how does it differ from BCC tools? bpftrace is a high-level tracing language for eBPF (like awk for kernel tracing). BCC tools are pre-built, single-purpose tools. bpftrace lets you write custom one-liners for ad-hoc tracing, such as counting syscalls by process, tracing specific file opens, or histogramming read sizes. It is more flexible but requires writing the tracing logic yourself. training/library/topics/ebpf-observability/primer.md ebpf/06a7b8c9d0e1 ebpf medium ebpf, bcc, network What three BCC tools would you use to diagnose network issues, and what does each show? tcpconnect shows every outbound TCP connection (what is this process connecting to?). tcpretrans shows TCP retransmissions (indicates network congestion or packet loss). tcplife shows full TCP session lifecycle with duration and bytes (how long do connections live and how much data flows?). training/library/topics/ebpf-observability/primer.md ebpf/17b8c9d0e1f2 ebpf hard ebpf, runqlat, scheduling How would you use eBPF tools to diagnose mystery latency spikes when CPU, memory, and disk metrics all look normal? Run runqlat-bpfcc to check scheduler run queue latency. Spikes in run queue time (e.g., 50-200ms) indicate CPU contention even when average CPU looks fine. Then use bpftrace with profile:hz:99 to sample which process is consuming CPU during the spike. This can reveal periodic CPU hogs (e.g., logrotate compressing large files) that cause scheduling delays. training/library/topics/ebpf-observability/primer.md ebpf/28c9d0e1f2a3 ebpf hard ebpf, bpftrace, tracing Write a bpftrace one-liner to count system calls by process name and explain when you would use it. bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }' — This attaches to the raw syscall entry tracepoint and aggregates a count per process name. Use it to identify which processes are making the most syscalls, which helps find noisy processes contributing to system overhead. Always filter by process name on busy servers to avoid excessive output. training/library/topics/ebpf-observability/primer.md ebpf/39d0e1f2a3b4 ebpf hard ebpf, kernel, requirements What kernel version requirements exist for eBPF, and what is BPF CO-RE? Full eBPF support requires kernel 4.9+. Many advanced features need 5.x+. BPF CO-RE (Compile Once, Run Everywhere) requires kernel 5.8+ and allows eBPF programs compiled on one kernel version to run on another without recompilation. Without CO-RE, BCC tools recompile at load time, requiring kernel headers to be installed on the target host. training/library/topics/ebpf-observability/primer.md ebpf/4ae1f2a3b4c5 ebpf easy ebpf, program-types, overview What are the main eBPF program types and when is each used? Tracing (kprobe, tracepoint, perf_event): attach to kernel functions or tracepoints for observability.\nNetworking (XDP, TC, socket): process packets at various points in the network stack.\nSecurity (LSM): enforce security policies at kernel hook points.\nCgroup: control resource usage per cgroup (container-level).\nEach program type has specific attach points and allowed helper functions. The verifier enforces type-specific constraints. ebpf/5bf2a3b4c5d6 ebpf medium ebpf, bpftrace, one-liners What are useful bpftrace one-liners for production debugging? Count syscalls by process: bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'\nTrace file opens: bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'\nHistogram of read sizes: bpftrace -e 'tracepoint:syscalls:sys_exit_read /args->ret > 0/ { @size = hist(args->ret); }'\nAlways filter by comm or pid on busy systems to avoid overwhelming output. ebpf/6ca3b4c5d6e7 ebpf hard ebpf, xdp, packet-processing What is XDP and why is it significant for packet processing? XDP (eXpress Data Path) runs eBPF programs at the earliest point in the network stack — before the kernel allocates an sk_buff. This makes it extremely fast (millions of packets per second). Actions: XDP_PASS (normal processing), XDP_DROP (discard), XDP_TX (bounce back), XDP_REDIRECT (send to another interface). Use cases: DDoS mitigation, load balancing (e.g., Facebook's Katran), and packet filtering at line rate without kernel overhead. ebpf/7db4c5d6e7f8 ebpf medium ebpf, cilium, kubernetes How does Cilium use eBPF for Kubernetes networking and security? Cilium replaces kube-proxy's iptables rules with eBPF programs for service routing, achieving better performance at scale (no iptables chain traversal). It provides: L3/L4/L7 network policies (filter by HTTP path, gRPC method), transparent encryption (WireGuard or IPsec), load balancing without SNAT, and deep observability via Hubble. Cilium attaches eBPF programs to TC hooks on each pod's veth interface. ebpf/8ec5d6e7f8a9 ebpf easy ebpf, bcc-tools, catalog What are the most commonly used BCC tools and what does each diagnose? execsnoop: new process execution.\nopensnoop: file opens.\nbiolatency: block I/O latency histogram.\next4slower/xfsslower: slow filesystem operations.\ntcpconnect: outbound TCP connections.\ntcpretrans: TCP retransmissions (network issues).\nrunqlat: CPU scheduler queue latency.\ncachestat: page cache hit/miss ratio.\nfiletop: top files by I/O.\nThese cover the most common production performance questions without custom code. ebpf/9fd6e7f8a9b0 ebpf hard ebpf, overhead, performance What is the actual performance overhead of eBPF tracing in production? eBPF overhead depends on the attach point and frequency. Tracepoints on hot paths (every syscall) can add 1-5% overhead under high load. Kprobes on rarely-hit functions add negligible overhead (<0.1%). XDP programs add microseconds per packet. BCC tools that recompile on load have a startup cost of several seconds. \nBest practice: use targeted tracing (filter by PID or comm), avoid tracing high-frequency events without filters, and prefer tracepoints over kprobes for stability. ebpf/0ae7f8a9b0c1 ebpf hard ebpf, security, observability How can eBPF be used for security observability? eBPF can monitor security-relevant events in real time: file access patterns (opensnoop), process execution chains (execsnoop), network connections (tcpconnect), privilege escalation attempts (tracing setuid/capabilities), and container escapes (monitoring namespace changes). Tools like Falco and Tetragon use eBPF to detect anomalous behavior without the overhead of kernel modules. eBPF LSM programs can enforce security policies at the kernel level, blocking unauthorized actions before they complete. ebpf/be5fc9dd8c39 ebpf hard ebpf, btf, co-re, portability What problem does BTF (BPF Type Format) and CO-RE solve for eBPF programs? eBPF programs access kernel structs whose layout changes between kernel versions. Without CO-RE, you must compile on the target kernel. BTF embeds type information in the kernel; CO-RE (Compile Once — Run Everywhere) uses BTF to relocate struct field offsets at load time, making one binary portable across kernel versions. training/library/topics/ebpf-observability/primer.md ebpf/58d95219e2a9 ebpf medium ebpf, maps, data-structures What are eBPF maps and what are the most common map types? Maps are key-value stores shared between eBPF programs and userspace. Common types: BPF_MAP_TYPE_HASH (general lookup), BPF_MAP_TYPE_ARRAY (fixed-size indexed), BPF_MAP_TYPE_RINGBUF (efficient event streaming to userspace), BPF_MAP_TYPE_LPM_TRIE (longest prefix match for IP routing). Maps persist across program invocations. training/library/topics/ebpf-observability/primer.md ebpf/1d08680697da ebpf medium ebpf, bpftool, debugging, verifier How do you debug an eBPF program that fails to load? The verifier rejects unsafe programs with an error log showing the instruction that failed. Use bpftool prog show to list loaded programs, bpftool prog dump xlated to see translated bytecode, and bpftool map dump to inspect map contents. Common failures: unbounded loops, invalid memory access, exceeding instruction limit. training/library/topics/ebpf-observability/primer.md --- ## Wiki Navigation ### Related Content - [Cilium & eBPF Networking](../../../../library/topics/cilium/index.md) (Topic Pack, L2) — eBPF - [Deep Dive: eBPF Explained](../../../../library/deep-dives/ebpf.explained.md) (deep_dive, L3) — eBPF - [Linux Performance Tuning](../../../../library/topics/linux-performance/index.md) (Topic Pack, L2) — eBPF - [eBPF & Modern Linux Observability](../../../../library/topics/ebpf-observability/index.md) (Topic Pack, L3) — eBPF