Ebpf¶
29 cards — 🟢 5 easy | 🟡 8 medium | 🔴 7 hard
🟢 Easy (5)¶
1. What is eBPF and why is it a breakthrough for Linux observability?
Show answer
eBPF lets you run sandboxed programs inside the Linux kernel without writing kernel modules, without rebooting, and without risking a kernel panic. It provides function-level tracing, network packet inspection, and performance profiling in production with negligible overhead (typically <1%).Name origin: eBPF = extended Berkeley Packet Filter. Original BPF (1993) was a packet filter for tcpdump. eBPF (2014, Linux 3.18+) extended it to general-purpose kernel programmability.
Who made it: Alexei Starovoitov and Daniel Borkmann drove eBPF's evolution in the Linux kernel. Brendan Gregg popularized its use for observability.
2. What does the execsnoop BCC tool show, and when would you use it?
Show answer
execsnoop shows every new process execution system-wide (command name, PID, PPID, return code, and arguments). Use it when you need to answer "what processes are being spawned?" — for example, diagnosing unexpected process creation or identifying what a cron job actually runs.Example: execsnoop-bpfcc reveals that a cron job spawns 50 shell processes per minute, each forking grep and awk — explaining the CPU spikes.
Debug clue: if you see unfamiliar processes in execsnoop, it could indicate a compromised host. Cross-reference with audit logs.
3. What are three safety guarantees the eBPF verifier provides before a program can run in the kernel?
Show answer
1) Memory safety: all memory accesses are checked.2) Guaranteed termination: programs must have bounded loops and provably terminate.
3) Restricted API: programs can only call approved kernel helper functions. The verifier rejects any program that violates these guarantees before it is loaded.
4. What are the main eBPF program types and when is each used?
Show answer
Tracing (kprobe, tracepoint, perf_event): attach to kernel functions or tracepoints for observability.Networking (XDP, TC, socket): process packets at various points in the network stack.
Security (LSM): enforce security policies at kernel hook points.
Cgroup: control resource usage per cgroup (container-level).
Each program type has specific attach points and allowed helper functions. The verifier enforces type-specific constraints.
5. What are the most commonly used BCC tools and what does each diagnose?
Show answer
execsnoop: new process execution.opensnoop: file opens.
biolatency: block I/O latency histogram.
ext4slower/xfsslower: slow filesystem operations.
tcpconnect: outbound TCP connections.
tcpretrans: TCP retransmissions (network issues).
runqlat: CPU scheduler queue latency.
cachestat: page cache hit/miss ratio.
filetop: top files by I/O.
These cover the most common production performance questions without custom code.
🟡 Medium (8)¶
1. What does the tcplife BCC tool show, and how can it help detect connection leaks?
Show answer
tcplife shows TCP session lifecycles: PID, command, local/remote addresses, ports, bytes transmitted/received, and session duration in milliseconds. Connection leaks show as thousands of long-lived connections with zero bytes transferred. This pattern indicates connections being opened but never closed or returned to a pool.2. How do you use biolatency to determine if disk I/O is causing performance problems?
Show answer
Run biolatency-bpfcc to get a histogram of block I/O latency. If operations frequently exceed 10ms, your storage is struggling. Compare against a baseline captured during healthy operation. The histogram shows the distribution, so you can see if latency is uniformly slow or has outlier spikes.3. What is bpftrace and how does it differ from BCC tools?
Show answer
bpftrace is a high-level tracing language for eBPF (like awk for kernel tracing). BCC tools are pre-built, single-purpose tools. bpftrace lets you write custom one-liners for ad-hoc tracing, such as counting syscalls by process, tracing specific file opens, or histogramming read sizes. It is more flexible but requires writing the tracing logic yourself.4. What three BCC tools would you use to diagnose network issues, and what does each show?
Show answer
tcpconnect shows every outbound TCP connection (what is this process connecting to?). tcpretrans shows TCP retransmissions (indicates network congestion or packet loss). tcplife shows full TCP session lifecycle with duration and bytes (how long do connections live and how much data flows?).5. What are useful bpftrace one-liners for production debugging?
Show answer
Count syscalls by process: bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'Trace file opens: bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s
", comm, str(args->filename)); }'
Histogram of read sizes: bpftrace -e 'tracepoint:syscalls:sys_exit_read /args->ret > 0/ { @size = hist(args->ret); }'
Always filter by comm or pid on busy systems to avoid overwhelming output.
6. How does Cilium use eBPF for Kubernetes networking and security?
Show answer
Cilium replaces kube-proxy's iptables rules with eBPF programs for service routing, achieving better performance at scale (no iptables chain traversal). It provides: L3/L4/L7 network policies (filter by HTTP path, gRPC method), transparent encryption (WireGuard or IPsec), load balancing without SNAT, and deep observability via Hubble. Cilium attaches eBPF programs to TC hooks on each pod's veth interface.7. What are eBPF maps and what are the most common map types?
Show answer
Maps are key-value stores shared between eBPF programs and userspace. Common types: BPF_MAP_TYPE_HASH (general lookup), BPF_MAP_TYPE_ARRAY (fixed-size indexed), BPF_MAP_TYPE_RINGBUF (efficient event streaming to userspace), BPF_MAP_TYPE_LPM_TRIE (longest prefix match for IP routing). Maps persist across program invocations.8. How do you debug an eBPF program that fails to load?
Show answer
The verifier rejects unsafe programs with an error log showing the instruction that failed. Use bpftool prog show to list loaded programs, bpftool prog dump xlated to see translated bytecode, and bpftool map dump to inspect map contents. Common failures: unbounded loops, invalid memory access, exceeding instruction limit.🔴 Hard (7)¶
1. How would you use eBPF tools to diagnose mystery latency spikes when CPU, memory, and disk metrics all look normal?
Show answer
Run runqlat-bpfcc to check scheduler run queue latency. Spikes in run queue time (e.g., 50-200ms) indicate CPU contention even when average CPU looks fine. Then use bpftrace with profile:hz:99 to sample which process is consuming CPU during the spike. This can reveal periodic CPU hogs (e.g., logrotate compressing large files) that cause scheduling delays.2. Write a bpftrace one-liner to count system calls by process name and explain when you would use it.
Show answer
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }' — This attaches to the raw syscall entry tracepoint and aggregates a count per process name. Use it to identify which processes are making the most syscalls, which helps find noisy processes contributing to system overhead. Always filter by process name on busy servers to avoid excessive output.3. What kernel version requirements exist for eBPF, and what is BPF CO-RE?
Show answer
Full eBPF support requires kernel 4.9+. Many advanced features need 5.x+. BPF CO-RE (Compile Once, Run Everywhere) requires kernel 5.8+ and allows eBPF programs compiled on one kernel version to run on another without recompilation. Without CO-RE, BCC tools recompile at load time, requiring kernel headers to be installed on the target host.4. What is XDP and why is it significant for packet processing?
Show answer
XDP (eXpress Data Path) runs eBPF programs at the earliest point in the network stack — before the kernel allocates an sk_buff. This makes it extremely fast (millions of packets per second). Actions: XDP_PASS (normal processing), XDP_DROP (discard), XDP_TX (bounce back), XDP_REDIRECT (send to another interface). Use cases: DDoS mitigation, load balancing (e.g., Facebook's Katran), and packet filtering at line rate without kernel overhead.5. What is the actual performance overhead of eBPF tracing in production?
Show answer
eBPF overhead depends on the attach point and frequency. Tracepoints on hot paths (every syscall) can add 1-5% overhead under high load. Kprobes on rarely-hit functions add negligible overhead (<0.1%). XDP programs add microseconds per packet. BCC tools that recompile on load have a startup cost of several seconds.Best practice: use targeted tracing (filter by PID or comm), avoid tracing high-frequency events without filters, and prefer tracepoints over kprobes for stability.
6. How can eBPF be used for security observability?
Show answer
eBPF can monitor security-relevant events in real time: file access patterns (opensnoop), process execution chains (execsnoop), network connections (tcpconnect), privilege escalation attempts (tracing setuid/capabilities), and container escapes (monitoring namespace changes). Tools like Falco and Tetragon use eBPF to detect anomalous behavior without the overhead of kernel modules. eBPF LSM programs can enforce security policies at the kernel level, blocking unauthorized actions before they complete.7. What problem does BTF (BPF Type Format) and CO-RE solve for eBPF programs?