Drill: Trace Syscalls with strace

Goal

Use strace to observe system calls made by a process to diagnose failures, permission issues, and missing files.

Setup

• Linux system with strace installed (apt install strace or yum install strace)
• Root or CAP_SYS_PTRACE capability to trace other users' processes

Commands

Trace a command from start to finish:

strace ls /nonexistent 2>&1 | tail -20

Attach to a running process:

strace -p <PID> -f -e trace=network

Filter by syscall category (file, network, process):

strace -e trace=file ls /tmp
strace -e trace=network curl -s https://example.com
strace -e trace=process bash -c 'echo hello'

Show timestamps and call duration:

strace -T -t ls /tmp 2>&1 | head -20

Summarize syscall statistics:

strace -c ls /tmp

Write trace output to a file:

strace -o /tmp/trace.out -f -e trace=open,openat,read,write mycommand

Follow forked children:

strace -f -p <PID>

What to Look For

• ENOENT (No such file or directory) reveals missing config files or libraries
• EACCES (Permission denied) pinpoints permission failures
• ECONNREFUSED shows network connection failures
• The -c summary reveals which syscalls dominate execution time

Common Mistakes

• Forgetting -f to follow child processes (misses forked workers)
• Tracing without filtering, producing overwhelming output
• Not redirecting strace output (it goes to stderr by default)
• Tracing production processes without understanding the performance overhead

Cleanup

rm -f /tmp/trace.out

Detach from traced processes with Ctrl+C. strace does not modify the traced process.