The /proc Filesystem: Linux's Hidden API

lesson
/proc
kernel-state
process-inspection
memory-maps
network-state
debugging ---# The /proc Filesystem: Linux's Hidden API

Topics: /proc, kernel state, process inspection, memory maps, network state, debugging Level: L1–L2 (Foundations → Operations) Time: 45–60 minutes Prerequisites: Basic Linux command line

The Mission¶

Every debugging tool you use — ps, top, free, netstat, lsof — reads from the same place: /proc. It's a virtual filesystem that exposes the kernel's internal state as files. You can read process memory maps, see open file descriptors, check network connections, and inspect CPU info — all with cat and ls.

Understanding /proc means understanding what your debugging tools actually see — and being able to find information they don't expose.

What /proc Is (And Isn't)¶

/proc is not on disk. It's a virtual filesystem generated by the kernel on every read. When you cat /proc/meminfo, the kernel computes the current memory state and returns it as text. The "file" doesn't exist anywhere — it's generated on demand.

# This file is 0 bytes on disk but returns content when read
ls -la /proc/meminfo
# → -r--r--r-- 1 root root 0 Mar 23 14:00 /proc/meminfo

cat /proc/meminfo | head -5
# → MemTotal:       16384000 kB
# → MemFree:          512000 kB
# → MemAvailable:   11264000 kB
# → Buffers:          256000 kB
# → Cached:         10240000 kB

Name Origin: /proc stands for "process." It was originally just process information (one directory per PID). Over time, Linux stuffed more and more kernel state into it: memory, networking, hardware, filesystems. The result is a messy but incredibly useful API to the kernel. Plan 9 (Bell Labs, 1992) took this further — everything is a file, including the CPU, network stack, and window system.

Per-Process: /proc/PID/¶

Every running process has a directory /proc/<PID>/ with everything the kernel knows about it:

# Pick a process (your shell)
ls /proc/$$/
# → cmdline  cwd  environ  exe  fd  maps  mem  mountinfo  net
#   oom_score  root  smaps  stack  stat  status  wchan  ...

The essential files¶

PID=$$  # Your current shell

# What command is this?
cat /proc/$PID/cmdline | tr '\0' ' '
# → /bin/bash

# What binary is running?
ls -la /proc/$PID/exe
# → lrwx------ 1 user user 0 ... /proc/1234/exe -> /usr/bin/bash

# Current working directory
ls -la /proc/$PID/cwd
# → lrwx------ 1 user user 0 ... /proc/1234/cwd -> /home/user

# Environment variables (ALL of them)
cat /proc/$PID/environ | tr '\0' '\n' | head
# → HOME=/home/user
# → PATH=/usr/local/bin:/usr/bin
# → SHELL=/bin/bash
# → DATABASE_URL=postgresql://...   ← secrets are here too!

# Process status (memory, state, threads)
cat /proc/$PID/status | head -15
# → Name:   bash
# → State:  S (sleeping)
# → Pid:    1234
# → PPid:   1233
# → Uid:    1000    1000    1000    1000
# → VmRSS:  10240 kB    ← actual memory usage
# → Threads: 1

Gotcha: /proc/PID/environ shows ALL environment variables — including secrets. If your app stores DATABASE_URL or API_KEY in environment variables, anyone who can read /proc/PID/environ (same user or root) can see them. This is why environment variables are not truly secure for secrets.

Open File Descriptors: /proc/PID/fd/¶

Every open file, socket, pipe, and device is a file descriptor:

# List all open file descriptors for a process
ls -la /proc/$PID/fd/
# → lrwx------ 1 user user 64 ... 0 -> /dev/pts/0        (stdin)
# → lrwx------ 1 user user 64 ... 1 -> /dev/pts/0        (stdout)
# → lrwx------ 1 user user 64 ... 2 -> /dev/pts/0        (stderr)
# → lr-x------ 1 user user 64 ... 3 -> /var/log/app.log  (log file)
# → lrwx------ 1 user user 64 ... 4 -> socket:[89012]    (network connection)
# → l-wx------ 1 user user 64 ... 5 -> pipe:[34567]      (pipe to another process)

# Count open files
ls /proc/$PID/fd/ | wc -l
# → 42

# This is what lsof reads!

Practical use: When you rm a large file but df doesn't show freed space, the file is still open. Find it: ls -la /proc/*/fd/ 2>/dev/null | grep deleted. The (deleted) suffix shows files removed from the directory but still held open.

Memory Maps: /proc/PID/maps and smaps¶

# See all memory regions of a process
cat /proc/$PID/maps | head -10
# → 55d8a0000000-55d8a0001000 r--p 00000000 08:01 12345  /usr/bin/bash
# → 55d8a0001000-55d8a0100000 r-xp 00001000 08:01 12345  /usr/bin/bash
# → 7f4e90000000-7f4e90021000 rw-p 00000000 00:00 0      [heap]
# → 7fff80000000-7fff80021000 rw-p 00000000 00:00 0      [stack]

# Detailed memory usage per region
cat /proc/$PID/smaps_rollup
# → Rss:           10240 kB    ← physical memory used
# → Pss:            8192 kB    ← proportional share (for shared libs)
# → Private_Dirty:  6144 kB    ← memory unique to this process
# → Swap:              0 kB    ← swapped out memory

Trivia: pmap, smem, and other memory tools all read /proc/PID/smaps. The difference between RSS (Resident Set Size) and PSS (Proportional Set Size) matters: RSS counts shared libraries fully for each process that uses them, making the total look inflated. PSS divides shared memory proportionally — if 10 processes share a 10MB library, each gets 1MB in PSS. Use PSS for accurate per-process memory accounting.

System-Wide: /proc Files¶

# CPU info (model, cores, speed)
cat /proc/cpuinfo | grep "model name" | head -1
# → model name : Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz

cat /proc/cpuinfo | grep processor | wc -l
# → 8 (number of logical CPUs)

# Memory info (what 'free' reads)
cat /proc/meminfo | head -5

# Load average (what 'uptime' reads)
cat /proc/loadavg
# → 0.15 0.10 0.05 1/234 5678
#   1m   5m   15m  running/total  last-pid

# Kernel command line (boot parameters)
cat /proc/cmdline
# → BOOT_IMAGE=/vmlinuz root=UUID=abc123 ro quiet

# Network connections (what 'ss' reads)
cat /proc/net/tcp | head -5
# → sl  local_address rem_address   st tx_queue rx_queue
# → 0: 0100007F:1F90 00000000:0000 0A 00000000:00000000
#   ↑ localhost:8080  ↑ listening    ↑ state 0A = LISTEN

# Filesystem mounts
cat /proc/mounts | head -5

# Kernel version
cat /proc/version

The Debugging Patterns¶

Pattern 1: "What is this process doing?"¶

PID=5678
cat /proc/$PID/cmdline | tr '\0' ' '    # What command?
cat /proc/$PID/status | grep State       # Running? Sleeping? Zombie?
cat /proc/$PID/wchan                     # What kernel function is it waiting in?
ls -la /proc/$PID/fd/ | wc -l           # How many open files?
cat /proc/$PID/status | grep VmRSS       # How much memory?

Pattern 2: "What file is this process holding open?"¶

# Find deleted files still consuming disk space
find /proc/*/fd -type l -exec ls -la {} \; 2>/dev/null | grep deleted

# Find which process has a specific file open
ls -la /proc/*/fd/* 2>/dev/null | grep "myapp.log"

Pattern 3: "What network connections does this process have?"¶

# All network connections for a process
ls -la /proc/$PID/fd/ | grep socket
# Then look up socket inodes in /proc/net/tcp, /proc/net/tcp6, /proc/net/unix

Pattern 4: "Why is the system slow?"¶

# Load average (is the system overloaded?)
cat /proc/loadavg

# Memory pressure (is it swapping?)
cat /proc/meminfo | grep -E "MemAvailable|SwapTotal|SwapFree"

# I/O stats per disk
cat /proc/diskstats

# Interrupts (is one CPU handling all interrupts?)
cat /proc/interrupts | head -5

Flashcard Check¶

Q1: /proc/PID/environ — what does it show?

ALL environment variables of the process, including secrets stored in env vars. Readable by the process owner and root.

Q2: What's the difference between RSS and PSS?

RSS counts shared libraries fully for each process. PSS divides shared memory proportionally. PSS is more accurate for total system memory accounting.

Q3: /proc/PID/fd/3 is a symlink to socket:[89012]. What does that mean?

File descriptor 3 is a network socket. The number 89012 is the inode. Cross-reference with /proc/net/tcp to find the actual IP and port.

Q4: You rm a 10GB file but df shows no change. How do you find it?

Check /proc/*/fd/ for symlinks marked (deleted). A process still has the file open. Truncate through the fd: > /proc/PID/fd/N

Cheat Sheet¶

Per-Process¶

What	File	Example
Command line	`/proc/PID/cmdline`	`cat /proc/PID/cmdline \\| tr '\0' ' '`
Environment	`/proc/PID/environ`	`cat /proc/PID/environ \\| tr '\0' '\n'`
Status/state	`/proc/PID/status`	`grep VmRSS /proc/PID/status`
Open files	`/proc/PID/fd/`	`ls -la /proc/PID/fd/`
Memory maps	`/proc/PID/maps`	`cat /proc/PID/maps`
Memory detail	`/proc/PID/smaps_rollup`	`cat /proc/PID/smaps_rollup`
Kernel wait	`/proc/PID/wchan`	`cat /proc/PID/wchan`
Binary path	`/proc/PID/exe`	`ls -la /proc/PID/exe`
Working dir	`/proc/PID/cwd`	`ls -la /proc/PID/cwd`

System-Wide¶

What	File	What reads it
Memory	`/proc/meminfo`	`free`
CPU	`/proc/cpuinfo`	`lscpu`
Load	`/proc/loadavg`	`uptime`, `top`
Network	`/proc/net/tcp`	`ss`, `netstat`
Mounts	`/proc/mounts`	`mount`, `df`
Kernel params	`/proc/sys/`	`sysctl`
Disks	`/proc/diskstats`	`iostat`

Takeaways¶

Every debugging tool reads /proc. ps, top, free, ss, lsof — they're all fancy wrappers around /proc files. Understanding /proc means understanding the source.
Environment variables in /proc are not secure. /proc/PID/environ is readable by the process owner. Don't store production secrets in env vars if other users share the system.
Deleted files stay open in /proc. rm removes the name; the data stays until all file descriptors close. /proc/PID/fd/ shows these ghost files.
RSS lies about shared memory. Use PSS for accurate per-process accounting. RSS double-counts shared libraries.
/proc is generated on every read. The files are always current — they reflect the kernel's state right now, not a snapshot from when you first looked.

strace: Reading the Matrix — syscall-level debugging that complements /proc
The Hanging Deploy — using /proc to find what a stuck process is doing
Out of Memory — /proc/meminfo and cgroup memory accounting