Skip to content

Portal | Level: L0: Entry | Topics: Linux Fundamentals, Bash / Shell Scripting, systemd, Package Management | Domain: Linux

Linux System Administration - Primer

Why This Matters

Linux runs the vast majority of production infrastructure. Whether your workloads run on bare metal, VMs, or containers, the underlying OS is almost always Linux. Knowing how to navigate, configure, troubleshoot, and secure a Linux system is the foundation of every DevOps role.

Core Concepts

The Filesystem Hierarchy

Linux organizes everything under a single root /. Key directories:

Path Purpose
/etc System configuration files
/var Variable data: logs (/var/log), spool, lib
/home User home directories
/tmp Temporary files (often tmpfs, cleared on reboot)
/opt Third-party software
/proc Virtual filesystem exposing kernel/process info
/sys Virtual filesystem for hardware/kernel parameters
/dev Device files (disks, terminals, null, random)
/usr User programs, libraries, documentation
/boot Kernel, initramfs, bootloader files

Everything is a file in Linux. Devices are files (/dev/sda). Processes expose their info as files (/proc/<pid>/). Kernel tunables are files (/sys/, /proc/sys/).

Name origin: The Filesystem Hierarchy Standard (FHS) traces back to the earliest Unix systems at Bell Labs. /etc originally stood for "et cetera" — a catch-all for files that did not fit elsewhere. Over time it became exclusively configuration. /var is short for "variable" — data that changes during operation. /usr originally meant "user" but now stands for "Unix System Resources."

Users and Permissions

Every file has an owner (user), a group, and three permission sets: user, group, other. Each set has read (r=4), write (w=2), execute (x=1).

# Read permissions
ls -la /etc/passwd
# -rw-r--r-- 1 root root 2847 Jan 15 10:00 /etc/passwd
#  ^owner ^group ^other

# Change permissions
chmod 750 script.sh        # rwxr-x---
chmod u+x script.sh        # Add execute for owner

# Change ownership
chown appuser:appgroup /opt/myapp -R

# Special permissions
chmod u+s /usr/bin/passwd   # SUID: runs as file owner
chmod g+s /shared/dir       # SGID: new files inherit group
chmod +t /tmp               # Sticky bit: only owner can delete

Key files: - /etc/passwd - user accounts (name, UID, GID, home, shell) - /etc/shadow - password hashes (root-readable only) - /etc/group - group definitions - /etc/sudoers - sudo privileges (always edit with visudo)

Process Management

A process is a running instance of a program. Every process has a PID, a parent (PPID), a user, and resource consumption.

# View processes
ps aux                     # All processes, full detail
ps -ef --forest            # Process tree
top / htop                 # Interactive, real-time

# Key signals
kill -15 <pid>             # SIGTERM: polite shutdown
kill -9 <pid>              # SIGKILL: force kill (last resort)
kill -1 <pid>              # SIGHUP: reload config (many daemons)

# Background jobs
command &                  # Run in background
nohup command &            # Survive terminal close
jobs                       # List background jobs
fg %1                      # Bring job 1 to foreground

Process states: Running (R), Sleeping (S), Stopped (T), Zombie (Z), Disk sleep/uninterruptible (D).

systemd

systemd is the init system on most modern Linux distributions. It manages services, mounts, timers, and the boot process.

# Service management
systemctl start nginx          # Start now
systemctl stop nginx           # Stop now
systemctl restart nginx        # Stop then start
systemctl reload nginx         # Reload config without restart
systemctl enable nginx         # Start on boot
systemctl disable nginx        # Don't start on boot
systemctl status nginx         # Current state, recent logs

# Viewing logs
journalctl -u nginx            # All logs for nginx
journalctl -u nginx --since "1 hour ago"
journalctl -u nginx -f         # Follow (like tail -f)
journalctl -p err -b           # All errors since boot
journalctl -k                  # Kernel messages only

# System state
systemctl list-units --failed  # What's broken?
systemctl list-timers          # Scheduled tasks (cron replacement)
systemd-analyze blame          # Boot time per service

Unit files live in /etc/systemd/system/ (admin overrides) and /usr/lib/systemd/system/ (package defaults). After editing, run systemctl daemon-reload.

Package Management

Two major families:

Family Distros Package tool Repo tool
RPM RHEL, CentOS, Rocky, Alma, Fedora rpm dnf (formerly yum)
DEB Debian, Ubuntu dpkg apt 
# RPM-based (dnf)
dnf install nginx              # Install
dnf remove nginx               # Remove
dnf update                     # Update all packages
dnf search nginx               # Search repos
dnf info nginx                 # Package details
dnf list installed             # All installed packages
rpm -qa | grep nginx           # Query installed RPMs
rpm -ql nginx                  # List files in package

# DEB-based (apt)
apt update                     # Refresh repo index
apt install nginx              # Install
apt remove nginx               # Remove (keep config)
apt purge nginx                # Remove + config
apt upgrade                    # Update all packages
dpkg -l | grep nginx           # Query installed
dpkg -L nginx                  # List files in package

Disk and Filesystem Operations

# Disk space
df -h                          # Filesystem usage
du -sh /var/log/*              # Directory sizes
lsblk                         # Block device tree
fdisk -l                      # Partition tables

# Finding large files
find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null
du -ah / | sort -rh | head -20

# Mounting
mount /dev/sdb1 /mnt/data
umount /mnt/data
# Persistent mounts: /etc/fstab

# LVM basics
pvs                            # Physical volumes
vgs                            # Volume groups
lvs                            # Logical volumes
lvextend -L +10G /dev/vg0/data # Grow a logical volume
resize2fs /dev/vg0/data        # Grow the filesystem (ext4)
xfs_growfs /mnt/data           # Grow the filesystem (XFS)

What Experienced People Know

  • journalctl is your best friend. Learn its flags cold. Most answers to "why did this break" live there.

    Remember: Permission number mnemonic: Read=4, Write=2, eXecute=1. Think "RWX = 421." So 755 = rwx (4+2+1) for owner, r-x (4+1) for group, r-x (4+1) for others.

  • Never run chmod -R 777 on anything. If you think you need it, you don't understand the permission model yet.

  • systemctl enable and systemctl start are separate operations. Enabling doesn't start; starting doesn't enable.
  • /tmp is not persistent storage. Don't put anything there you can't afford to lose.
  • The difference between kill -9 and kill -15 matters. Always try SIGTERM first. SIGKILL leaves no chance for cleanup.
  • When disk is full, df -h might look fine if the issue is inode exhaustion. Check df -i too.

Networking Tools

When a production system has network issues, you need to diagnose fast. The right tool answers the right question: is the interface up? Is the port open? Where is the packet being dropped? Are we hitting a firewall? Linux provides a powerful toolkit, but knowing which tool to reach for — and how to interpret its output — separates quick resolution from hours of guessing.

ip — The Swiss Army Knife

Fun fact: The ifconfig, route, and netstat commands (from the net-tools package) were deprecated in favor of ip, ss, and other iproute2 tools. net-tools was last released in 2001. If you see ifconfig in documentation, it is at least 20 years out of date. Some minimal container images do not include net-tools at all.

The ip command (from iproute2) replaces ifconfig, route, and arp.

# Interface status
ip link show
ip -br link show              # brief view
ip link set eth0 up/down

# IP addresses
ip addr show
ip addr add 10.0.0.1/24 dev eth0
ip addr del 10.0.0.1/24 dev eth0

# Routes
ip route show
ip route get 8.8.8.8          # which route would be used?
ip route add 10.1.0.0/16 via 10.0.0.254
ip route del 10.1.0.0/16

# Neighbors (ARP table)
ip neigh show
ip neigh flush dev eth0

# Namespaces (container debugging)
ip netns list
ip netns exec <ns> ip addr show

ss — Socket Statistics

Replaces netstat. Faster, more detailed.

One-liner: Quick check for anything listening on a specific port: ss -tlnp 'sport = :8080' — shows the process name and PID bound to port 8080.

# All listening TCP sockets with process info
ss -tlnp

# All established connections
ss -tnp state established

# Connections to a specific port
ss -tnp dport = :443

# Show socket memory usage
ss -tmn

# UDP sockets
ss -ulnp

# Summary statistics
ss -s

# Filter by state
ss -tn state time-wait
ss -tn state close-wait       # often indicates app not closing connections

tcpdump — Packet Capture

The essential debugging tool. See exactly what is on the wire.

# Capture on interface, human-readable
tcpdump -i eth0 -nn

# Filter by host and port
tcpdump -i eth0 host 10.0.0.5 and port 80

# Write to file for Wireshark analysis
tcpdump -i eth0 -w /tmp/capture.pcap -c 1000

# Read from file
tcpdump -r /tmp/capture.pcap

# Show packet contents (ASCII)
tcpdump -i eth0 -A port 80

# DNS queries
tcpdump -i eth0 port 53

# SYN packets only (connection attempts)
tcpdump -i eth0 'tcp[tcpflags] & tcp-syn != 0'

# Packets with RST flag (connection resets)
tcpdump -i eth0 'tcp[tcpflags] & tcp-rst != 0'

ethtool — NIC Diagnostics

# Link status and speed
ethtool eth0

# Driver and firmware info
ethtool -i eth0

# Interface statistics (errors, drops)
ethtool -S eth0

# NIC ring buffer sizes
ethtool -g eth0

# Offload settings
ethtool -k eth0

# Change speed/duplex (rare)
ethtool -s eth0 speed 1000 duplex full autoneg on

Key counters to watch: - rx_errors, tx_errors — physical layer problems - rx_dropped — kernel dropping packets (ring buffer full?) - rx_crc_errors — cable or hardware issues

nmcli — NetworkManager CLI

# Show connections
nmcli con show
nmcli con show --active

# Device status
nmcli dev status

# Connect to WiFi
nmcli dev wifi connect "SSID" password "pass"

# Create static IP connection
nmcli con add type ethernet con-name eth0-static ifname eth0 \
  ipv4.addresses 10.0.0.5/24 ipv4.gateway 10.0.0.1 \
  ipv4.dns "8.8.8.8" ipv4.method manual

# Modify existing
nmcli con mod eth0 ipv4.dns "8.8.8.8 8.8.4.4"
nmcli con up eth0

mtr — Combined Traceroute + Ping

# Interactive (default)
mtr 8.8.8.8

# Report mode (non-interactive)
mtr -rw -c 100 8.8.8.8

# TCP mode (when ICMP is blocked)
mtr -T -P 443 8.8.8.8

# Show AS numbers
mtr -z 8.8.8.8

Interpreting mtr output: - Loss at intermediate hop only: Usually ICMP rate limiting, not real loss - Loss at intermediate hop AND all subsequent: Real problem at that hop - High latency at one hop: Could be ICMP deprioritization or real congestion

nmap — Network Scanner

# Ping sweep (discover hosts)
nmap -sn 10.0.0.0/24

# Port scan (common ports)
nmap 10.0.0.5

# Specific ports
nmap -p 22,80,443 10.0.0.5

# Service version detection
nmap -sV 10.0.0.5

# OS detection
nmap -O 10.0.0.5

# UDP scan (slow)
nmap -sU -p 53,161 10.0.0.5

Other Useful Network Tools

# DNS lookup
dig @8.8.8.8 example.com A +short
dig example.com MX
host example.com

# curl for HTTP debugging
curl -vvv https://example.com 2>&1 | head -30
curl -o /dev/null -s -w '%{http_code} %{time_total}s\n' https://example.com

# iperf3 for bandwidth testing
iperf3 -s                    # server
iperf3 -c 10.0.0.1 -t 30    # client, 30 second test

# nc (netcat) for port testing
nc -zv 10.0.0.5 80           # test if port is open
nc -l 8080                   # listen on port

Networking Quick Reference

Question Tool Command
Is the interface up? ip ip -br link show
What is my IP? ip ip -br addr show
What route to host? ip ip route get 8.8.8.8
Who is listening? ss ss -tlnp
What is on the wire? tcpdump tcpdump -i eth0 -nn
NIC errors? ethtool ethtool -S eth0
Where is the hop? mtr mtr -rw 8.8.8.8
Is port open? nmap/nc nmap -p 80 host or nc -zv host 80
DNS resolution? dig dig example.com +short
HTTP response? curl curl -vvv https://host

See Also

Wiki Navigation

Next Steps

Related Content