Skip to content

Mount & Filesystems - Primer

Why This Matters

Mounts are where storage meets the operating system. Every disk, every network share, every tmpfs backing your containers passes through the mount subsystem. A single typo in /etc/fstab can render a server unbootable. A hung NFS mount can freeze your application and cascade into a full outage. A forgotten nofail option means your server never comes back after a reboot. These are not edge cases — they are the bread and butter of ops firefighting.

Core Concepts

1. Mount Basics

A mount attaches a filesystem to a directory (the mount point) in the VFS tree. The kernel tracks all mounts internally; userspace sees them through /proc/mounts (authoritative) and /etc/mtab (legacy, often a symlink to /proc/mounts).

# Mount a block device
mount /dev/sdb1 /mnt/data

# Mount with explicit filesystem type
mount -t ext4 /dev/sdb1 /mnt/data

# Show all current mounts
mount

# Show mounts from the kernel's perspective
cat /proc/mounts

Key rules: - The mount point directory must exist before mounting. - Mounting over a non-empty directory hides its contents (they return when you unmount). - A device can be mounted at multiple points simultaneously.

Unmounting:

# Standard unmount
umount /mnt/data

# Lazy unmount — detaches immediately, cleans up when no longer busy
umount -l /mnt/data

# Force unmount (NFS only — dangerous)
umount -f /mnt/nfsshare

2. /etc/fstab

The filesystem table tells the system what to mount at boot. Each line has six fields:

<device>  <mountpoint>  <fstype>  <options>  <dump>  <pass>
Field Purpose Example
device Block device, UUID, or LABEL UUID=a1b2c3d4-...
mountpoint Where to attach /mnt/data
fstype Filesystem type ext4, xfs, nfs
options Comma-separated mount options defaults,noatime
dump Backup flag (0 = skip) 0
pass fsck order (0 = skip, 1 = root, 2 = other) 2

Real example:

UUID=3e8c4f1a-7b2d-4e9f-a1c3-8d5e6f7a0b1c  /mnt/data  ext4  defaults,noatime,nofail  0  2

Common options:

Option Effect
defaults rw, suid, dev, exec, auto, nouser, async
noatime Do not update access times (big perf win on busy filesystems)
nosuid Block setuid/setgid bits (security hardening)
noexec Prevent binary execution (security hardening for /tmp, /var)
nofail Do not halt boot if this mount fails
ro Read-only
_netdev Wait for network before mounting (required for NFS/iSCSI in fstab)

UUID vs device path: Always use UUID= or LABEL= in fstab. Device paths like /dev/sdb1 can shift when disks are added or removed. Find UUIDs with:

blkid
# /dev/sdb1: UUID="3e8c4f1a-7b2d-4e9f-a1c3-8d5e6f7a0b1c" TYPE="ext4"

lsblk -f
# Shows filesystem type, label, UUID, and mountpoint in one view

3. Filesystem Types

Type Use Case Key Traits
ext4 General-purpose Linux Journaling, mature, max 1 EiB volume, good default
xfs Large files, high throughput Cannot shrink, excellent parallel I/O, default on RHEL
btrfs Snapshots, compression, RAID Copy-on-write, subvolumes, still maturing for production
tmpfs Temporary data in RAM Volatile — gone on reboot, fast, size-limited by RAM
nfs Network-shared storage Remote filesystem, latency-sensitive, requires running daemon
cifs Windows/Samba shares SMB protocol, credential-based, mount.cifs package needed

Choose ext4 as your default. Use xfs for large-file workloads or RHEL environments. Use tmpfs for /tmp or build caches where persistence is not needed. Use btrfs when you need snapshots or transparent compression and can tolerate its operational complexity.

4. NFS Mounts

NFS is the most common source of mount-related outages. When the NFS server goes away, clients can hang indefinitely.

Mounting NFS:

# Basic NFS mount
mount -t nfs nfs-server:/export/data /mnt/nfsdata

# With options
mount -t nfs -o hard,timeo=300,retrans=3,bg nfs-server:/export/data /mnt/nfsdata

Critical NFS options:

Option Meaning
hard Retry indefinitely (default). Processes hang until server returns.
soft Give up after retrans retries. Returns I/O error to application.
timeo=N Timeout in tenths of a second before retry (default 600 = 60s)
retrans=N Number of retries before soft mount gives up (default 3)
bg Retry mount in background if first attempt fails
intr Allow signals to interrupt hung NFS operations (deprecated in NFSv4)
_netdev Required in fstab — wait for network before mounting

Diagnosing hung NFS:

# Check if any NFS operations are stuck
cat /proc/self/mountstats | grep -A 5 "nfs"

# See NFS client statistics
nfsstat -c

# Find processes stuck on NFS
ps aux | grep " D "   # D = uninterruptible sleep, often NFS

# Check NFS server availability
showmount -e nfs-server
rpcinfo -p nfs-server

fstab line for NFS:

nfs-server:/export/data  /mnt/nfsdata  nfs  hard,timeo=300,retrans=3,bg,_netdev,nofail  0  0

5. Troubleshooting Unmount — "Device Is Busy"

The most common mount headache: you cannot unmount because something is using the filesystem.

# Find what is using the mount point
lsof +D /mnt/data

# Find processes using the mount (verbose)
fuser -vm /mnt/data
#                      USER        PID ACCESS COMMAND
# /mnt/data:           root       1234 ..c.. bash
#                       app        5678 F.... python3

# Kill all processes using the mount
fuser -km /mnt/data    # Sends SIGKILL — use with caution

# Check for open file descriptors pointing at deleted files
lsof +L1 /mnt/data

ACCESS column in fuser output: - c — current directory - e — running executable - f — open file - F — open file for writing - r — root directory - m — mmap'd file

6. Read-Only Remount

Remounting read-only is an emergency operation for when a filesystem is corrupt or you need to guarantee no writes before maintenance:

# Remount root filesystem read-only
mount -o remount,ro /

# Remount a specific filesystem read-only
mount -o remount,ro /mnt/data

# Remount read-write after maintenance
mount -o remount,rw /mnt/data

When to use: - Filesystem errors detected — remount ro to prevent further damage. - Pre-snapshot on LVM — remount ro, take snapshot, remount rw. - Emergency disk diagnostics — prevent writes while investigating. - System stuck in read-only after crash — mount -o remount,rw / to recover.

7. Bind Mounts

A bind mount makes a directory (or file) accessible at a second location in the filesystem tree. It does not involve a block device.

# Bind mount a directory
mount --bind /var/log /mnt/chroot/var/log

# Make it read-only (two-step)
mount --bind /var/log /mnt/chroot/var/log
mount -o remount,bind,ro /mnt/chroot/var/log

fstab entry for bind mount:

/var/log  /mnt/chroot/var/log  none  bind  0  0

Use cases: - Containers and chroots: expose host directories inside isolated environments. - Selective sharing: give a service access to a specific host path without symlinks. - Build environments: overlay source directories into build chroots.

Docker and Kubernetes use bind mounts extensively under the hood for volume mounts (hostPath volumes in k8s are bind mounts).

8. findmnt and lsblk

These are the modern tools for inspecting mount state. Prefer them over parsing mount output.

# Tree view of all mounts
findmnt
# TARGET                       SOURCE     FSTYPE  OPTIONS
# /                            /dev/sda2  ext4    rw,relatime
# |-/boot                      /dev/sda1  ext4    rw,relatime
# |-/mnt/data                  /dev/sdb1  xfs     rw,noatime
# |-/tmp                       tmpfs      tmpfs   rw,nosuid,noexec

# Find where a device is mounted
findmnt -S /dev/sdb1

# Find what is mounted at a path
findmnt -T /mnt/data

# Output as JSON for scripting
findmnt -J

# Show block devices and their mount points
lsblk
# NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
# sda      8:0    0   50G  0 disk
# |-sda1   8:1    0  512M  0 part /boot
# |-sda2   8:2    0 49.5G  0 part /
# sdb      8:16   0  100G  0 disk
# |-sdb1   8:17   0  100G  0 part /mnt/data

# Show filesystem info with lsblk
lsblk -f

9. Dangers: fstab Mistakes

The single most dangerous mount-related mistake: a bad fstab entry without nofail.

What happens: 1. You add a line to /etc/fstab for a new disk or NFS share. 2. You forget the nofail option. 3. The server reboots. 4. The disk is missing, or the NFS server is down. 5. systemd-fstab-generator creates a mount unit that blocks boot. 6. The server never comes up. You need console access to fix it.

Prevention:

# ALWAYS test fstab changes before rebooting
mount -a
# If this fails, fix fstab BEFORE rebooting

# ALWAYS add nofail for non-critical mounts
UUID=xxx  /mnt/data  ext4  defaults,nofail  0  2

# For NFS, add both _netdev and nofail
nfs-server:/share  /mnt/share  nfs  defaults,_netdev,nofail  0  0

Other common fstab mistakes: - Wrong UUID (copy-paste error) — server hangs at boot waiting for a device that does not exist. - Missing _netdev on network filesystems — mount attempted before network is up, fails or hangs. - Setting pass field to 1 on non-root — fsck runs in serial, slowing boot. Use 2 for non-root or 0 to skip. - Stale NFS entry — server decommissioned, fstab still references it, boot hangs.

Recovery when fstab breaks boot: 1. Boot into single-user mode or use a rescue disk. 2. Remount root read-write: mount -o remount,rw / 3. Edit /etc/fstab — comment out the bad line. 4. Reboot normally.