--- tags: - devops - l1 - flashcard-deck - mergerfs --- [Portal](../../../../library/portal/index.md) | **Level:** [L1: Foundations](../../../../library/portal/levels.md) | **Topics:** [mergerfs](../../../../library/portal/topics.md) | **Domain:** DevOps & Tooling id category difficulty tags question answer source_path mergerfs/a1b2c3d4e5f6 mergerfs easy mergerfs, basics, fuse What is mergerfs? A FUSE-based union filesystem that logically combines multiple filesystem paths into a single mount point. Created by Antonio SJ Musumeci (trapexit), licensed under ISC.\n\nKey distinction: mergerfs is NOT RAID -- it provides no redundancy, parity, or striping. It is a thin routing layer that proxies filesystem operations to underlying branches.\n\nName origin: "merger filesystem" -- it merges directory trees from multiple drives into one unified view. training/library/topics/mergerfs/primer.md mergerfs/b2c3d4e5f6a7 mergerfs easy mergerfs, branch, pool What is the difference between a branch and a pool in mergerfs? A **branch** is a single filesystem path that participates in the mergerfs union (e.g., /mnt/disk1). The **pool** is the mergerfs mount point -- the unified view of all branches (e.g., /storage).\n\nBranches are specified as a colon-delimited list: /mnt/disk1:/mnt/disk2:/mnt/disk3\n\nGlobs are supported in fstab: /mnt/disk* merges all matching paths. training/library/topics/mergerfs/primer.md mergerfs/c3d4e5f6a7b8 mergerfs easy mergerfs, branch-modes What are the three branch modes in mergerfs (RW, RO, NC)? **RW** (Read-Write): default, eligible for all policy categories.\n**RO** (Read-Only): excluded from create and action policies, only participates in search.\n**NC** (No-Create): excluded from create policies only, still allows modifications and deletions.\n\nSyntax: /mnt/disk1=RW:/mnt/disk2=NC:/mnt/disk3=RO\n\nUse NC to gracefully drain a drive -- it stops receiving new files but existing files can still be modified or deleted. training/library/topics/mergerfs/primer.md mergerfs/d4e5f6a7b8c9 mergerfs easy mergerfs, categories What are the three policy categories in mergerfs? **Create**: controls where new files/directories are placed (create, mkdir, mknod, symlink)\n**Action**: controls which branches are affected by modifications (chmod, chown, rename, unlink, etc.)\n**Search**: controls how files are located (open, getattr, access, readlink, etc.)\n\nMnemonic: **CAS** -- Create, Action, Search -- like CAS latency in RAM, every access has a lookup cost. training/library/topics/mergerfs/primer.md mergerfs/e5f6a7b8c9d0 mergerfs medium mergerfs, policies, create What is the default create policy (pfrd) and how does it work? **pfrd** = Probabilistic Free-space Random Distribution. Selects a branch randomly with probability proportional to available space. A branch with 2 TB free is twice as likely to be selected as one with 1 TB free.\n\nThis naturally balances data across drives without strict round-robin. Drives with more free space receive more files, approaching even fill over time.\n\nDefault for: category.create training/library/topics/mergerfs/primer.md mergerfs/f6a7b8c9d0e1 mergerfs medium mergerfs, policies What does the epmfs policy do and when should you use it? **epmfs** = Existing Path, Most Free Space. Among branches where the parent directory already exists, picks the one with the most free space.\n\nUse case: media servers. When Sonarr adds a new episode to /storage/tv/ShowName/Season02/, epmfs ensures it lands on the same drive that already has the Season02 directory. This keeps series data colocated for better read performance.\n\nThe 'ep' prefix means "only consider branches with an existing path." training/library/topics/mergerfs/primer.md mergerfs/a7b8c9d0e1f2 mergerfs easy mergerfs, policies What is the default action policy and why? **epall** (Existing Path, All) is the default action policy. It applies the operation to ALL branches where the target path exists.\n\nWhy: when you chmod or rename a file, all instances across branches should be affected. If a file exists on multiple branches (e.g., a directory), epall ensures the operation is consistent everywhere. training/library/topics/mergerfs/primer.md mergerfs/b8c9d0e1f2a3 mergerfs easy mergerfs, policies What is the default search policy (ff) and how does it work? **ff** = First Found. Returns the first matching file in branch mount order.\n\nFast and simple -- stops searching as soon as a match is found. Branch order matters: branches listed first are checked first.\n\nThis is why listing an SSD branch first with ff as the search policy gives you fast lookups for frequently accessed files. training/library/topics/mergerfs/primer.md mergerfs/c9d0e1f2a3b4 mergerfs medium mergerfs, policies, prefix What do the ep* and msp* policy prefixes mean? **ep** (Existing Path): only considers branches where the target directory already exists. Falls back to nothing if no branch has the path.\n\n**msp** (Most Specific Path): like ep* but if the exact path does not exist, retries with the parent directory, then grandparent, etc.\n\nExamples:\n- epmfs: among existing paths, most free space\n- mspmfs: like epmfs, but walks up directory tree if needed\n- eppfrd: among existing paths, weighted random by free space\n- msppfrd: like eppfrd, walks up if needed training/library/topics/mergerfs/primer.md mergerfs/d0e1f2a3b4c5 mergerfs medium mergerfs, config What does minfreespace do and what is a good value? Branches with less than minfreespace available are excluded from **create** policies. Default is 4G, which is dangerously low for media setups.\n\nDoes NOT affect action or search policies. Does NOT prevent writes to already-open files on full branches (that is moveonenospc's job).\n\nRecommended values:\n- Media server: 100G-250G\n- General NAS: 20G-50G\n- Set to at least 2x your largest expected file size. training/library/topics/mergerfs/primer.md mergerfs/e1f2a3b4c5d6 mergerfs medium mergerfs, config What does moveonenospc do and what is its critical limitation? When a write fails with ENOSPC (drive full), mergerfs moves the file to another branch and retries the write.\n\nCritical limitation: moveonenospc only affects **writes**, NOT creates. If the create policy selects a full branch for a new file, the create fails with ENOSPC and moveonenospc does NOT intervene.\n\nThis is why minfreespace must be set high enough -- it is the create-time protection. moveonenospc is only the write-time safety net. training/library/topics/mergerfs/primer.md mergerfs/f2a3b4c5d6e7 mergerfs medium mergerfs, config, cache What does dropcacheonclose do and why is it important for media servers? When enabled, mergerfs calls posix_fadvise(DONTNEED) when a file is closed, telling the kernel to drop that file's page cache.\n\nWithout it, streaming a 50 GB movie through Plex fills the page cache with data that will never be read again, evicting useful cached data and causing system sluggishness.\n\nAlways enable for media server workloads: dropcacheonclose=true training/library/topics/mergerfs/primer.md mergerfs/a3b4c5d6e7f8 mergerfs hard mergerfs, exdev What is EXDEV and how does rename-exdev handle it? EXDEV (error 18) = "Invalid cross-device link." Returned when rename() or link() crosses filesystem boundaries. Since each mergerfs branch is a separate filesystem, renaming between branches triggers EXDEV.\n\nrename-exdev options:\n- **passthrough** (default): returns EXDEV to the application\n- **rel-symlink**: creates a relative symlink as workaround\n- **abs-symlink**: creates an absolute symlink as workaround\n\nGotcha: some apps check file type after rename and reject symlinks, so test before enabling symlink modes. training/library/topics/mergerfs/primer.md mergerfs/b4c5d6e7f8a9 mergerfs medium mergerfs, runtime How do you query and change mergerfs settings at runtime? Use the .mergerfs pseudo-file at the mount point with xattr operations:\n\nQuery all: getfattr -d /storage/.mergerfs\nRead one: getfattr -n user.mergerfs.category.create /storage/.mergerfs\nChange: setfattr -n user.mergerfs.category.create -v mfs /storage/.mergerfs\n\nAdd branch: setfattr -n user.mergerfs.srcmounts -v '+>/mnt/disk5' /storage/.mergerfs\nRemove branch: setfattr -n user.mergerfs.srcmounts -v '-/mnt/disk3' /storage/.mergerfs\n\nChanges are NOT persisted -- update fstab to persist. training/library/topics/mergerfs/primer.md mergerfs/c5d6e7f8a9b0 mergerfs easy mergerfs, fstab Write a basic mergerfs fstab line for a media server with four drives. /mnt/disk* /storage fuse.mergerfs defaults,allow_other,use_ino,cache.files=off,moveonenospc=true,dropcacheonclose=true,minfreespace=250G,category.create=epmfs,fsname=mergerfs 0 0\n\nKey options explained:\n- allow_other: non-root access (Docker containers)\n- use_ino: consistent inodes for NFS/Samba\n- cache.files=off: prevents stale metadata\n- dropcacheonclose=true: prevents page cache pollution\n- epmfs: path-preserving create policy training/library/topics/mergerfs/street_ops.md mergerfs/d6e7f8a9b0c1 mergerfs hard mergerfs, snapraid Why must parity drives NEVER be in the mergerfs pool? mergerfs will write user data to any branch in the pool. If the parity drive is included, user files overwrite parity data, silently destroying SnapRAID protection.\n\nThe fix is a naming convention: data at /mnt/disk1, /mnt/disk2, parity at /mnt/parity1. Glob only /mnt/disk* in mergerfs.\n\nThis is the #1 most destructive mergerfs mistake. Your entire parity protection is permanently ruined with no warning. training/library/topics/mergerfs/footguns.md mergerfs/e7f8a9b0c1d2 mergerfs medium mergerfs, footgun What is a recursive mergerfs mount and how do you avoid it? Mounting mergerfs at a path that is inside one of its own branch paths. Example: branches at /mnt/* with pool at /mnt/storage -- the pool is caught by its own glob.\n\nSymptoms: system hangs, ls never returns, I/O errors.\n\nFix: always mount the pool outside the branch namespace. Branches at /mnt/disk*, pool at /storage. If stuck, use umount -l for lazy unmount. training/library/topics/mergerfs/footguns.md mergerfs/f8a9b0c1d2e3 mergerfs medium mergerfs, policies When would you use lup (Least Used Percentage) as a create policy? Use lup when you have drives of different sizes and want to keep them balanced by percentage rather than absolute free space.\n\nExample: a 4 TB drive at 50% and a 14 TB drive at 50% -- lup treats them as equally full. mfs would always pick the 14 TB drive (7 TB free vs 2 TB free).\n\nBest for: general NAS with mixed drive sizes where visual balance matters. training/library/topics/mergerfs/primer.md mergerfs/a9b0c1d2e3f4 mergerfs hard mergerfs, threading Explain mergerfs threading: read-thread-count and process-thread-count. read-thread-count (default 0): threads reading FUSE kernel messages. With process-thread-count=-1, creates one combined thread per CPU core (max 8).\n\nprocess-thread-count (default -1): threads processing messages. -1 disables separate pool (processing on read threads). 0 creates one per CPU core.\n\nprocess-thread-queue-depth (default 2): max queued requests per process thread.\n\nFor high concurrency (10+ streams): set both to 0 for separate read and process pools. training/library/topics/mergerfs/primer.md mergerfs/b0c1d2e3f4a5 mergerfs easy mergerfs, config What does cache.files control and what are its modes? Controls page caching for file data in FUSE:\n\n- **off** (default): no caching, safest for multi-writer\n- **partial**: cached while file handle is open\n- **full**: cached across opens\n- **auto-full**: cached if mtime/size unchanged between opens\n- **per-process**: selective caching by process name\n\nFor media servers, use off. For database-like workloads, auto-full + cache.writeback=true can help. training/library/topics/mergerfs/primer.md mergerfs/c1d2e3f4a5b6 mergerfs medium mergerfs, nfs What NFS export settings are needed for mergerfs? 1. Set explicit fsid per export: fsid=1 (NFS needs stable device IDs, FUSE provides synthetic ones)\n2. Enable nfsopenhack=all in mergerfs options (fixes file creation issues)\n3. Use crossmnt for subtree exports\n4. Set use_ino in mergerfs for consistent inodes\n\nExample: /storage 192.168.1.0/24(rw,fsid=1,no_subtree_check,crossmnt,all_squash,anonuid=1000,anongid=1000) training/library/topics/mergerfs/primer.md mergerfs/d2e3f4a5b6c7 mergerfs easy mergerfs, docker How do you mount mergerfs storage into Docker containers? Bind-mount the mergerfs pool path into containers:\n\nvolumes:\n - /storage/media:/data/media\n - /storage/downloads:/data/downloads\n\nKey practices:\n- Use PUID=1000 and PGID=1000 on all containers for consistent ownership\n- Put application configs on SSD, not mergerfs (e.g., Plex DB on NVMe)\n- Point containers at the pool mount (/storage), not individual branches training/library/topics/mergerfs/street_ops.md mergerfs/e3f4a5b6c7d8 mergerfs hard mergerfs, snapraid, recovery What is the correct order for drive replacement with SnapRAID + mergerfs? 1. Remove failed branch from pool: setfattr -n user.mergerfs.srcmounts -v '-/mnt/diskN' /storage/.mergerfs\n2. Physically replace drive\n3. Format replacement: mkfs.ext4 -m 0 -T largefile4 /dev/sdX1\n4. Mount replacement at same path\n5. Restore data: snapraid fix -d dN\n6. Re-add to pool: setfattr ...srcmounts -v '+>/mnt/diskN' ...\n7. THEN sync: snapraid sync\n\nCRITICAL: Never sync before fix. Syncing after loss recalculates parity without the missing data, destroying recovery ability. training/library/topics/mergerfs/street_ops.md mergerfs/f4a5b6c7d8e9 mergerfs medium mergerfs, func How do func.* and category.* overrides interact? category.* sets the policy for all functions in that category:\ncategory.create=mfs (affects create, mkdir, mknod, symlink)\n\nfunc.* sets the policy for one specific function:\nfunc.mkdir=all (only affects mkdir)\n\nfunc.* overrides take precedence over category.*. So you can set category.create=mfs but override func.mkdir=all to create directories on all branches while creating files on the most-free-space branch. training/library/topics/mergerfs/primer.md mergerfs/a5b6c7d8e9f0 mergerfs medium mergerfs, config What is the all policy and when is it useful? The **all** policy applies the operation to every branch. For mkdir/mknod/symlink, it creates the directory on all branches. For create (file creation), it acts like ff.\n\nCommon use: func.mkdir=all ensures directory structures exist on all branches, so future epmfs/eplfs create operations always find an existing path.\n\nThis is why the default action policy is epall -- modifications should propagate to all branches that have the file. training/library/topics/mergerfs/primer.md mergerfs/b6c7d8e9f0a1 mergerfs easy mergerfs, basics What does mergerfs NOT provide? mergerfs does NOT provide:\n- RAID (no striping, mirroring, or parity)\n- Redundancy (losing a drive loses its files)\n- Checksums or data integrity verification\n- Snapshots\n- Automatic rebalancing of existing files\n- File splitting across branches\n- Copy-on-write semantics like OverlayFS\n\nFor redundancy, pair with SnapRAID. For checksums, use btrfs/ZFS on the branches or SnapRAID scrub. training/library/topics/mergerfs/primer.md mergerfs/c7d8e9f0a1b2 mergerfs hard mergerfs, cache, performance What does cache.writeback do and when should you enable it? When cache.files is enabled and cache.writeback=true, the kernel aggregates small writes into larger FUSE requests before sending them to mergerfs. This can dramatically improve throughput for apps that write many small chunks.\n\nWithout writeback, each write goes through FUSE individually (writethrough). With writeback, the kernel buffers writes and flushes them in larger batches.\n\nEnable for: database workloads, build systems, apps with many small sequential writes.\nAvoid for: media streaming (use cache.files=off instead). training/library/topics/mergerfs/primer.md mergerfs/d8e9f0a1b2c3 mergerfs medium mergerfs, globbing Why does glob-based branch specification only work at mount time? mergerfs evaluates glob patterns (/mnt/disk*) only when the filesystem is mounted. If you add /mnt/disk5 after mounting, it will NOT automatically appear in the pool.\n\nTo add new branches after mount, either:\n1. Remount: umount /storage && mount /storage\n2. Use xattr: setfattr -n user.mergerfs.srcmounts -v '+>/mnt/disk5' /storage/.mergerfs\n\nThe xattr method requires no downtime -- services keep running. training/library/topics/mergerfs/primer.md mergerfs/e9f0a1b2c3d4 mergerfs easy mergerfs, basics How do you check which physical branch a file resides on? getfattr -n user.mergerfs.fullpath /storage/path/to/file\n\nThis returns the actual path on the underlying branch, e.g., /mnt/disk2/path/to/file.\n\nUseful for debugging policy behavior, verifying file placement after migration, and planning drive removal (find all files on a specific branch). training/library/topics/mergerfs/street_ops.md mergerfs/f0a1b2c3d4e5 mergerfs hard mergerfs, link-exdev What are the link-exdev modes and when does abs-base-symlink differ from abs-pool-symlink? link-exdev handles EXDEV errors on hard link operations:\n- **passthrough** (default): returns EXDEV to caller\n- **rel-symlink**: relative symlink from new to old location\n- **abs-base-symlink**: absolute symlink using the underlying branch path (/mnt/disk1/...)\n- **abs-pool-symlink**: absolute symlink using the pool mount path (/storage/...)\n\nabs-base-symlink bypasses mergerfs for the link target. abs-pool-symlink goes through mergerfs, which means search policies apply when following the link. training/library/topics/mergerfs/primer.md mergerfs/a0b1c2d3e4f5 mergerfs medium mergerfs, fuse, performance How does fuse_msg_size affect mergerfs performance? fuse_msg_size controls the maximum FUSE message size in pages (4 KiB per page). Default is 256 (1 MiB max). Available on Linux 4.20+ via the max_pages feature.\n\nDoubling the message size approximately halves the number of kernel-to-userspace round trips for large I/O. Since mergerfs defaults to the maximum (256), there is no reason to change it.\n\nIf you see it set lower than 256, increase it. The only cost is slightly higher memory usage for message buffers. training/library/topics/mergerfs/primer.md mergerfs/b1c2d3e4f5a6 mergerfs medium mergerfs, policies What is the newest policy and when would you use it? **newest** selects the branch with the largest mtime (modification time) on the target file or directory.\n\nUse case: when files may exist on multiple branches (e.g., after a migration or manual copy) and you want operations to target the most recently modified version.\n\nNot commonly used as a create policy. More useful as a search policy when you have overlapping content across branches and want the freshest version. training/library/topics/mergerfs/primer.md mergerfs/c2d3e4f5a6b7 mergerfs easy mergerfs, isc, license What license is mergerfs released under? The ISC license -- one of the most permissive open-source licenses. It is a two-clause license (similar to simplified BSD/MIT) that permits personal and commercial use with minimal restrictions.\n\nISC = Internet Systems Consortium, the same organization behind BIND (DNS server).\n\nThe creator, Antonio SJ Musumeci (trapexit), chose ISC for maximum adoption with minimum legal overhead. training/library/topics/mergerfs/primer.md mergerfs/d3e4f5a6b7c8 mergerfs hard mergerfs, security Why should you disable security_capability when using cache.files? With cache.files enabled and security_capability=true (default), the kernel sends a getxattr for security.capability before EVERY SINGLE WRITE. This xattr check is not cached, so it adds a FUSE round trip per write.\n\nDisabling with security_capability=false returns ENOATTR immediately without the round trip. This can significantly improve write performance when file caching is enabled.\n\nOnly relevant when cache.files is not off. If cache.files=off, this has no effect. training/library/topics/mergerfs/primer.md mergerfs/e4f5a6b7c8d9 mergerfs medium mergerfs, statfs What is the difference between statfs=base and statfs=full? **statfs=base** (default): uses all branches for df/statfs calculations. The pool always reports total capacity across all drives.\n\n**statfs=full**: only includes branches where the queried path exists. If /storage/movies only exists on disk1 and disk2, df /storage/movies only reports those drives' capacity.\n\nUse base for simple setups. Use full when you need accurate per-directory space reporting (rare). training/library/topics/mergerfs/primer.md --- ## Wiki Navigation ### Related Content - [mergerfs](../../../../library/topics/mergerfs/index.md) (Topic Pack, L2) — mergerfs