--- tags: - linux - l1 - flashcard-deck - linux-data-hoarding --- [Portal](../../../../library/portal/index.md) | **Level:** [L1: Foundations](../../../../library/portal/levels.md) | **Topics:** [Linux Data Hoarding](../../../../library/portal/topics.md) | **Domain:** Linux id category difficulty tags question answer source_path linux-data-hoarding/a1b2c3d4e5f6 linux-data-hoarding easy jbod, philosophy What does JBOD stand for, and why do data hoarders prefer it over traditional RAID? JBOD = Just a Bunch of Disks. Each drive is an independent filesystem combined via a union filesystem (mergerfs). Advantages over RAID: mix any drive sizes, add one drive at a time, any single drive is readable on its own, and a failed drive only loses the files physically on that drive.\n\nMnemonic: "JBOD = Just Buy One Drive" — you scale one drive at a time, not matching sets. training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/b2c3d4e5f6a7 linux-data-hoarding easy snapraid, parity What is SnapRAID and how does it differ from real-time RAID? SnapRAID is a snapshot parity tool created by Andrea Mazzoleni (2011, GPLv3). Unlike real-time RAID, SnapRAID computes parity on a schedule (typically daily via cron). Between syncs, newly added files have NO parity protection.\n\nThis makes it ideal for write-once-read-many workloads (media libraries) but wrong for databases or VMs with high write churn. training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/c3d4e5f6a7b8 linux-data-hoarding easy snapraid, config What is the most critical rule about the parity drive in SnapRAID? The parity drive must be at least as large as the largest data drive. If your biggest data drive is 12TB, the parity drive must be >= 12TB. SnapRAID stores one parity block per data block, so it needs capacity equal to the largest single drive. training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/d4e5f6a7b8c9 linux-data-hoarding medium snapraid, operations What does `snapraid sync` do, and when should you run it? `snapraid sync` reads all data drives, computes parity blocks, and writes them to the parity drive(s). Run it daily via cron. Between syncs, newly added files have zero parity protection.\n\nAlways run `snapraid diff` first (or use snapraid-runner) to check for unexpected mass deletions before syncing. training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/e5f6a7b8c9d0 linux-data-hoarding medium snapraid, operations What does `snapraid scrub` do and what is the default scrub percentage? `snapraid scrub` verifies data integrity by reading data blocks and comparing checksums. The default scrub verifies ~8% of data per run. Running weekly, this covers all data roughly once every 12 weeks.\n\nUse `snapraid scrub -p 100` for a full verification, but only when needed — it reads every block on every drive. training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/f6a7b8c9d0e1 linux-data-hoarding medium snapraid, recovery How do you recover files from a failed drive using SnapRAID? 1. Mount the replacement drive at the same mount point\n2. Run: `snapraid fix -d d3` (replace d3 with the failed drive label)\n3. Verify: `snapraid check -d d3`\n4. Update parity: `snapraid sync`\n\nSnapRAID reconstructs files from parity data on the remaining drives. Expected time: 4-12 hours for a full drive. training/library/topics/linux-data-hoarding/street_ops.md linux-data-hoarding/a7b8c9d0e1f2 linux-data-hoarding hard snapraid, parity How many parity levels does SnapRAID support, and what does each level provide? SnapRAID supports 1 through 6 parity levels (configured as parity, 2-parity, 3-parity, etc.). Each level adds tolerance for one additional simultaneous disk failure:\n\n1-parity = survive 1 failure (like RAID5)\n2-parity = survive 2 failures (like RAID6)\n3-parity = survive 3 failures (like RAID-Z3)\n4-6 parity = for very large arrays (20+ disks)\n\nRule of thumb: 1 parity per 4 data disks for home use. training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/b8c9d0e1f2a3 linux-data-hoarding easy snapraid, automation What is snapraid-runner and why use it instead of bare cron? snapraid-runner (by Chronial) is a Python wrapper that adds safety checks to automated SnapRAID syncs. Key feature: it runs `snapraid diff` first and aborts if deletions exceed a configurable threshold (deletethreshold). This prevents parity destruction when a drive fails to mount and SnapRAID sees all its files as "deleted."\n\nGitHub: github.com/Chronial/snapraid-runner training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/c9d0e1f2a3b4 linux-data-hoarding easy filesystem, ext4 Why is ext4 the default filesystem choice for SnapRAID data drives? ext4 is boring, reliable, and universally supported. It has mature fsck recovery tools, works on every Linux distro, and has decades of battle testing. It lacks checksums and snapshots, but SnapRAID provides checksumming and parity externally.\n\nMnemonic: "ext4 = Toyota Corolla" — not exciting, but it starts every morning. training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/d0e1f2a3b4c5 linux-data-hoarding medium filesystem, xfs When should you choose XFS over ext4 for data drives? Choose XFS when storing predominantly large files (4K video, ISOs, disk images). XFS has a better allocator for large sequential I/O and supports reflink copies (cp --reflink). XFS scales to 8 EiB volumes.\n\nDownside: XFS cannot be shrunk (only grown). Historically fragile on power loss, but v5 format (default since 2014) fixed most issues. training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/e1f2a3b4c5d6 linux-data-hoarding hard filesystem, btrfs Why should you never use btrfs RAID5 or RAID6 for data you care about? btrfs RAID5/6 has an unfixed write hole bug. If power is lost during a write, parity can become inconsistent with data. On the next scrub, btrfs may "fix" good data with bad parity — making corruption worse. The kernel now warns when creating RAID5/6 profiles.\n\nSafe btrfs options: RAID1 (mirroring) or RAID10. For parity protection, use SnapRAID externally instead. training/library/topics/linux-data-hoarding/footguns.md linux-data-hoarding/f2a3b4c5d6e7 linux-data-hoarding hard filesystem, zfs What are the pros and cons of ZFS for data hoarding compared to the JBOD+mergerfs+SnapRAID approach? ZFS pros: CoW, built-in checksums, RAID-Z levels, ARC cache, zfs send/recv for backups, proven track record.\n\nZFS cons: Memory hungry (~1GB per TB rule of thumb), cannot easily add single drives to an existing pool, kernel module not in mainline Linux, no fsck equivalent (corrupt pool = data loss).\n\nKey: ZFS is a different paradigm — it replaces mergerfs+SnapRAID entirely. Don't mix them. training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/a3b4c5d6e7f8 linux-data-hoarding easy backup, borg What are BorgBackup's key features for data hoarding backups? BorgBackup (2015, BSD-3-Clause, Python+C): content-defined chunking deduplication, compression (lz4/zstd/zlib/lzma), AES-256-CTR encryption, and append-only repos (ransomware protection).\n\nKey commands: `borg init`, `borg create`, `borg prune`, `borg check`.\n\nChoose borg when: local/SFTP targets, want max compression, need append-only repos. training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/b4c5d6e7f8a9 linux-data-hoarding easy backup, restic What makes restic different from BorgBackup? restic (2015, BSD-2-Clause, Go): single static binary, native multi-backend support (S3, B2, Azure, GCS, SFTP, rclone), always-on AES-256 encryption, lock-free concurrent backups.\n\nChoose restic when: backing up to cloud storage (S3/B2), want zero dependencies, need multi-platform support. training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/c5d6e7f8a9b0 linux-data-hoarding medium backup, rclone What is rclone's crypt overlay and why is it important for offsite backups? rclone's crypt overlay provides client-side encryption as a transparent layer on any of rclone's 70+ storage backends. Data is encrypted locally before upload. The crypt remote wraps another remote, so `rclone sync /data b2-crypt:backups/` encrypts on-the-fly.\n\nIMPORTANT: rclone is a sync/transfer tool, NOT a backup tool. It has no versioning or deduplication. Pair with borg/restic for those features. training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/d6e7f8a9b0c1 linux-data-hoarding easy integrity, smart What did Google's study reveal about SMART's ability to predict drive failures? Google's 2007 study of 100,000+ drives found that 36% of failed drives showed ZERO SMART warnings beforehand. SMART catches gradual degradation (sector reallocation) but misses sudden failures (head crashes, PCB failures, firmware bugs).\n\nLesson: Use SMART as an early warning system, not a crystal ball. Always have parity + backups as additional layers. training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/e7f8a9b0c1d2 linux-data-hoarding medium integrity, smart Which 5 SMART attributes are most important to monitor for drive health? The big five: (1) Reallocated_Sector_Ct (#5) — bad sectors remapped, >0 is a concern; (2) Reported_Uncorrect (#187) — uncorrectable errors; (3) Command_Timeout (#188) — controller communication failures; (4) Current_Pending_Sector (#197) — sectors awaiting reallocation; (5) Offline_Uncorrectable (#198) — sectors failing offline tests.\n\nCheck: `smartctl -A /dev/sdX` training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/f8a9b0c1d2e3 linux-data-hoarding medium integrity, burn-in What is drive burn-in and why do data hoarders do it before trusting new drives? Burn-in tests a new drive before putting data on it, catching infant mortality failures (drives that fail in the first weeks). Steps:\n1. `smartctl -t long /dev/sdX` (SMART extended test, 8-24h)\n2. `badblocks -wsv -b 4096 /dev/sdX` (destructive write+read test)\n3. Check SMART attributes after — any non-zero Reallocated/Pending sectors = RMA\n\nDrives passing burn-in are statistically more reliable. training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/a9b0c1d2e3f4 linux-data-hoarding easy integrity, par2 What is par2 and what problem does it solve? par2 (Parchive v2) uses Reed-Solomon error correction to create recovery blocks for files. Originally created for Usenet transfers (2001-2002, by Tobias Rieper, Stefan Wehlus, and Howard Fukada).\n\nUse case: `par2 create -r10 archive.par2 *.tar.gz` creates 10% redundancy. If files are damaged, `par2 repair archive.par2` can reconstruct them. Ideal for cold storage archives and long-term preservation. training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/b0c1d2e3f4a5 linux-data-hoarding medium dedup, jdupes How do jdupes, fdupes, and rdfind compare for duplicate detection? jdupes: fastest (7x faster than fdupes), C, supports hardlink/softlink/delete modes, hash-based, by Jody Bruchon.\nfdupes: the original (1999, by Adrian Lopez), simpler interface, MD5 + byte-by-byte comparison.\nrdfind: C++, ranking-based dedup, O(N log N) time.\n\nAll match only 100% identical files (no fuzzy matching). jdupes is the standard choice for data hoarding. training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/c1d2e3f4a5b6 linux-data-hoarding easy drives, management What is the advantage of using /dev/disk/by-id/ instead of /dev/sdX in fstab? /dev/sdX device names (sda, sdb, sdc) can change between reboots depending on detection order. /dev/disk/by-id/ paths contain the drive model and serial number, creating stable identifiers that survive reboots, cable swaps, and controller changes.\n\nExample: /dev/disk/by-id/ata-WDC_WD120EMFZ-11A6JA0_SERIAL-part1\n\nAlternative: /dev/disk/by-uuid/ (filesystem UUID, also stable). training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/d2e3f4a5b6c7 linux-data-hoarding medium drives, spindown How do you spin down idle drives to save power, and which tools handle it? Two approaches:\n1. hdparm -S 242 /dev/sdX (value * 5 seconds, 242 = 1 hour standby timeout)\n2. hd-idle -i 600 /dev/sdX (600 seconds idle before spindown, more reliable for USB/some SATA)\n\nSpindown reduces power (~8W active vs ~0.5W standby per drive) and wear on infrequently accessed archive drives. Caveat: frequent spin-up/down cycles also cause wear — balance idle timeout with access patterns. training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/e3f4a5b6c7d8 linux-data-hoarding easy stack, perfect-media-server What is the "Perfect Media Server" stack and who created it? The Perfect Media Server stack was popularized by Alex Kretzschmar (ironicbadger, host of the Self-Hosted podcast). It combines: mergerfs (union filesystem), SnapRAID (snapshot parity), ext4/XFS (per-drive filesystems), Docker (containers for media apps), rclone (cloud backup), and smartd (monitoring).\n\nSite: perfectmediaserver.com training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/f4a5b6c7d8e9 linux-data-hoarding hard footgun, snapraid What happens if a drive fails to mount before running `snapraid sync`? If a data drive doesn't mount, its mount point is empty. SnapRAID sees all files on that drive as "deleted" and syncs parity accordingly — destroying the parity protection for those files. This is the most dangerous SnapRAID failure mode.\n\nPrevention: (1) Use snapraid-runner with deletethreshold, (2) Check `df -h /mnt/disk*` before sync, (3) Write a pre-sync script that verifies all drives are mounted. training/library/topics/linux-data-hoarding/footguns.md linux-data-hoarding/a5b6c7d8e9f0 linux-data-hoarding medium footgun, backup Why is "backup on the same drive as source" not actually a backup? A backup on the same physical drive protects against accidental deletion (maybe) but not against drive failure, ransomware affecting the whole filesystem, fire, or theft. Common mistakes: borg repo in /mnt/disk1/backups backing up /mnt/disk1/data, or restic repo on the same mergerfs pool.\n\nThe 3-2-1 rule requires: 3 copies, 2 different media types, 1 offsite. Same-drive copies satisfy none of these. training/library/topics/linux-data-hoarding/footguns.md linux-data-hoarding/b6c7d8e9f0a1 linux-data-hoarding hard footgun, encryption What is the circular dependency problem with encryption key storage? If your encryption keys (LUKS, borg repokey, rclone crypt password) are stored only on the encrypted system itself, losing that system means losing access to all backups too.\n\nCommon traps: passphrase in a password manager, password manager backup on the encrypted drive; borg repokey stored only in the repo; rclone.conf with crypt passwords on the encrypted volume.\n\nFix: Store keys in 2+ independent locations (printed paper in safe + password manager with separate backup). training/library/topics/linux-data-hoarding/footguns.md linux-data-hoarding/c7d8e9f0a1b2 linux-data-hoarding medium fstab, mount-options Why are noatime and nofail essential mount options for data hoarding drives? noatime: prevents updating access time metadata on every read, eliminating unnecessary writes. Critical for media streaming (constant reads) and SnapRAID (fewer changes to sync).\n\nnofail: if a drive fails or disconnects, boot continues instead of hanging. You get a degraded array instead of an unbootable server.\n\nAlways use: defaults,noatime,nofail for data drives. training/library/topics/linux-data-hoarding/footguns.md linux-data-hoarding/d8e9f0a1b2c3 linux-data-hoarding medium snapraid, content Why must SnapRAID content files be stored on multiple different drives? Content files are the checksum database — they store hashes for every file in the array. If all content file copies are lost, SnapRAID cannot verify data integrity or perform recovery. Store at least 2 copies on different physical drives (e.g., one on a data drive, one on /var).\n\nLosing content files while also losing a data drive = unrecoverable data loss. training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/e9f0a1b2c3d4 linux-data-hoarding hard snapraid, config What is SnapRAID split parity (v11.0+) and when would you use it? Split parity allows a single parity level to span multiple smaller drives using comma-separated paths: `parity /mnt/p1/snap.parity,/mnt/p2/snap.parity`. The next file starts growing when the previous one fills up.\n\nUse case: you have two 8TB drives but your largest data drive is 12TB. Split parity combines them into one 16TB parity target, satisfying the "parity >= largest data drive" requirement. training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/f0a1b2c3d4e5 linux-data-hoarding easy media, tools What are the core tools in the *arr stack and what does each do? Sonarr: TV show management + download automation (monitors RSS, renames, organizes)\nRadarr: Movie management (Sonarr fork, same pattern for movies)\nLidarr: Music management (same architecture)\nProwlarr: Indexer management (feeds search results to Sonarr/Radarr/Lidarr)\n\nAll run as Docker containers pointed at your mergerfs mount. They handle media lifecycle; the storage stack handles durability. training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/a1b2c3d4e5a6 linux-data-hoarding medium backup, strategy What is the correct relationship between SnapRAID parity and backups? SnapRAID parity is NOT backup. Parity protects against disk failure (hardware). Backup protects against deletion, ransomware, fire, and theft (everything else). You need both.\n\nParity = survive a drive dying\nBackup = survive rm -rf, ransomware, house fire\n\nMinimum: SnapRAID for parity + borg/restic to a separate drive + rclone to offsite cloud. training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/b2c3d4e5a6b7 linux-data-hoarding hard backblaze, statistics What does Backblaze's public drive data tell us about failure rates? Backblaze publishes quarterly stats from 290,000+ drives. Key findings:\n- 2024 AFR (Annualized Failure Rate): 1.57% overall\n- 2025 AFR: dropped to 1.36%\n- Failure rates vary dramatically by model and age\n- High-capacity drives (20TB+) show lower AFR (~0.77% in Q4 2024)\n- Some 12TB models exceed 5% AFR\n\nData is publicly downloadable at backblaze.com/cloud-storage/resources/hard-drive-test-data training/library/topics/linux-data-hoarding/primer.md linux-data-hoarding/c3d4e5a6b7c8 linux-data-hoarding medium drives, new-drive What are the steps to add a new drive to an existing data hoarding array? 1. Burn-in (SMART long test + badblocks): ~24 hours\n2. Format: mkfs.ext4 -L diskN /dev/sdX\n3. Mount: add to fstab with noatime,nofail, mount at /mnt/diskN\n4. Add to mergerfs (live: xattr on .mergerfs control file, or edit fstab and remount)\n5. Add to snapraid.conf (data dN /mnt/diskN/, plus content file)\n6. Run snapraid sync\n\nmergerfs automatically starts using the new drive for new files based on create policy (e.g., mfs = most free space). training/library/topics/linux-data-hoarding/street_ops.md --- ## Wiki Navigation ### Related Content - [Linux Data Hoarding](../../../../library/topics/linux-data-hoarding/index.md) (Topic Pack, L2) — Linux Data Hoarding