--- tags: - security - l1 - flashcard-deck - disaster-recovery --- [Portal](../../../../library/portal/index.md) | **Level:** [L1: Foundations](../../../../library/portal/levels.md) | **Topics:** [Disaster Recovery](../../../../library/portal/topics.md) | **Domain:** Security id category difficulty tags question answer source_path disaster-recovery/4ac59e5b7b39 disaster-recovery easy disaster-recovery, rto, rpo What do RPO and RTO mean? RPO (Recovery Point Objective) is how much data you can afford to lose, measured as the time window before the disaster. RTO (Recovery Time Objective) is how long you can be down before business impact, measured as the time window after the disaster until service is restored. training/library/topics/disaster-recovery/primer.md disaster-recovery/5a724f492f77 disaster-recovery easy disaster-recovery, backup, 3-2-1 What is the 3-2-1 backup rule? Keep 3 copies of your data, on 2 different storage media, with 1 copy offsite. For example: production database (live), local backup server (Borg repo on ZFS), and offsite S3 bucket (restic to Backblaze B2). training/library/topics/disaster-recovery/primer.md disaster-recovery/885e960c1c71 disaster-recovery easy disaster-recovery, testing, restore Why is the statement "if you have never restored from your backups, you do not have backups" important? The most common DR failure is backing up but never testing restores. Backups can silently fail, produce corrupt archives, or miss critical files. Only a verified restore proves the backup works. Schedule monthly restore drills. training/library/topics/disaster-recovery/primer.md disaster-recovery/9115f5fb6466 disaster-recovery medium disaster-recovery, borg, deduplication How does Borg backup achieve space efficiency through deduplication? Borg splits data into chunks and stores each unique chunk only once. If a 50 GB backup changes only 10 GB between runs, only the new 10 GB of changed chunks are stored. Three 50 GB logical backups might only use 80 GB of physical storage depending on the change rate. training/library/topics/disaster-recovery/primer.md disaster-recovery/9513a33b7c48 disaster-recovery medium disaster-recovery, restic, offsite How does restic differ from Borg and when would you choose it? Restic natively supports cloud backends (S3, B2, Azure, GCS) and is simpler to set up for offsite backups. Choose restic when your primary need is pushing backups to cloud storage. Both support deduplication and encryption, but Borg is faster for local or LAN-based backups. training/library/topics/disaster-recovery/primer.md disaster-recovery/d564781deb3f disaster-recovery medium disaster-recovery, tiers, strategy What are the four DR tiers and their corresponding RPO/RTO targets? Tier 1 (critical): RPO < 1 hour, RTO < 1 hour, uses real-time replication and hot standby. Tier 2 (important): RPO < 4 hours, RTO < 4 hours, uses frequent snapshots and warm standby. Tier 3 (standard): RPO < 24 hours, RTO < 24 hours, uses daily backups and cold restore. Tier 4 (archival): RPO < 7 days, RTO < 72 hours, uses weekly offsite backups. training/library/topics/disaster-recovery/primer.md disaster-recovery/4cc7145235a2 disaster-recovery medium disaster-recovery, runbook, structure What are the essential sections of a DR runbook? Classification (RPO, RTO, tier, last tested, owner), Dependencies (upstream and downstream services), Backup Details (tool, schedule, retention, location, encryption), Recovery Procedure (step-by-step restore, verification, DNS cutover, notifications), Failover Procedure (promote standby, redirect traffic, verify integrity), and Post-Recovery Checklist. training/library/topics/disaster-recovery/primer.md disaster-recovery/077cc5d357d5 disaster-recovery hard disaster-recovery, pitfalls, common-mistakes What are the most dangerous pitfalls in disaster recovery engineering? Backing up to the same disk as production (both die together), no encryption on offsite backups, forgetting to back up config alongside data (TLS certs, secrets), backup window exceeding backup interval (creating gaps), not versioning DR runbooks (referencing decommissioned servers), single point of failure in the backup pipeline, and retention too aggressive (3 days of backups when corruption happened 4 days ago). training/library/topics/disaster-recovery/primer.md disaster-recovery/3f66da6bcb5f disaster-recovery hard disaster-recovery, verification, automation Describe an automated backup verification script pattern. Extract the latest Borg archive to a temp directory. Verify critical files exist (e.g., etc/hosts, PostgreSQL PG_VERSION). Run pg_verifybackup if available. Check file sizes and counts against expected baselines. Output PASS/FAIL with details. Clean up the temp directory. Schedule this as a cron job or CI pipeline to run after each backup completes. training/library/topics/disaster-recovery/primer.md disaster-recovery/6afcd3fc4425 disaster-recovery hard disaster-recovery, offsite, replication What key decisions must be made when designing offsite backup replication? Bandwidth (large initial seeds may need physical media), encryption (always encrypt before leaving your network — Borg and restic do this by default), retention policy (offsite retention is usually longer than local), cost (S3 Standard vs Glacier vs B2 can differ by 10x), and restore speed (Glacier retrieval takes hours, which must be factored into your RTO). training/library/topics/disaster-recovery/primer.md disaster-recovery/7ba8e4c2d1f6 disaster-recovery medium disaster-recovery, rto-rpo, planning How do you determine appropriate RTO and RPO targets for a service? Start from business impact: revenue loss per hour of downtime, contractual SLAs, regulatory requirements, and customer tolerance. A payment system may need RPO < 1 minute and RTO < 15 minutes, while an internal wiki may tolerate RPO < 24 hours and RTO < 8 hours. Document these in the service's DR runbook and validate that your backup frequency and restore speed actually meet the targets through regular testing. disaster-recovery/8cb9f5d3e2a7 disaster-recovery easy disaster-recovery, testing, frequency How often should you test disaster recovery and what should each test cover? Minimum quarterly for critical services, annually for others. Tests should include: full restore from backup to a clean environment, verification that restored data is complete and consistent, timing the restore to confirm it meets RTO, testing failover to standby systems, and validating that runbooks are accurate and current. Document results and fix any gaps immediately. disaster-recovery/9dc0a6e4f3b8 disaster-recovery hard disaster-recovery, active-passive, active-active What is the difference between active-passive and active-active DR architectures? Active-passive: standby site receives replicated data but serves no traffic until failover. Lower cost, simpler, but risk of stale standby and longer failover time.\nActive-active: both sites serve traffic simultaneously with data replication between them. Faster failover (just remove one site from DNS/LB) but requires conflict resolution for writes and costs roughly 2x. Active-active is preferred for services with near-zero RTO requirements. disaster-recovery/0ed1b7f5a4c9 disaster-recovery medium disaster-recovery, database, replication What are the main database DR strategies and their RPO implications? Synchronous replication: RPO = 0 (zero data loss) but adds write latency. Asynchronous replication: RPO = replication lag (seconds to minutes), better performance. Periodic backups: RPO = backup interval (hours). Point-in-time recovery (WAL archiving for PostgreSQL, binlog for MySQL): RPO = seconds, restores to any moment. \nBest practice: combine async replication for fast failover with WAL archiving for point-in-time recovery. disaster-recovery/1fe2c8a6b5d0 disaster-recovery medium disaster-recovery, runbook, maintenance What makes a DR runbook effective and how do you keep it current? Effective runbooks have: exact commands (not "restore the database" but "pg_restore -d mydb -F c /backup/latest.dump"), contact info for escalation, decision trees for different failure modes, and verification steps after each action. Keep current by: updating after every DR test, requiring runbook review during on-call handoffs, versioning in git alongside the service code, and including a "last tested" date prominently at the top. disaster-recovery/2af3d9b7c6e1 disaster-recovery hard disaster-recovery, communication, plan What should a DR communication plan include? Internal notification chain (who is informed and in what order), customer communication templates (pre-written for common scenarios), status page update procedures, regulatory notification requirements (e.g., GDPR 72-hour breach notification), partner/vendor contact list, and executive briefing cadence during extended outages. Pre-drafted templates save critical minutes during an actual disaster when stress is high. disaster-recovery/3ba4e0c8d7f2 disaster-recovery hard disaster-recovery, cloud, region-failover What are the key considerations for cloud region failover DR? Data replication lag between regions (async replication = potential data loss). DNS TTL (low TTLs enable faster failover but increase DNS query load). Stateful services (databases, caches) are the hardest to fail over. Infrastructure-as-code ensures the standby region can be provisioned identically. Test regularly because cross-region permissions, quotas, and AMI availability often differ. Cost: running warm standby in a second region roughly doubles infrastructure spend. disaster-recovery/69bd1379ede9 disaster-recovery hard disaster-recovery, split-brain, active-active What causes split-brain in active-active database failover and how do you resolve it? Split-brain occurs when both nodes believe they are primary after a network partition, accepting conflicting writes. Resolution strategies: quorum-based fencing (STONITH), witness/arbiter nodes that break ties, or application-level conflict resolution (last-write-wins, CRDTs). Prevention is better — use proper fencing before failover. training/library/topics/disaster-recovery/primer.md disaster-recovery/bdcaa10a8c88 disaster-recovery medium disaster-recovery, pitr, snapshot, trade-offs What is the trade-off between point-in-time recovery (PITR) and snapshot restore? Snapshots are fast to restore but only recover to the snapshot moment — you lose all changes after it. PITR replays WAL/binlog from a base backup to any point in time, recovering more data but taking longer. \nBest practice: use snapshots for fast RTO, PITR for precise RPO. training/library/topics/disaster-recovery/primer.md disaster-recovery/46e02063828b disaster-recovery medium disaster-recovery, chaos-dr, testing Why should you test disaster recovery plans with controlled failure injection? DR plans that are never tested tend to fail when needed — stale credentials, changed schemas, missing runbook steps. Chaos-DR exercises (killing a primary database, simulating region outage) validate the plan end-to-end and expose gaps before a real disaster. Schedule quarterly at minimum. training/library/topics/disaster-recovery/primer.md --- ## Wiki Navigation ### Related Content - [Disaster Recovery & Backup Engineering](../../../../library/topics/disaster-recovery/index.md) (Topic Pack, L2) — Disaster Recovery