Skip to content

Lab 18: Zero-Downtime Migration

Field Value
Tier 4 — Advanced
Estimated Time 90 minutes
Prerequisites k3s cluster, PostgreSQL basics
Auto-Grade Yes

Scenario

The database team needs to migrate the primary PostgreSQL database from version 15 to version 16. The application handles 500 requests per second and any downtime costs real money. The migration must be zero-downtime: the application must continue serving traffic throughout the migration. You need to run the old and new databases simultaneously, use logical replication to sync data, perform a cutover, and verify data integrity.

This lab simulates the migration using two PostgreSQL instances running as pods. You will set up replication from old to new, seed test data, verify it replicates, perform the cutover by updating the application's database connection, and verify data integrity by comparing row counts and checksums.

Objectives

  • Deploy PostgreSQL 15 (source) and PostgreSQL 16 (target) in namespace lab-migration
  • Seed the source database with test data (3 tables, 100+ rows each)
  • Configure logical replication from source to target
  • Verify data is replicating (row counts match)
  • Perform cutover: update the app to point to the new database
  • Verify data integrity post-migration (checksums match)
  • Write migration report to /tmp/lab-migration/migration-report.txt

Setup

./setup.sh

Deploys the source PostgreSQL 15 instance and a sample application.

Hints

Hint 1: PostgreSQL logical replication On the source: `ALTER SYSTEM SET wal_level = logical;` and restart. Create a publication: `CREATE PUBLICATION migration_pub FOR ALL TABLES;` On the target: `CREATE SUBSCRIPTION migration_sub CONNECTION '...' PUBLICATION migration_pub;`
Hint 2: Schema migration Export the schema first: `pg_dump --schema-only -U postgres source_db | psql -U postgres target_db`. Logical replication copies data but not schema.
Hint 3: Verifying replication Compare row counts: run `SELECT count(*) FROM ` on both databases. Compare checksums: `SELECT md5(string_agg(row::text, '' ORDER BY id)) FROM
`.
Hint 4: Cutover procedure 1. Pause writes (or accept brief inconsistency window) 2. Wait for replication lag to reach zero 3. Update app's DB connection string (ConfigMap or Secret) 4. Restart app pods to pick up new config 5. Verify app works against new database
Hint 5: Rollback plan Keep the old database running until the new one is verified. If anything goes wrong, switch the connection string back. This is your safety net.

Grading

./grade.sh

Solution

See the solution/ directory for the complete migration procedure.