managing-database-migrations

# Database Migration Manager

## Overview

Create, validate, and execute database schema migrations with full rollback support across PostgreSQL, MySQL, and MongoDB.

## Prerequisites

- Database credentials with DDL permissions (CREATE, ALTER, DROP TABLE)
- Migration framework installed and configured (Flyway, Alembic, Prisma, Knex, or raw SQL versioning)
- Version control for migration files (Git repository)
- Access to a staging database matching production schema for testing migrations
- `psql` or `mysql` CLI for executing and verifying migrations
- Current schema baseline documented or captured via `pg_dump --schema-only`

## Instructions

1. Capture the current schema state before making changes. Run `pg_dump --schema-only -f schema_before.sql` (PostgreSQL) or `mysqldump --no-data > schema_before.sql` (MySQL) to create a reference point.

2. Define the desired schema change clearly: specify table name, column additions/removals/modifications, constraint changes, and index updates. Document whether the change is additive (safe) or destructive (requires data migration).

3. Generate a versioned migration file following the framework's naming convention:
   - Flyway: `V20240115_001__add_status_column_to_orders.sql`
   - Alembic: `alembic revision --autogenerate -m "add status column to orders"`
   - Prisma: Edit `schema.prisma` then `npx prisma migrate dev --name add_status_to_orders`
   - Knex: `npx knex migrate:make add_status_to_orders`

4. Write the UP migration (forward change) with these safety practices:
   - Add new columns as nullable first, then backfill, then set NOT NULL
   - Use `IF NOT EXISTS` for CREATE operations to make migrations idempotent
   - Add explicit transaction wrapping: `BEGIN; ... COMMIT;`
   - Include comments explaining the business reason for each change

5. Write the DOWN migration (rollback) that exactly reverses the UP migration. For column additions, the DOWN drops the column. For table renames, the DOWN renames back. For data transformations, the DOWN must restore original data (store it in a backup column or table if needed).

6. Validate the migration on staging by running the full migration sequence:
   - Apply the UP migration and verify schema matches expectations
   - Run the application test suite against the migrated schema
   - Apply the DOWN migration and verify the schema returns to its original state
   - Re-apply the UP migration to confirm idempotency

7. For zero-downtime migrations on production, follow the expand-contract pattern:
   - Phase 1 (expand): Add new column/table without removing old ones. Deploy application code that writes to both old and new.
   - Phase 2 (migrate): Backfill new column/table from old data in batches.
   - Phase 3 (contract): Deploy application code that reads only from new. Drop old column/table in a future migration.

8. Handle large table migrations (>10M rows) with online DDL tools: `pg_repack` for PostgreSQL, `pt-online-schema-change` for MySQL, or `gh-ost` for MySQL. These tools create a shadow table, replicate changes, then swap atomically.

9. Update the migration history table and verify the migration version matches expectations. Run `flyway info` or `alembic current` to confirm.

10. Document the migration in a changelog with: migration version, description, tables affected, estimated execution time, rollback procedure, and any required application deployments.

## Output

- **Migration files** (UP and DOWN) in the target framework's format
- **Pre-flight validation script** checking prerequisites before migration execution
- **Data backfill scripts** for non-nullable column additions on existing tables
- **Rollback runbook** with step-by-step instructions for reverting in production
- **Migration changelog entry** documenting the change for team reference

## Error Handling

| Error | Cause | Solution |
|-------|-------|---------|
| Migration fails with lock timeout | Long-running queries blocking DDL locks on the target table | Set `lock_timeout = '5s'` to fail fast; retry during low-traffic period; use `pg_repack` for lock-free operations |
| Column cannot be dropped because of dependent views | Views or materialized views reference the column being removed | Drop or recreate dependent views first; use `CASCADE` only after verifying all dependents are acceptable to drop |
| NOT NULL constraint violation during migration | Existing rows have NULL values in a column being made NOT NULL | Add a backfill step: `UPDATE table SET column = default_value WHERE column IS NULL` before adding NOT NULL |
| Migration version conflict | Two developers created migrations with the same version number | Use timestamp-based versioning; resolve by renaming one migration and re-running |
| Rollback fails because data was inserted after UP migration | New data in added columns has no place to go in the old schema | Design rollback to handle new data (backup column, log discarded data); accept that some rollbacks are data-lossy |

## Examples

**Adding a status enum column to an orders table**: Generate a migration that: (1) creates the enum type `CREATE TYPE order_status AS ENUM ('pending', 'shipped', 'delivered')`, (2) adds column `ALTER TABLE orders ADD COLUMN status order_status`, (3) backfills `UPDATE orders SET status = 'delivered' WHERE shipped_at IS NOT NULL`, (4) sets NOT NULL `ALTER TABLE orders ALTER COLUMN status SET NOT NULL`. Rollback drops the column and enum type.

**Splitting a monolith users table into users and profiles**: Expand phase adds `profiles` table and triggers to sync data. Migrate phase copies existing profile data in batches of 10,000 rows. Contract phase drops profile columns from users table after application code is updated.

**Renaming a column without downtime**: Create a migration that adds the new column, adds a trigger to sync writes between old and new columns, backfills existing data, deploys application code using the new column name, then drops the old column and trigger in a follow-up migration.

## Resources

- Flyway documentation: https://documentation.red-gate.com/fd
- Alembic tutorial: https://alembic.sqlalchemy.org/en/latest/tutorial.html
- Prisma Migrate: https://www.prisma.io/docs/orm/prisma-migrate
- pg_repack (lock-free table rewrite): https://reorg.github.io/pg_repack/
- gh-ost (GitHub online schema change): https://github.com/github/gh-ost