managing-database-recovery

# Database Recovery Manager

## Overview

Plan and execute database backup and recovery procedures for PostgreSQL and MySQL, including point-in-time recovery (PITR), logical and physical backups, WAL archiving, and disaster recovery testing. This skill covers the full backup lifecycle from configuration through automated verification, ensuring Recovery Point Objective (RPO) and Recovery Time Objective (RTO) targets are met.

## Prerequisites

- Database superuser or replication-role credentials
- Backup storage destination (local disk, NFS mount, S3, GCS, or Azure Blob)
- `pg_basebackup`, `pg_dump`, `pg_restore` (PostgreSQL) or `mysqldump`, `xtrabackup` (MySQL)
- `tar`, `rsync`, or `aws s3` CLI for backup transfer and storage
- WAL archiving configured for PITR (PostgreSQL: `archive_mode = on`, `archive_command`)
- Sufficient storage for backup retention (estimate 2-3x database size for full + incremental)

## Instructions

1. Assess the current backup situation by checking existing backup configurations. For PostgreSQL: verify `archive_mode`, `archive_command`, and `wal_level` in `postgresql.conf`. For MySQL: check if binary logging is enabled with `SHOW VARIABLES LIKE 'log_bin'`.

2. Define RPO and RTO targets based on business requirements:
   - RPO (acceptable data loss): determines backup frequency and WAL archiving interval
   - RTO (acceptable downtime): determines backup type and recovery procedure complexity
   - Typical targets: RPO < 1 hour (WAL archiving), RTO < 30 minutes (physical backup restore)

3. Configure WAL archiving for PostgreSQL PITR:
   - Set `wal_level = replica` and `archive_mode = on`
   - Configure `archive_command = 'test ! -f /archive/%f && cp %p /archive/%f'` (or use pgBackRest/WAL-G for S3)
   - Verify archiving works: `SELECT * FROM pg_stat_archiver`
   - For MySQL, enable binary logging: `log_bin = mysql-bin`, `binlog_format = ROW`

4. Create a full physical backup using `pg_basebackup -D /backups/base -Ft -z -P` (PostgreSQL) or `xtrabackup --backup --target-dir=/backups/full` (MySQL). Physical backups are faster to restore than logical backups for databases larger than 10GB.

5. Create logical backups for portability and selective restoration: `pg_dump -Fc -f database.dump dbname` (PostgreSQL) or `mysqldump --single-transaction --routines --triggers dbname > database.sql` (MySQL).

6. Upload backups to remote storage for disaster recovery: `aws s3 cp /backups/base.tar.gz s3://backup-bucket/postgres/$(date +%Y%m%d)/` with server-side encryption enabled. Implement a retention policy (e.g., daily backups for 30 days, weekly for 90 days, monthly for 1 year).

7. Test recovery by restoring to a separate server or container:
   - Restore physical backup: `pg_restore -d testdb database.dump` or untar base backup and start PostgreSQL
   - For PITR: restore base backup, copy WAL files to `pg_wal`, create `recovery.signal` with `recovery_target_time = '2024-01-15 14:30:00'`
   - Verify data integrity by running application test suite against restored database
   - Measure actual recovery time to validate RTO target

8. Automate backup verification with a daily cron job that: takes backup, restores to a test instance, runs integrity checks (`pg_catalog.pg_class` row counts, checksum verification), and sends a success/failure notification.

9. Document the recovery runbook with exact commands for each recovery scenario: full database restore, PITR to a specific timestamp, single table restore, and cross-region failover.

10. Schedule monthly disaster recovery drills to verify the runbook works and the team can execute recovery within RTO targets.

## Output

- **Backup configuration files** for PostgreSQL (postgresql.conf changes, archive_command) or MySQL (my.cnf changes)
- **Backup scripts** (shell) for automated full and incremental backups with S3/GCS upload
- **Recovery runbook** with step-by-step commands for each recovery scenario
- **Backup verification scripts** that automate restore-and-check procedures
- **Retention policy configuration** with automated cleanup of old backups

## Error Handling

| Error | Cause | Solution |
|-------|-------|---------|
| WAL segment not found during PITR | Gap in WAL archiving due to archive_command failure | Check `pg_stat_archiver` for `last_failed_wal`; fix archive_command; consider WAL-G or pgBackRest for reliable archiving |
| `pg_basebackup` fails with "replication connection" error | Missing replication permissions or `max_wal_senders` exhausted | Grant REPLICATION role; increase `max_wal_senders`; add entry to `pg_hba.conf` for replication connections |
| Backup storage full | Retention policy not enforced or backup size grew unexpectedly | Implement automated cleanup script; compress backups with `gzip` or `zstd`; monitor storage usage with alerts at 80% |
| Recovery takes longer than RTO target | Database grew since RTO was last validated, or restore is I/O-bound | Use physical backups instead of logical; restore to SSD storage; parallelize restore with `pg_restore -j 4`; consider standby replica for faster failover |
| Restored database has corruption | Backup taken during crash or disk error | Enable `data_checksums` in PostgreSQL; verify backups with `pg_verifybackup`; run `ANALYZE` and `REINDEX` after restore |

## Examples

**Point-in-time recovery after accidental table drop**: At 14:30 a developer runs `DROP TABLE orders` on production. Recovery: restore the most recent physical backup (taken at 02:00), replay WAL files up to 14:29:59 using `recovery_target_time`, verify orders table is intact, then swap the recovered database into production. Total recovery time: 25 minutes for a 200GB database.

**Automated daily backup to S3 with verification**: A cron job runs at 02:00 UTC: takes `pg_basebackup`, compresses with `zstd`, uploads to S3 with server-side encryption, restores to a Docker container, runs row count checks against production, and sends a Slack notification with backup size and verification status.

**Cross-region disaster recovery drill**: Monthly exercise: restore the latest S3 backup to a different AWS region, replay WAL files to catch up, run the application test suite, measure total failover time, and document results. Target: full recovery in a different region within 1 hour.

## Resources

- PostgreSQL backup and recovery: https://www.postgresql.org/docs/current/backup.html
- pgBackRest (production backup tool): https://pgbackrest.org/
- WAL-G (WAL archiving to cloud storage): https://github.com/wal-g/wal-g
- Percona XtraBackup: https://docs.percona.com/percona-xtrabackup/
- MySQL point-in-time recovery: https://dev.mysql.com/doc/refman/8.0/en/point-in-time-recovery.html