planning-disaster-recovery

# Planning Disaster Recovery

## Overview

Design disaster recovery (DR) plans for cloud infrastructure covering RTO/RPO requirements, multi-region failover, data replication, and automated recovery procedures. Generate runbooks, Terraform for standby infrastructure, and automated failover scripts for databases, compute, and networking.

## Prerequisites

- Complete inventory of production infrastructure components and dependencies
- Defined RTO (Recovery Time Objective) and RPO (Recovery Point Objective) per service tier
- Cloud provider CLI authenticated with permissions for multi-region resource management
- Cross-region networking configured (VPC peering, Transit Gateway, or VPN)
- Backup and replication mechanisms already in place or planned

## Instructions

1. Catalog all production services with their criticality tier (Tier 1: < 15 min RTO, Tier 2: < 1 hour, Tier 3: < 24 hours)
2. Map dependencies between services to identify single points of failure and cascading failure paths
3. Design the DR strategy per tier: active-active for Tier 1, pilot light or warm standby for Tier 2, backup-restore for Tier 3
4. Generate Terraform for standby region infrastructure: VPC, subnets, security groups, and scaled-down compute
5. Configure database replication: RDS cross-region read replicas, DynamoDB global tables, or Cloud SQL cross-region replicas
6. Set up DNS failover using Route 53 health checks, Cloud DNS routing policies, or global load balancers
7. Create automated failover scripts: promote read replica to primary, update DNS records, scale up standby compute
8. Document the DR runbook with step-by-step procedures, responsible parties, and communication plans
9. Schedule quarterly DR drills: simulate region failure, execute the runbook, measure actual RTO/RPO, and document gaps
10. Set up monitoring for replication lag, backup freshness, and standby infrastructure health

## Output

- DR plan document with service tiers, RTO/RPO targets, and recovery procedures
- Terraform modules for standby region infrastructure
- Automated failover scripts (database promotion, DNS switching, compute scaling)
- DR drill checklist and post-drill assessment template
- Monitoring dashboards for replication lag and backup status

## Error Handling

| Error | Cause | Solution |
|-------|-------|---------|
| `Replication lag exceeds RPO` | Network throughput insufficient or write volume too high | Increase replication instance size, enable compression, or implement write throttling during peak |
| `DNS failover not triggering` | Health check misconfigured or TTL too high | Verify health check endpoint returns proper status; reduce DNS TTL to 60 seconds before drill |
| `Standby database promotion failed` | Replication broken or standby in inconsistent state | Check replication status; if broken, restore from latest snapshot and re-establish replication |
| `Insufficient capacity in DR region` | Instance types unavailable in standby region | Pre-provision reserved capacity in DR region or use multiple instance type options |
| `Application cannot connect after failover` | Connection strings hardcoded to primary region endpoints | Use DNS-based endpoints (CNAME/Route 53) instead of direct IPs; parameterize region in config |

## Examples

- "Create a disaster recovery plan for a 3-tier web application on AWS with < 15 minute RTO for the API layer and < 1 hour for batch processing."
- "Generate Terraform for a warm standby in us-west-2 with RDS cross-region read replica, scaled-down ECS cluster, and Route 53 failover routing."
- "Design a DR drill that simulates primary region failure, executes automated failover, and validates data integrity in the standby region."

## Resources

- AWS Disaster Recovery: https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/
- GCP DR planning: https://cloud.google.com/architecture/dr-scenarios-planning-guide
- Azure Business Continuity: https://learn.microsoft.com/en-us/azure/reliability/
- DR strategy patterns: https://aws.amazon.com/blogs/architecture/disaster-recovery-dr-architecture-on-aws-part-i/