Designing Resilient Multi-Region AWS Architectures: An SRE Handbook
No single AWS region is immune to outages. Whether it is a major fiber-optic cable cut, a utility power failure, or a corrupted API deployment at the cloud provider level, relying on a single region leaves your business vulnerable. For systems requiring 99.99% availability or higher, multi-region architecture is the gold standard.
However, moving to multi-region is not a simple switch. It introduces database consistency issues, traffic routing complexities, and increased costs. This handbook highlights SRE design patterns for multi-region resilience on AWS.
Multi-Region Strategies: Active-Passive vs. Active-Active
The first decision in multi-region design is traffic distribution:
| Strategy | Active-Passive | Active-Active |
|---|---|---|
| Description | One region handles all read/write traffic; the standby region is idle or read-only until failover. | Both regions actively accept read and write requests simultaneously. |
| RTO (Recovery Time) | Minutes (requires DNS updates and database promotion). | Seconds to sub-second (automated routing). |
| RPO (Recovery Point) | Low (some data replication lag). | Sub-second (requires active-active data syncing). |
| Complexity | Moderate. | Extremely High (requires conflict resolution). |
Data Replication: The Hardest Problem
Stateless applications are easy to run in multiple regions. The hard part is the data layer. SREs must design around the CAP theorem (Consistency, Availability, and Partition tolerance).
AWS provides excellent tools to simplify replication:
1. Amazon DynamoDB Global Tables
DynamoDB Global Tables provide a fully managed active-active database. It replicates writes across regions automatically, usually in less than a second.
- SRE Tip: DynamoDB uses a "last-writer-wins" strategy for conflict resolution based on timestamp. Design your application keys to minimize concurrent writes to the same record from different regions to avoid data loss.
2. Amazon Aurora Global Databases
For relational workloads, Aurora Global Databases replicate data from a primary cluster in one region to up to five secondary clusters in other regions.
- Latency: Under 1 second.
- Failover: In an outage, you must promote a secondary region to become the primary. SREs should automate this using AWS SDKs or Terraform, checking health checks and triggering the promotion programmatically.
Traffic Routing & Failover Orchestration
Routing users to the correct region requires smart DNS and edge configuration:
- AWS Route 53 Latency-Based Routing: Directs users to the AWS region that yields the lowest network latency. This improves speed for global users and spreads load.
- Route 53 Failover Routing (Active-Passive): Uses Route 53 health checks to monitor your primary region. If the primary endpoint fails, Route 53 automatically updates DNS records to route users to the backup region.
- AWS Global Accelerator: Utilizes the AWS global network to route TCP/UDP traffic. It provides static IP addresses that automatically failover to healthy endpoints globally without waiting for local DNS caches to clear.
The Threat of Split-Brain Scenarios
In active-passive designs, a "split-brain" scenario occurs when the primary region is healthy but network connectivity between the monitoring service and the primary region is lost. If the standby region is prematurely promoted, both regions will attempt to act as the primary, leading to massive database corruption and conflicting writes.
To prevent split-brain:
- Double-check health checks: Monitor health from multiple independent geographic perspectives before executing a database promotion.
- Implement manual or semi-automated confirmation: SRE tools should require validation of database connectivity failures before promoting primary nodes.
Summary
Building a multi-region AWS setup requires careful planning around data consistency and failover logic. By leveraging DynamoDB Global Tables, Aurora Global Database, and AWS Route 53 routing policies, you can design systems that remain fully operational even if an entire AWS region goes dark.