Active-Active Disaster Recovery for Financial Services

Active-active multi-region disaster recovery for a financial services firm, replacing an untested cold-standby arrangement with automated distributed routing and...

[Overview] Scope of the engagement

This anonymized Ampity engagement involved transforming a cold-standby disaster recovery arrangement into an active-active multi-region architecture for a financial services firm. The work covered application rearchitecting for multi-region operation, globally distributed database configuration, automated routing and chaos-engineering validation.

This case focuses on the architecture, implementation approach and operating responsibilities. It does not provide verified before-and-after measurements or a regulatory assessment.

[Context] Cold standby and its limits

The client operated a cold-standby disaster recovery environment: servers in a secondary data center powered off until needed, with data replicated through daily backups. The recovery procedure had never been fully tested under production conditions, and the estimated recovery time was based on team assessment rather than a recorded exercise.

Backup-based recovery depends on the last recoverable state, not simply the scheduled backup frequency. For a similar system, determine which acknowledged business operations a restore can reproduce and which would need reconciliation.

The review problem was whether the recovery arrangement could support the application's required operations after a failure. No regulator finding or external endorsement is asserted here.

[Diagnosis] Cold standby is not tested resilience

The account frames the problem as the difference between having a DR environment and having a proven recovery capability. An untested cold-standby environment provides unknown protection. The human steps required for cold-standby recovery (decision-making, communication, server activation, data restoration, validation) extend actual recovery time well beyond estimates derived from partial tests.

The application architecture assumed synchronous, local database access. Adding a second region without changing that assumption shifts the location of the failure risk rather than removing it.

[Decisions] Warm standby, active-passive or active-active

The account records that warm standby (infrastructure running but not serving traffic) was evaluated and rejected. The stated reasons are that it reduces but does not eliminate failover delay, that standby infrastructure is not exercised under real load and so still provides uncertain protection, and that it adds cost without meeting the target recovery requirement.

Active-passive (one active region, one ready standby) was rejected because it still requires a failover event with associated configuration-drift risk and a recovery window during which the application is unavailable.

The recorded choice was active-active operation, with regional deployments serving traffic rather than waiting to be activated. Traffic routing and data-write authority still have failure transitions. Active traffic exercises part of the system, but does not prove spare capacity, dependency parity or readiness for every regional failure.

This is not a claim that active-active guarantees zero data loss or zero user-visible impact in every failure scenario. The actual behavior depends on the specific replication configuration and the failure mode encountered.

[Design] Distributed database and traffic routing

The account names CockroachDB and describes a distributed commit path for critical records. It does not establish the product version, replica placement or commit-acknowledgment configuration. For reuse, review the configured quorum rules: which partitions may acknowledge writes, which must stop, and how clients resolve an ambiguous response?

Global routing uses location and health information to direct traffic between regional deployments. The account does not provide a measured end-to-end failover time.

Compare the acknowledged-write completion path with the application's latency budget. Placement and acknowledgment rules affect the tradeoff; a synchronous-replication label alone does not establish acceptable latency or the recovery boundary.

For a similar system, distinguish a database commit from a completed business operation. An external settlement, notification or partner action may remain uncertain even when a database record has committed.

[Approach] Phased activation with chaos engineering

The account describes a phased implementation. Infrastructure was deployed in the second region first. Database replication was configured and validated. Active traffic was then enabled to both regions. Chaos engineering game days were conducted to validate failover behavior by intentionally failing components and observing the effect on the active system.

Regular game days are described as an ongoing mechanism for maintaining confidence in the recovery design, not only as a delivery-time exercise.

[Delivery scope] What the implementation covered

The account describes regional application operation, distributed state management, routing and a recovery-exercise approach. It does not establish a recovery-time objective achieved, a data-loss result or successful exercise outcomes.

Active-active operation does not mean all failures are invisible to users. Network partitions, dependency failures, application bugs that affect both regions simultaneously and cascading errors can still produce visible degradation. Design monitoring and incident response for those scenarios separately from the traffic-routing change.

[Operating considerations] Questions for similar implementations

Define the failure scenarios the design is expected to handle and those it is not. Check regional infrastructure faults separately from application defects or shared third-party failures that can affect every region.

Test the routing change under production-level traffic before relying on it. A routing change that succeeds in a staging environment may encounter DNS caches, connection pool behavior or client retry logic that extends actual recovery time.

Establish a reconciliation procedure for any state written to one region and not yet replicated when a failure occurs. Even synchronous replication has a commit boundary. Understand what state is at risk at that boundary and define ownership for resolving it.

Agree the recovery objectives and required exercise evidence with the system owner and qualified risk reviewers. Record the failure conditions, recovery steps, observations and unresolved gaps.

[Applicability] When active-active applies

Active-active multi-region architecture is relevant when business or regulatory requirements specify a recovery objective that human-mediated failover cannot reliably meet, when a DR environment that is never exercised provides insufficient assurance, and when the cost of dual active regions is justified by the business impact of extended unavailability.

The approach requires applications designed for multi-region operation. Adding a second region to an application that assumes single-region, synchronous, local state access will not produce active-active behavior without application changes. The database technology must support the consistency model the workload requires.

Do not infer RPO from the presence of synchronous replication alone. Define the acknowledged-commit boundary and the failure set covered by the configured database. Verify recovery of those acknowledged operations with the chosen version and topology.

Technical reference

CockroachDB survival-goal documentation distinguishes zone and Region survival and explains the write-latency trade-off. The product name alone does not establish this deployment's configured survival goal or tested recovery behavior.