Cloud Reliability Engineering

Run a workload reliability review with user-facing objectives, dependency failure tests, capacity limits, recovery evidence, and an owned improvement backlog.

trigger="A critical workload has recurring incidents, an untested recovery plan, or a material architecture or traffic change." owner="The engineering owner accountable for the workload's production behavior." participants={["Product owner", "SRE or platform lead", "Database owner", "Security representative", "On-call responders"]} prerequisites={[ "A named workload and user journey, dependency inventory, and access to production telemetry.", "Documented business impact, data sensitivity, and existing availability or recovery commitments.", "A representative test environment, current backup evidence, and approved limits for failure experiments." ]} outputs={[ "A reliability contract with measured indicators, objective windows, and accepted degradation.", "A dependency and capacity register with owned mitigation and recovery procedures.", "A tested recovery record and prioritized backlog with acceptance evidence." ]} doneWhen={[ "The product and service owners accept the reliability contract and unresolved risks.", "Failure and restore exercises demonstrate the agreed behavior or produce explicitly owned gaps.", "Responders can find the runbook, access the recovery environment, and verify data correctness.", "Every improvement has an owner, completion evidence, and a review trigger." ]} />

Scope: one workload, one verifiable outcome

Use this playbook to decide whether a production workload can continue, degrade safely, and recover under its relevant failure conditions. The outcome is an evidence-backed operating plan, not a blanket availability guarantee. A deployment across several zones is one control, not proof that the user journey remains usable.

Start with one journey, such as submitting an order or retrieving a customer record. Include authentication, application services, data stores, queues, external providers, and the operational tools needed to recover them. Identify which dependencies are shared with other workloads. A shared identity provider or database can dominate reliability even when application instances are redundant.

This is an engineering procedure, not a report of a client implementation. Thresholds and recovery objectives must come from the workload's commitments and measurements. If telemetry cannot distinguish a successful request from an incorrect result, improving that evidence comes before claiming an objective is met.

1. Agree the reliability contract

The product owner defines the consequence of failure. The service owner translates it into observable behavior and proposes a service-level objective, or SLO. Availability, latency, correctness, and freshness may need separate indicators. HTTP success alone does not establish that the right customer received the right result.

Google's SLO implementation guidance describes selecting useful indicators and obtaining stakeholder agreement. Apply that approach to the specific journey, including a defined measurement window and documented exclusions. Do not choose a target solely because another system uses it.

| Contract field | Decision to record | Acceptance evidence | | --- | --- | --- | | User journey | Who needs which outcome, from which locations? | Product owner confirms the boundary | | Good event | Correct result, allowed latency, and relevant freshness | Query or probe can classify good and bad events | | Eligible events | Which requests count, including retries and client cancellations? | Sampling and exclusions are documented | | Objective and window | Target, rolling or calendar window, and reporting owner | Historical baseline and stakeholder approval | | Degradation | Which features may be unavailable or stale? | Product and security acceptance | | Recovery | Recovery time objective and recovery point objective | Restore and reconciliation exercise results |

For a request-based availability SLO, the allowed bad-event budget is the eligible event count multiplied by one minus the objective. Do not convert that automatically into outage minutes: low-volume periods and traffic variation can make the two views materially different. Record the chosen accounting model and use it consistently.

Agree what happens when reliability deteriorates. A policy might restrict high-risk releases while permitting fixes, security patches, and approved mitigations. Name who can make an exception and what evidence they need. An exhausted error budget does not by itself identify the cause or justify stopping every change.

2. Map failure boundaries and ownership

Draw the smallest dependency map that explains how the journey can fail. Add the recovery control plane separately: credentials, DNS, configuration, backups, and access to deployment tools may be unavailable during the same incident.

"type": "svg-architecture", "title": "Review the full user journey", "nodes": [ ], "links": [ ], "caption": "Logical review boundaries, not a deployment topology. Recovery controls require their own failure and access checks." }} />

For each dependency, record its owner, timeout behavior, capacity constraint, redundancy boundary, and observable failure signal. Distinguish a process crash from a slow dependency: a slow response can occupy connections and worker slots until apparently healthy callers also fail.

Review the actual managed-service configuration and regional support. Multi-zone placement may address a zone disruption, but it does not automatically address bad configuration, corrupted data, a regional outage, or an account-access problem. Keep these scenarios separate so the review does not mistake redundancy for recovery.

3. Make degradation safe

A fallback is useful only when its output remains acceptable. Do not serve an old authorization decision or financial value simply to improve an availability chart. Security-sensitive checks should follow the approved identity and authorization policy, including denial or an explicit unavailable response when a valid decision cannot be made.

| Failure | Candidate behavior | Required condition | | --- | --- | --- | | Recommendation service unavailable | Omit recommendations | Core transaction remains correct | | Catalogue dependency slow | Serve an approved stale representation | Freshness limit, tenant isolation, and visible status are acceptable | | Payment outcome unknown | Show pending and reconcile | No new charge is attempted without a duplicate-safe contract | | Read replica behind | Route eligible reads elsewhere or reject them | Required freshness is known; fallback capacity is available | | Cache unavailable | Bypass only within the database budget | Admission control prevents a miss storm | | Identity dependency unavailable | Apply documented security behavior | No ad hoc extension of expired credentials |

Route read-after-write and other freshness-sensitive operations to the authoritative path unless the selected replica can prove the required committed state is visible. A fixed wait does not establish that a replica caught up. The detailed routing and invalidation checks are in Read Replicas vs Caching.

For remote calls, set deadlines from the end-to-end request budget. Define which layer owns retries, which failures qualify, and when the caller stops. Backoff reduces retry frequency but cannot make an unsafe duplicate write safe. AWS's retry guidance identifies idempotency and overload as key conditions.

4. Budget capacity for degraded operation

Test the capacity remaining after the failure you intend to tolerate. A workload that fits across all instances may not fit after losing a zone. Include connection establishment, cache warming, queue drain, and background work rather than measuring only steady-state application throughput.

The database owner should allocate an aggregate connection budget across maximum application replicas, workers, maintenance, administration, and deployment overlap. A per-process pool limit without a deployment-wide calculation can exceed the database limit during autoscaling. There is no universally correct pool size based only on processor cores or disks.

Record the maximum useful concurrency, queue length, queue-age limit, and admission policy for expensive operations. Define which requests can be shed and how clients are told to retry. If the fallback path cannot handle the failed component's traffic, reduce admitted work instead of forwarding unlimited requests to it.

The output is a capacity worksheet, not merely a utilization graph. It should explain the workload mix, tested configuration, bottleneck, safety margin chosen by the owner, and the next load condition that requires another test.

5. Test failures with a stop condition

The experiment owner writes the hypothesis and abort conditions before injecting a fault. Start in a representative non-production environment. Production exercises require explicit authorization, a bounded blast radius, named responders, and a working way to stop the experiment.

| Experiment | Observe | Stop or fail the test when | | --- | --- | --- | | Dependency latency or timeout | End-to-end latency, concurrency, retry volume | Work grows beyond the approved budget | | Cache loss | Database demand, rejection behavior, cold-start latency | Fallback threatens authoritative writes | | Instance or zone loss | Remaining capacity and user-journey probes | Impact exceeds the approved boundary | | Queue consumer interruption | Oldest message age, duplicates, drain rate | Recovery cannot meet the backlog objective | | Replica lag | Routing and stale-read detection | A protected read receives unapproved stale data | | Restore into isolation | Recovery time and application-level invariants | Missing keys, logs, or dependencies prevent validation |

A test passes when the predicted control worked and the user-facing result was acceptable. “The infrastructure restarted” is insufficient if queued operations are duplicated or restored records cannot be used. Save start and end times, telemetry links, affected versions, and the operator's observations.

6. Rehearse recovery and reconciliation

Recovery time objective, or RTO, is the target time to restore the agreed service. Recovery point objective, or RPO, is the target limit on data loss measured in time. The business owner accepts those targets; an exercise establishes what the tested configuration actually achieved. AWS's disaster recovery guidance treats definition, strategy, and testing as distinct responsibilities.

"type": "flow", "title": "A controlled recovery sequence", "steps": [ ], "caption": "Advance only after the previous gate passes. A failed reconciliation holds affected operations for investigation." }} />

Before promoting a replacement writer, prevent the old writer from accepting conflicting work using the platform's supported fencing procedure. Record which writes may be missing or ambiguous. After a restore, compare business records with durable events and external-provider records where appropriate. A restored database does not automatically undo messages, charges, or notifications already sent.

Failback is a separate planned change. Do not point traffic back at an old primary whose data has diverged. Rebuild or reconcile it using the engine's supported procedure, validate backups and replication, and obtain a new cutover decision.

7. Keep the runbook executable

Use this record for every material failure mode. Store identifiers and evidence links, not credentials or sensitive customer payloads.

Workload / journey:
Failure scenario and customer impact:
Incident owner / technical operator / communication owner:
Detection query and dashboard:
Immediate containment:
Required access and dependencies:
Recovery command or approved procedure reference:
Data at risk and reconciliation method:
Abort condition and escalation:
Rollback or forward-recovery decision:
Evidence of user-journey recovery:
Unresolved risk / owner / due date:
Last exercise configuration / next review trigger:

Assign the incident commander decision authority and make escalation paths usable outside normal working hours. Verify access from the environment responders will actually use. If recovery needs an unavailable corporate identity system, the runbook must address that dependency through an approved access mechanism.

After an exercise or incident, separate detection gaps, mitigation gaps, and data-correctness gaps. Prioritize the work by business exposure and recurrence, not by the visual prominence of a dashboard. Each backlog item needs an observable acceptance condition.

8. Review reliability as a product decision

Reliability competes with feature work, cost and architectural simplicity. The product owner and service owner should review the same evidence: user impact, objective performance, budget consumption, incident patterns, recovery exercises, operational load and planned changes.

Do not tighten an objective merely because the current service often exceeds it. A tighter target can require additional capacity, fault isolation, testing and support. Do not relax it merely because the current architecture misses it. Compare the user consequence, contractual commitments, remediation options and cost of the new target.

Use an error budget policy to decide when normal change may continue and when reliability work receives priority. Define which failures count, how missing telemetry is handled, who can approve an exception and what evidence restores normal policy. A dashboard that shows remaining budget without a decision policy is reporting, not governance.

Review dependency objectives carefully. An upstream service may meet its own aggregate target while a particular consumer journey suffers. Use the end-to-end journey objective for user impact and dependency indicators for diagnosis. Attribution changes who repairs the problem, not whether the user experienced it.

9. Plan reliability capacity and ownership

Operational ownership requires time to remove recurring failure, not only time to respond. Estimate incident interruption, support requests, recovery exercises, dependency upgrades and corrective work. If these consume the team’s available capacity, reduce scope, add capability or change the service commitment.

Assign one accountable owner for the journey and named owners for specialized recovery actions. A database team may control promotion, a platform team may control routing and the service team may reconcile business work. Their runbooks must meet at explicit decision points so responders do not issue conflicting commands.

Maintain primary and backup ownership, access paths and escalation outside normal working hours where the service requires it. Test emergency access without storing credentials in the runbook. Remove access when responsibility changes and verify that the new owner can perform the recovery exercise.

Use a quarterly or material-change review to inspect whether failure assumptions, recovery dependencies, traffic, data volume, staffing or provider behavior have changed. Version the reliability contract rather than silently updating a target or runbook that earlier decisions relied on.

10. Make the next improvement testable

Select the failure mode with the greatest combination of user consequence, recurrence and weak recovery evidence. Define one change, the mechanism it should improve, the failure exercise that will test it and the operational result required for closure.

If the exercise fails, restore the known operating boundary and retain the evidence. Do not mark a reliability item complete because code merged, capacity was purchased or a document changed. Closure requires the agreed user, system or recovery behavior under the relevant condition.

11. Retain the reliability decision record

Journey, owner, eligible population, and objective:
Failure boundaries, critical dependencies, and degraded behavior:
Capacity under normal, peak, and required failure conditions:
Detection evidence, missing-signal behavior, and escalation:
Recovery objectives, procedure, access, and reconciliation:
Error budget policy, exception authority, and review trigger:
Exercise configuration, observed outcome, and unresolved risk:

Keep the record with the service and update it after material architecture, traffic, data, provider or ownership changes. An old recovery test does not prove that the current version, credentials, backup or receiving team can recover the workload.

Review temporary mitigations separately. Additional retries, disabled validation, broader access or manual reconciliation can help contain one incident while increasing future risk. Assign each temporary condition an owner, expiry and removal test. If it becomes part of the normal service, update the reliability and security contracts openly.

During handover, ask the receiving operator to explain the failure boundary and run one recovery step without coaching. Missing access, unclear authority or undocumented prerequisites remain readiness gaps even when the original author can complete the procedure.

Record the handover result, remaining coaching need and final acceptance owner in the service record.

Revisit acceptance when the recovery procedure or receiving team changes materially.

Keep that trigger visible to operators and reviewers throughout the support period.

Completion and handoff checklist

"Product and engineering owners agree the journey, SLO accounting, degradation policy, RTO, and RPO.", "Critical dependencies, shared failure boundaries, and recovery access have named owners.", "Retries, queues, connection pools, and fallback paths have deployment-wide limits.", "Failure exercises include slow dependencies, cold caches, replica lag, and unavailable capacity.", "A restore was validated with application-level checks and uncertain external effects were reconciled.", "Failover, failback, communication, and escalation procedures have been rehearsed.", "Remaining risks are explicitly accepted or assigned, with evidence required for closure." ]} />

Use Building Reliable Distributed Systems for operation-level retry and reconciliation design. When the review needs cross-team facilitation, Ampity's cloud reliability review is the related service. The service scope and acceptance criteria should follow the workload evidence, not a promised availability number.

Primary references and implementation checks

Validate commands, replication behavior, and managed-service limitations against the deployed versions. These references support the design checks; they do not certify a particular workload or replace its recovery exercise.