Building Reliable Distributed Systems

Design cross-service operations with bounded retries, durable idempotency, explicit unknown outcomes, safe degradation, and tested reconciliation procedures.

trigger="A business operation crosses a service, queue, or external-provider boundary and must remain correct through partial failures." owner="The technical owner accountable for the operation's end-to-end business outcome." participants={["Calling-service owner", "Receiving-service owner", "Database owner", "Platform or SRE lead", "Security reviewer", "Business operations owner"]} prerequisites={[ "A documented operation, source of truth, business invariants, and external side effects.", "Known timeout, retry, idempotency, and retention behavior for every participant.", "A representative failure-test environment and an approved reconciliation process." ]} outputs={[ "An operation contract with deadlines, retry ownership, duplicate behavior, and explicit pending outcomes.", "A durable state and event design with recovery responsibilities at each failure boundary.", "Failure-test evidence, a reconciliation runbook, and a staged rollout decision." ]} doneWhen={[ "Duplicate, delayed, concurrent, and ambiguous operations preserve the documented invariants.", "Retry and queue limits prevent unbounded work during dependency failures.", "An operator can resolve unknown outcomes without blindly repeating an external effect.", "Security behavior, compensation limits, and unresolved risks have named owners." ]} />

Design for an operation, not a collection of patterns

A remote call can time out after the receiver has already committed its work. A message can be delivered again after a consumer finishes but fails to acknowledge it. These are not simply “failed requests.” They are uncertainty about a business outcome.

Start with one operation, such as accepting an order and requesting payment. State what must be true afterward, which system owns each fact, and what the user should see while the outcome is unknown. Adding retries, circuit breakers, and queues without that contract can make the system busier without making it more reliable.

This playbook provides a procedure and reusable records. Its examples are illustrative engineering designs, not client results or a guarantee of exactly-once effects across independent systems. Verify the actual guarantees of the database, broker, and provider used in the implementation.

1. Write the operation contract and invariants

The business owner identifies unacceptable outcomes: duplicate charges, granting access before approval, missing inventory reservations, or a completed status without a completed external action. The technical owner turns those into testable invariants.

| Contract field | Required decision | | --- | --- | | Operation identity | How is the same logical request recognized across retries and redelivery? | | Authority | Which system owns the business record and each external effect? | | Success | Which durable facts must exist before success is returned? | | Pending or unknown | How can the caller query progress without creating another operation? | | Duplicate handling | What happens for the same identity and same or different payload? | | Deadline | How long can the synchronous request and background workflow continue? | | Cancellation | What does cancellation mean after work has been accepted or dispatched? | | Recovery | Who can reconcile, retry, compensate, or close an unresolved operation? |

Distinguish request timeout from workflow expiry. A user may stop waiting while durable work continues. Conversely, the system may need to stop further side effects after a business deadline even if a queue still contains the message. Record which timestamps and state transitions enforce each condition.

Do not report an operation as complete merely because it entered a queue. Return an accepted or pending state with a stable reference when the contract allows asynchronous completion. The client should have a defined way to discover success, failure, or a request for intervention.

2. Allocate deadlines and retry ownership

Define an end-to-end deadline and allocate downstream budgets within it, including connection setup, execution, and time to return a response. Propagate cancellation where supported, but do not assume that canceling a caller undoes a committed remote effect.

Choose one intentional retry owner for each boundary and inspect retries already performed by SDKs, proxies, workers, and clients. Layered retry policies can multiply work. Bound attempts, total elapsed time, concurrent retries, and queued retry work. Use backoff with jitter where supported to avoid synchronized retry waves.

AWS's retry-with-backoff guidance emphasizes transient failures, idempotency, and the risk of overload. Treat those as eligibility checks, not an instruction to retry every error.

| Observed result | Decision | | --- | --- | | Invalid input or permanent policy rejection | Return the documented failure; retrying unchanged input is not useful | | Authentication or authorization failure | Follow the security contract, not an availability fallback | | Explicit transient rejection before work is accepted | Retry only within the operation's deadline and retry budget | | Timeout after a side effect may have started | Query or reconcile the original operation, or retry only under a verified duplicate-safe contract | | Dependency remains overloaded | Reduce admitted work; backoff alone may not protect it | | Retry budget or business deadline exhausted | Move to the defined failed, pending, or intervention state |

Honor the dependency's documented retry signals and restrictions. Record why a particular status is considered retryable for this operation. A status code alone may not tell you whether a business effect happened.

3. Make duplicate handling durable

An idempotency key identifies a logical operation; it is not a guarantee by itself. Scope it to the tenant or caller and operation type. Validate that a repeated key carries an equivalent request, usually through a canonical payload representation or hash. Reject conflicting reuse rather than returning an unrelated old result.

For a local operation whose business state and deduplication record share one transactional database, design their commit together. Use a unique constraint or equivalent concurrency control so two simultaneous requests cannot both claim the same operation. Store enough result information to answer a legitimate retry consistently.

External effects introduce another boundary. If the worker sends a request and crashes before recording the response, the ledger may say “pending” even though the provider completed the action. Do not treat an old pending timestamp as permission to execute it again. Use the provider's operation lookup or documented idempotency contract, or route the case to reconciliation.

| Ledger state | Meaning | Safe next action | | --- | --- | --- | | Accepted | Durable intent exists; dispatch may not have occurred | Use controlled worker ownership to advance | | In progress | A worker owns execution or dispatch is being resolved | Observe or query; do not start a competing effect | | Succeeded | Required durable outcome is known | Return the stored result | | Failed definitively | The contract establishes that the intended effect did not complete | Apply the documented terminal or new-operation policy | | Needs reconciliation | Evidence is insufficient to establish the outcome | Investigate before repeating or compensating |

Define retention from the longest supported retry, message-redelivery, offline-client, replay, and dispute or operational-recovery window. Account for data minimization and deletion requirements as well. There is no universal retention period that covers every workflow. When a key's record expires, specify whether reuse becomes a new operation or is rejected.

For a concrete provider example, Stripe's idempotent-request documentation defines its own result-storage, parameter-checking, and key-pruning behavior. That provider-specific contract should not be generalized to all APIs.

4. Separate local atomicity from message delivery

When a database change must produce an event, store business state and event intent in the same local transaction where the database supports that design. A separate publisher delivers committed events. This transactional outbox avoids treating a database write and a broker send as an atomic pair.

"type": "svg-architecture", "title": "Durable intent with explicit external uncertainty", "nodes": [ ], "links": [ ], "caption": "The local transaction boundary does not include the external provider. A lost response requires duplicate-safe lookup or reconciliation." }} />

The publisher can still emit a duplicate, for example after sending an event but before persisting its progress. Consumers therefore need duplicate handling aligned with their side effects. AWS's transactional-outbox guidance describes the dual-write problem and ordering and duplicate-message considerations.

Define ordering where the business needs it, such as per order or account, rather than assuming global ordering. Include an aggregate identifier and sequence or version when appropriate, then specify how consumers handle gaps, late events, and incompatible schemas.

Monitor the oldest unpublished event, consumer delay, delivery attempts, and unresolved operations. A growing outbox is not just a storage issue; it may mean the product reports accepted work that cannot complete. Define backpressure or admission limits before that backlog becomes unmanageable.

5. Bound failures without weakening security

A circuit breaker can stop new attempts to a persistently failing dependency. Its failure classification must match the dependency, and its recovery probes need limits. Authorization denials and invalid requests should not normally be treated as evidence that a dependency is unavailable.

Use separate concurrency budgets for critical and optional work where the runtime and deployment support that isolation. A queue also needs a maximum age, capacity policy, and a decision for expired work. Otherwise it can defer an outage into a later surge.

| Dependency failure | Permitted degradation example | Boundary that remains intact | | --- | --- | --- | | Optional enrichment unavailable | Omit enrichment | Core operation stays correct | | Non-critical read model delayed | Disclose an approved stale view | Freshness and tenant policy remain enforced | | External effect uncertain | Return pending with an operation reference | Do not claim completion or duplicate the effect | | Identity service unavailable | Follow the reviewed verification and denial policy | Do not invent a token-expiry exception | | Worker capacity exhausted | Reject or defer according to the contract | Bound accepted work and disclose the state |

JWT expiry is a security condition, not an incident-control knob. RFC 7519, section 4.1.4 defines expiration processing and limited clock-skew leeway. Clock-skew handling is not permission to extend expired credentials during an outage. Any offline verification or emergency-access design must be reviewed separately, including key rotation, revocation, audience, issuer, and audit requirements.

6. Design reconciliation and compensation before launch

Reconciliation establishes what happened by comparing authoritative records, provider state, and durable event history. Compensation is a new business action intended to offset a completed action. Neither is equivalent to rolling back one database transaction.

For example, canceling a reservation may be possible while reversing a shipped item is not. A refund can have different timing, permissions, and failure modes from the original payment. The business owner must accept the allowed compensations and the cases requiring a person.

"type": "flow", "title": "Resolve an unknown outcome before taking another effect", "steps": [ ], "caption": "Still-unknown or conflicting evidence remains unresolved. The flow does not authorize automatic retry or compensation." }} />

Give reconciliation jobs least-privilege access, bounded batches, rate limits, and an audit record. Avoid placing full sensitive payloads in operational logs. Every manual decision should identify the operator, evidence, chosen action, and resulting operation state.

A dead-letter queue is a holding area, not a recovery strategy. Before replay, confirm the defect is fixed, messages are still relevant, schema versions are supported, and duplicate controls remain valid. Start with a bounded batch and check outcomes before increasing it. Stop when evidence conflicts or the downstream budget is exceeded.

7. Test the failure windows

The technical owner prepares controlled faults at the boundaries where state can diverge. Each test needs a business assertion, not only a transport-level success check.

| Fault to inject | Required assertion | | --- | --- | | Two simultaneous requests with the same key | One logical operation owns execution; both callers get contract-consistent outcomes | | Same key with a different payload | Conflicting reuse is rejected | | Process loss before local commit | No partial local operation is exposed as complete | | Process loss after local commit, before publication | Committed intent is eventually recovered within the workflow policy | | Duplicate or out-of-order event | Consumer preserves the relevant invariant and ordering rule | | Provider completes but response is lost | Original effect is found or held for reconciliation, not blindly repeated | | Lease or worker ownership expires mid-call | A second worker cannot assume the first effect did not occur | | Provider outage under load | Retry, concurrency, and backlog limits remain bounded | | Replay near or beyond deduplication retention | The defined expiry policy prevents unintended new effects | | Expired or revoked credentials | Availability behavior does not bypass the security contract |

Record the tested service, SDK, broker, and database versions. SDK retry defaults and provider behavior can change. Repeat relevant tests after those changes and after a material change in workflow duration or replay policy.

8. Roll out and keep recovery operable

Roll out a bounded operation category or caller cohort with independent controls for new admissions, background dispatch, and optional downstream effects. Observe pending age, duplicate conflicts, reconciliation volume, and user-visible completion, not merely API response latency.

If a regression appears, stop expanding and decide whether to halt new admissions or pause dispatch. Preserve durable intent and ledger state. Reverting application code is safe only if it can read the state and event versions already written. Once an external effect occurs, code rollback does not reverse it.

Use this record to hand the operation to on-call and business operations:

Operation and accountable owner:
Business invariants and sources of truth:
Stable identity / scope / payload equivalence:
Success, pending, failure, and expiry semantics:
Deadline and retry owner at each boundary:
Ledger transitions and concurrency controls:
Event ordering / duplicate handling / retention:
External lookup and idempotency contract:
Reconciliation evidence and operator permissions:
Permitted compensation / approval required:
Admission and dispatch stop controls:
Rollback compatibility limit / forward recovery:
Failure-test evidence and unresolved cases:

9. Review the dependency contract before expansion

For each remote dependency, record the supported operation, identity, deadline, retry behavior, idempotency or lookup contract, rate and concurrency limits, version policy, failure response and escalation owner. Verify the current client configuration rather than relying on library defaults.

Exercise one timeout before the dependency accepts work and one timeout after acceptance. Confirm that the caller distinguishes safe retry from unknown outcome. Then exercise overload while several callers retry. The dependency and callers should remain within their shared work budget rather than amplifying failure.

If a provider cannot expose operation status or support duplicate-safe retry, keep the action behind human review or a reconciliation queue appropriate to its consequence. Do not describe a write as reliable merely because the normal response is fast.

Expand only when the service owner and business operator can explain how pending and unknown work will be resolved. The launch decision should include retained history, reviewer capacity and the point at which an unresolved operation expires or escalates.

10. Keep schemas and recovery compatible

Version messages and stored operation state so the previous and current consumers can coexist through the rollout and recovery window. Test old producers with new consumers and new producers with every consumer that remains supported. Unknown fields, removed meanings and changed defaults need an explicit policy.

A code rollback is unsafe when the previous version cannot read state already written by the candidate. Use expansion before contraction, preserve compatible event history and define forward recovery when the compatibility window closes. Record the last version that can safely process the current ledger and message shapes.

Rehearse replay with external-effect credentials removed or isolated. A replay intended to rebuild derived state must not resend emails, charges or other business actions. Verify the rebuilt view at a known source position before switching readers.

Completion checklist

"The operation has a stable identity and explicit success, pending, failure, and unknown outcomes.", "Every remote boundary has a deadline, retry owner, and bounded work budget.", "Concurrent duplicates and conflicting key reuse are tested against durable state.", "External effects have lookup, duplicate-safe retry, or an owned reconciliation path.", "Outbox and consumer recovery handle duplicates, ordering, schema versions, and backlog growth.", "Degradation preserves security and data-correctness requirements.", "Compensation and replay require the documented evidence and authority.", "Rollback compatibility and forward recovery are rehearsed before broader rollout." ]} />

For workload-level objectives and recovery exercises, continue with Cloud Reliability Engineering. Ampity's system architecture design is the related service when the operation crosses teams or needs an explicit architecture and ownership review.

Primary references

Validate implementation details against the deployed versions and provider agreements. These sources support the design checks; passing the operation-specific failure tests is still required.