Eight Distributed Systems Failure Patterns: Diagnosis and Recovery

Diagnose cascading failures, retry storms, stalled consumers and unsafe failover with recovery boundaries, checked examples and a reusable failure exercise.

Start with the resource that cannot recover

A dependency recovers, but your application is still failing. The database answers requests again, yet workers are busy replaying old work. A restarted service warms every cache at once. A timeout hides a successful payment and the caller sends another charge.

These are different failure mechanisms. The useful diagnostic question is: what is preventing the system from returning to a stable operating state? Trace useful completions alongside attempted requests, queue age, in-flight work, connection occupancy and retries. CPU alone cannot tell you whether more capacity will help.

This article is an incident-preparation guide. The distributed-computing fallacies explain the assumptions behind these failures; resilient microservice patterns cover implementation choices. The examples below are illustrative, not incident claims about Ampity clients.

1. A slow dependency exhausts its callers

A checkout service holds a connection while waiting for inventory. More requests arrive, but existing requests retain memory and concurrency slots. Eventually checkout fails even for operations that do not need inventory.

Bound concurrent calls and waiting work per dependency. Set a total request deadline and allocate time within it, including connection establishment and response handling. Check that cancellation actually releases local resources; it may not stop remote execution.

A circuit breaker can temporarily refuse calls, but its recovery probes and fallback also consume capacity. Test them. A fallback that calls the same database through a different endpoint does not isolate the dependency. There is no universal timeout multiplier: choose a tolerable false-timeout rate from measured latency and the caller's deadline. AWS's timeout and retry guidance explains these tradeoffs.

2. Recovery triggers a thundering herd

A popular cache entry expires while many requests are waiting. They all fetch the same value. Alternatively, a deployment starts many cold instances that simultaneously populate their caches.

Coalesce concurrent loads for the same key, stagger refreshes and bound origin traffic. Coalescing within one process still allows a fetch from every other process. Decide whether that residual load is acceptable before introducing a shared lock with its own failure modes.

Serving an older value may be safe for a product description but unsafe for an entitlement or stock reservation. Define which data may be stale, for how long, and what happens after that limit. Avoid turning a cache availability optimization into an authorization bypass.

3. The overload sustains itself

A burst can leave a system in a state where retries, queue processing and cache misses consume the capacity needed to recover. Removing the original burst is then insufficient. The HotOS paper on metastable failures describes this sustaining effect, which distinguishes the failure from a temporary spike.

Reduce admitted work, preserve essential traffic and rebuild capacity in controlled increments. Reject work before spending most of its processing cost. Separate recovery traffic from new demand so replay cannot consume the entire service budget.

A restart can make this worse by discarding warm state. Record the recovery condition before using it: for example, useful completions exceed admitted work, oldest actionable message age decreases and error rates remain within the service's agreed operating limits. Google SRE's overload guidance describes admission control and the danger of retry amplification.

4. A consumer falls behind

Suppose arrivals remain at 1,000 messages per second and successful processing is 900 per second. In this simplified example, backlog grows by 100 per second, or 30,000 messages in five minutes. If capacity later reaches 1,200 per second while arrivals stay at 1,000, the net drain is 200 per second. Clearing that backlog takes 150 seconds, before accounting for variability or new failures.

Scaling consumers only helps if the partition layout, ordering requirements and downstream capacity permit more useful work. A hot key may remain serial even when more workers exist.

Treat each message type differently. An expired location update may be replaceable by a newer update. An accepted invoice command needs an auditable completion, rejection or compensation outcome. Do not discard it because a generic TTL elapsed. A dead-letter destination also needs an owner, alert, retention policy and safe replay process; it is not successful delivery.

5. Timestamps create a false history

Wall-clock time is useful for observations, but clock skew can misorder events across machines. Lamport clocks guarantee that a happened-before relationship implies increasing logical timestamps. The reverse implication does not hold. Lamport's original paper defines that distinction.

Consider two processes with counters initially zero:

| Event | Logical value | What is known | | --- | --- | --- | | A changes a local record | A: 1 | No message connects this event to B. | | B performs two local operations | B: 1, then B: 2 | B's two events are ordered locally. | | A sends a message, then B receives it | Send: 2; receive: 3 | The send happened before the receive. |

A's first event has a lower timestamp than B's second event, but that does not establish causality between them. Vector clocks can represent concurrent histories under their algorithm's assumptions, at the cost of additional metadata and membership handling.

Idempotency addresses duplicate application, not arbitrary reordering. “Set status to cancelled” and “set status to shipped” can each be idempotent and still produce different results when reordered. Enforce allowed state transitions with versions or another suitable concurrency-control mechanism.

6. An obsolete leader still writes

A process pauses, loses leadership and later resumes. If it still has storage credentials, it may perform writes after a replacement leader has taken over.

Consensus-based leadership and fencing solve related but different problems. With a fixed five-voter Raft configuration, a majority is three; a two-voter partition cannot form a majority. That statement depends on Raft's election and log rules, not just counting acknowledgements in an arbitrary system. See the Raft paper.

Protect external side effects separately. A fencing design issues monotonically increasing tokens and requires the protected resource to reject stale tokens. Martin Kleppmann's distributed-locking analysis illustrates why enforcement at the resource matters. Test the pause, expiry, replacement and resumed-write sequence. If the target cannot enforce fencing, choose an alternative such as serialized commands or conditional version checks.

7. A health check causes a second incident

Use separate signals for separate actions:

| Signal | Action | Boundary | | --- | --- | --- | | Liveness | Restart a stuck process | A shared DB outage is not fixed by restarting every client. | | Readiness | Remove an instance from routing | Check that this helps. A shared DB check may remove all instances. | | Startup | Wait for local startup | Allow realistic startup time; detect a stuck start. | | Diagnostics | Alert or choose a safe degraded mode | Report remote faults without forcing local restarts. |

These distinctions follow Kubernetes probe semantics. For a service that cannot safely answer any request without its database, failing readiness may be appropriate. For one that can still serve independent endpoints, an all-or-nothing dependency check may unnecessarily remove useful capacity.

Keep probes bounded and cheap. Exercise dependency loss and restoration across the whole fleet, not only one container. Inspect whether connection storms, readiness flapping or restart loops delay recovery.

8. Retries multiply demand

Distinguish attempts from retries. In an illustrative three-layer call chain, each layer permits three total attempts: one original plus two retries. If every downstream attempt fails and every layer exhausts its allowance, one incoming operation can trigger 3 × 3 × 3 = 27 attempts at the deepest dependency. This is a worst-case count, not a prediction from a particular failure percentage.

Use an explicit retry owner where practical, bounded attempts, jitter and an overall deadline. A retry budget should be derived from available recovery capacity and useful-completion goals, not copied as a universal percentage. Include SDK retries when calculating it.

Retry only operations whose error and side-effect semantics permit it. A timeout means the caller lacks a result; it does not establish that no write occurred. Preserve the operation's idempotency key, provide a status lookup where possible and reconcile unknown outcomes before issuing a different operation.

Run one bounded recovery exercise

Use an approved non-production environment first. Record the workload, affected dependency, abort condition, person authorized to stop the exercise and rollback procedure. Production fault injection requires separate approval.

| Exercise | Evidence | Recovery proof | | --- | --- | --- | | Slow one service | In-flight calls, deadlines, useful results | Other paths retain capacity; waiting work remains bounded. | | Restore it with a backlog | Oldest message age, new arrivals, drain rate | Replay does not starve new work; no accepted command disappears. | | Lose a response after a write | Operation ID, stored result, retry attempts | One intended effect and a discoverable final outcome. | | Pause an old leader | Leadership term and accepted write tokens | Resumed stale work cannot alter the protected resource. |

Close the exercise only when the business state is reconciled as well as the infrastructure being healthy. List unresolved outcomes explicitly. If you need help defining that evidence and the recovery boundary, bring the dependency map and latest incident timeline to an Ampity reliability review.