Event-Driven Architecture: When Async Stops Being Worth It

Decide where asynchronous work earns its operational cost, then migrate safely with durable handoff, one side-effect owner, and reconciliation evidence.

trigger="A queue, event bus, or synchronous dependency is creating an operational problem, or a new workflow needs a communication boundary." owner="The service owner responsible for the user-visible outcome across the proposed boundary." participants={["Producer owner", "Consumer owner", "Data owner", "SRE or on-call representative", "Product owner"]} prerequisites={[ "A traced business operation, its success definition, and its acceptable pending and failure states.", "Observed normal and burst traffic, dependency behavior, queue age, retries, and incident evidence.", "A named authority for each state change and an inventory of consumers, side effects, and replay obligations." ]} outputs={[ "A boundary decision with alternatives, measured assumptions, and a reassessment trigger.", "An owned handoff contract, migration record, and side-effect isolation plan.", "Load, failure, cutover, and reconciliation evidence supporting the selected design." ]} doneWhen={[ "The caller can distinguish completion, acceptance, rejection, and an unknown outcome.", "Failure and burst tests meet locally agreed objectives without overwhelming dependencies.", "Only the selected executor performs each side effect, including during rollback.", "Outstanding work, duplicate handling, and recovery have been reconciled and assigned." ]} />

Decide on a boundary, not an architectural identity

A system can use synchronous calls for decisions that need immediate answers and asynchronous work for operations that can safely remain pending. Neither approach makes the whole system inherently simple, scalable, or reliable.

This playbook helps decide whether a particular asynchronous boundary earns its complexity. Use event-driven systems that scale for implementation guidance after that decision. Removing a broker is not success if its retry and buffering responsibilities reappear as fragile application code.

Start with one operation, such as confirming a booking or generating an export. Record what the user believes happened when the interface reports success. A design that responds quickly but silently loses required work has not improved that outcome.

1. Map facts, commands, and completion

The producer and consumer owners describe the boundary together. Is the message a request to perform work, or a fact that has already occurred? Who may reject it? Who owns the authoritative status? What does the caller do when the answer is unavailable?

| Question | Required evidence | | --- | --- | | Must the caller know the result now? | User journey and deadline, not only a latency target | | Can the work remain pending? | Visible status, expiry rule, cancellation behavior, support path | | Can it be retried? | Stable operation identity and protected external effects | | Does order matter? | Specific entity or partition, ordering rule, version-conflict behavior | | Can it be replayed? | Retained source, compatible schema, authorized replay procedure | | Who operates it? | Producer, consumer, backlog, and incident ownership |

Synchronous transport does not guarantee global request ordering. Concurrent requests can execute in a different order, and a timeout does not prove that the server did nothing. A call across services also does not extend one database's ACID transaction to every participant. Define concurrency control, atomic updates, and ambiguous-outcome recovery explicitly.

Asynchronous transport likewise does not make a fact durable by itself. Broker configuration, publication, consumer acknowledgments, and retention all affect what survives a failure.

2. Name the benefit and test the alternative

Choose a decision hypothesis that can be disproved. “A durable queue lets intake continue during a bounded reporting-service outage” is testable. “Events are future-proof” is not.

| Candidate | Fits when | Cost that must be owned | | --- | --- | --- | | Direct call | Immediate result matters and dependency latency fits the request budget | Timeouts, admission control, dependency availability, unknown outcomes | | Durable work queue | Work can remain pending and buffering or retry isolation is valuable | Backlog age, duplicate execution, expiry, worker recovery | | Event publication | Independent consumers need a committed fact | Schema evolution, consumer inventory, delivery and replay behavior | | In-process background work | Loss on process termination is explicitly acceptable | No implied durability; lifecycle and resource limits remain | | Database-backed jobs | A durable local work record suits the workload | Claiming, leases, contention, cleanup, and operational ownership |

An existing queue with little lag may be working as intended. Test burst arrival, consumer outages, and downstream rate limits before concluding that buffering adds no value. Microsoft's queue-based load-leveling pattern describes buffering and the need to constrain downstream load. A queue cannot absorb sustained overload indefinitely.

Gate: retain, simplify, or replace the boundary only after comparing equivalent user outcomes and failure behavior. Broker cost alone is not a complete comparison.

3. Establish a workload and operations baseline

The SRE or service owner gathers arrival rate, processing duration, oldest-work age, completion latency, failure categories, retry volume, and downstream saturation. Separate normal traffic from bursts and recovery after outages. Queue depth alone can hide a small number of very old, important operations.

Record what support staff need to answer: Was this operation accepted? Which stage owns it? Has an external effect occurred? Can it be retried safely? Missing correlation or status records may explain debugging difficulty more directly than the transport choice.

Estimate the full operating cost using your deployment: infrastructure, retention, network transfer, observability, upgrades, incident response, and engineering maintenance. Do not import a universal broker price or staffing ratio. Existing capabilities and managed-service responsibilities change the comparison.

Set acceptance conditions with product and dependency owners. For example, the backlog must drain before the business deadline without exceeding the downstream service's agreed capacity. Derive the actual deadline and capacity from the workload, not a generic message-count threshold.

4. Make acceptance a durable, owned handoff

For work that must survive a process crash, persist its intent durably before telling the caller it has been accepted. When a business change and message intent share a transactional store, an outbox can commit them together. A publisher subsequently sends the recorded intent.

AWS's transactional outbox guidance addresses the database-plus-message dual-write problem. Publication can still be repeated, so the consumer needs duplicate handling. CDC is another possible publication mechanism when its transaction, retention, ordering, and recovery behavior are understood.

"type": "svg-architecture", "title": "Commit intent before handing off durable work", "nodes": [ ], "links": [ ], "caption": "The outbox closes a local commit gap, not a transaction across every system. External side effects still require idempotency or an explicit uncertain-outcome and reconciliation path." }} />

An idempotency record written before an external call is not sufficient if a crash leaves the call's outcome unknown. Use the downstream system's supported idempotency and lookup behavior where available. Otherwise define how uncertain operations are investigated before a retry can repeat a charge, notification, or fulfillment action.

Gate: kill the process at each handoff in a test environment. Accepted work must either complete or appear in a recoverable, owned state. A log entry is not a durable work record.

5. Compare behavior without duplicating effects

The migration owner defines cohorts by a stable operation identifier and records the selected path. Do not execute a live synchronous call and an event consumer against the same external effect merely to compare them.

A shadow consumer may validate schemas or calculate a proposed result in an isolated store. Disable production credentials and outbound actions so a configuration mistake cannot charge, email, or fulfill twice. Compare proposed decisions with the authoritative result, including disagreements and missing records.

If migrating from direct calls to queued execution, first establish durable intent and status. Then route selected new operations to one executor. If migrating back, keep durable operation identity and recovery where required; a direct call does not remove ambiguous outcomes.

An illustrative export workflow can return a completed file synchronously for small requests while retaining durable jobs for large exports. The decision belongs to a versioned admission rule, not a user guessing which button is safe. Both paths must produce the same authorized content and traceable final status.

6. Cut over, drain, and preserve reversibility

Use an explicit checkpoint rather than a fixed percentage or waiting period.

"type": "flow", "title": "Retire a queue only after its responsibilities are covered", "steps": [ ], "caption": "Changing the route for new work does not move in-flight operations. Rollback must preserve their executor assignment and reconcile any uncertain effects." }} />

Before cutover, test the candidate path with representative burst and outage scenarios. Keep schemas compatible with in-flight messages and delayed retries. A consumer that looks inactive may serve a reporting, audit, or recovery process on a different schedule.

During cutover, monitor both new-path completion and old-path drain. Count pending, running, failed, expired, and completed operations against the authoritative operation inventory. Stop expansion if outcomes diverge, work is orphaned, or downstream protection fails.

Rollback changes routing for new operations. It must not blindly replay already completed work or switch an uncertain operation to another executor. Pause dispatch where needed, preserve the operation record, and let the recovery owner resolve its state.

Do not delete the old queue, schemas, or recovery tooling until the retention and reconciliation obligations are met. Removal of infrastructure needs a separately approved, exact target list.

7. Test failure and recovery before declaring simplification

| Exercise | Expected outcome | Accountable owner | | --- | --- | --- | | Producer crashes after commit | Durable intent is eventually found and published | Producer owner | | Consumer crashes after an external call | Unknown outcome is reconciled without an unguarded repeat | Consumer owner | | Duplicate or older message arrives | Duplicate is harmless; stale transition is rejected or explicitly reconciled | Data owner | | Dependency remains unavailable | Intake and retries respect capacity, expiry, and user status rules | Service owner | | Poison message fails repeatedly | Isolated failure has evidence and an authorized repair path | On-call owner | | Rollback occurs with work in flight | Old and new operations retain one executor and traceable status | Migration owner |

Replay should be a controlled operation with a selected scope, reason, authorization, and expected effects. Fixing a parser does not automatically authorize re-sending every historical message.

8. Define ordering and delivery semantics at the business boundary

Broker terminology is not the user contract. Apache Kafka's design documentation on delivery semantics explains producer, broker and consumer responsibilities and the scope of exactly-once features. Even where the platform prevents duplicate Kafka writes, an external API or database side effect may still require its own idempotency and reconciliation.

Choose the ordering key from the invariant. An account sequence, inventory item or workflow instance may need ordered transitions, while global ordering would reduce concurrency without adding correctness. Record what happens when two independent keys interact or when a late event refers to an older entity version.

For each operation, state whether delivery can be repeated, whether processing can be repeated, and whether the external effect can be repeated. These are separate questions. Store stable operation identity long enough to cover expected retries and replay. If deduplication expires, the replay procedure must know that old work may no longer be safe to resubmit automatically.

Dead-letter handling is not completion. AWS documents dead-letter queue behavior and retention considerations for SQS. Whatever broker is selected, a failed operation needs an owner, evidence, permitted repair, retention policy and path back to processing or final rejection. A growing dead-letter queue can represent unfulfilled customer work.

Test schema and semantic evolution with delayed messages. A compatible parser may still apply a value according to old business meaning. Retain the producer version or event contract needed to interpret the record, and decide how long consumers must support it.

9. Protect the asynchronous path as a security boundary

Authenticate producers and consumers separately. Authorize which identities can publish each command or fact, which can read it, and which can replay or purge it. Possession of broker credentials should not grant permission for every tenant or operation.

Carry the minimum context needed to enforce policy and trace the operation. Do not trust a tenant identifier supplied by an end user or a model without binding it to authenticated context. Avoid placing secrets or unnecessary personal data in message bodies, headers, partition keys or diagnostic logs.

Encrypt and restrict retained messages, retry stores, dead-letter queues and comparison data according to their content. Record retention and deletion behavior for every copy. If a user or tenant loses access, test whether delayed work can still deliver now-forbidden content or act with stale permissions. Recheck authorization at execution when the business action requires current access.

Replay and repair interfaces are privileged operations. Require exact scope, reason, approver where appropriate, dry-run or preview evidence, rate controls and retained results. A broad “replay all” button can duplicate effects, overload dependencies or resurrect data that should remain unavailable.

10. Compare the operating burden before simplifying

Build an ownership ledger for the producer, broker or queue, schema, consumer, operation status, dead-letter path, replay tooling, observability and external dependency. Record support hours and escalation. Removing a broker can reduce components but may move durability, admission control and recovery into the caller or database.

Run a two-hour or workload-appropriate dependency outage in an authorized environment. Measure accepted work, oldest age, expiry, user status, recovery rate, downstream pressure and operator actions. Then run the synchronous alternative under the same failure: observe timeouts, connection or thread use, retries, admission and unknown outcomes.

Price both designs with current workload assumptions. Include infrastructure, retention, transfer, support, schema and replay maintenance, application capacity during dependency slowdown, and incident effort. Do not count existing shared broker cost as avoidable if other workloads still need it. Do not call the direct path cheaper if it requires much more peak capacity or produces additional failed work.

The decision record should state which responsibility is removed, which responsibility moves, and who accepts it. Simplification is credible when fewer components also mean fewer unresolved failure states and a supportable recovery model.

Set a post-change review trigger

Review the boundary after representative peaks, at least one downstream incident, a schema change, or a material ownership change. Compare actual backlog, timeout, retry, support and recovery behavior with the assumptions used in the decision.

If direct calls begin accumulating local retry queues, background workers or ad hoc status tables, the asynchronous responsibilities may have returned without an explicit design. If the queue remains but every operation requires immediate synchronous confirmation, the original user contract may have changed. Reopen the decision rather than adding compensating complexity silently.

Retain the prior architecture and migration evidence for the agreed recovery window. Remove obsolete credentials, topics, status paths and alerts only after in-flight work and retention obligations close.

Reusable boundary decision record

Record the operation and user promise; authoritative state; current failure; candidate designs; normal and burst workload; required durability; ordering scope; duplicate strategy; timeout and expiry behavior; consumers; owner; operating cost; test evidence; migration routing; rollback conditions; unresolved assumptions; and a reassessment trigger.

Attach one worked operation trace that includes a failed attempt and recovery. This is more useful than a diagram that shows only the successful path.

"The user-visible meaning of accepted, completed, rejected, and unknown is explicit.", "The selected boundary has a measured benefit and a named operating owner.", "Durable handoff and duplicate handling survive tested crash points.", "Shadow execution cannot create external production side effects.", "Burst and outage tests protect downstream capacity and business deadlines.", "Cutover and rollback keep one executor per operation and reconcile in-flight work.", "Queue retirement preserves required consumers, retention, and replay obligations." ]} />

Limitations and review triggers

This decision does not certify a broker configuration or prove end-to-end exactly-once behavior. Revisit it when the user promise, traffic shape, dependency reliability, retention need, or ownership model changes. The right simplification removes a responsibility that is no longer needed, or moves it to a place that can demonstrably fulfill it.