Event-Driven Systems That Scale

Design event workflows with explicit consistency boundaries, atomic duplicate handling, bounded retries, compatible schemas, and controlled replay.

trigger="A business workflow needs asynchronous processing, independent consumers, or recoverable work beyond a request deadline." owner="The service owner accountable for the workflow's final business state." participants={["Producer and consumer owners", "Database engineer", "Messaging platform owner", "Operations lead", "Security and data owners", "Business process owner"]} prerequisites={[ "A named workflow, its consistency requirements, and the acceptable delay and failure outcomes.", "Known producer, consumer, broker, database, and external-provider contracts.", "A safe test environment with failure injection, representative events, and approved replay access." ]} outputs={[ "An event contract and decision record defining ordering, retention, and transaction boundaries.", "A tested producer handoff, atomic consumer duplicate guard, and external-effect ledger.", "A retry, quarantine, replay, and reconciliation runbook with operational evidence." ]} doneWhen={[ "Concurrent duplicates and crashes at each commit boundary preserve the agreed business invariant.", "Unknown external outcomes are reconciled instead of blindly repeated.", "Consumers meet workload-specific completion objectives during normal demand and recovery.", "Operators can stop, inspect, and resume replay without unapproved duplicate effects or data exposure." ]} />

Start with the business invariant

An event records something that happened. A command asks a system to do something. Decide which contract the workflow needs before choosing a broker. A command accepted into a queue is not proof that its business action completed.

Use asynchronous processing when the caller can tolerate delayed completion and the team can operate durable state, retries, and reconciliation. Keep a synchronous boundary when an immediate authoritative decision is required and the dependencies can meet it. A hybrid can accept a request synchronously and expose the later operation's status.

Write the invariant in business terms: an order must not be charged twice for the same payment operation; a projection must eventually reflect an accepted update; a notification must not go to the wrong tenant. The required controls differ. A throughput number or “exactly once” label does not define correctness.

This playbook does not require event sourcing or CQRS. Those are separate design decisions. Introduce them only when their benefits justify additional state, evolution, and recovery responsibilities.

1. Define the event and operating contract

The producer and consumer owners agree meaning, ownership, allowed readers, schema, identifiers, time semantics, maximum size, ordering scope, and retention. Avoid placing secrets or unnecessary personal data in events that will be widely copied and replayed.

| Decision | Required answer | Acceptance evidence | | --- | --- | --- | | Completion | What business state counts as complete? | End-to-end assertion, not just broker acknowledgment | | Delay | How long can this work remain pending? | Event-age and completion measurements | | Ordering | Which entity or operation must be sequential? | Out-of-order and missing-event tests | | Duplication | What identifies the same operation? | Concurrent redelivery tests | | Recovery | What can be replayed, for how long, and by whom? | Controlled replay and reconciliation exercise | | Loss and retention | What failure or expiry is tolerable? | Documented durability and recovery assumptions |

Choose retention from outage recovery, replay needs, privacy constraints, and storage cost. Verify the selected broker's current configuration, quotas, and regional behavior. Do not select a service from a timeless vendor table of throughput or retention ranges.

Gate: the business owner accepts the pending, failed, expired, and compensated outcomes. If the only acceptable state is an immediate cross-system atomic result, revisit the boundary before adding a queue.

2. Make the producer's database-to-broker handoff recoverable

When a state change and its event must agree, store the state change and an outbox record in the same local database transaction. A relay publishes committed outbox records and records progress. Design its retry, ordering, retention, and alerting behavior explicitly.

AWS's transactional outbox guidance explains the dual-write problem and warns that duplicates remain possible. The local transaction avoids one split between database state and publication intent; it does not guarantee successful downstream completion under every outage.

A relay can crash after the broker accepts an event but before publication progress is saved. Repeating publication is therefore a normal case. Use stable event IDs, monitor the age of the oldest unpublished row, and retain enough evidence to investigate gaps.

For change-data capture, test connector checkpoints, log retention, schema changes, restart, and recovery after a prolonged outage. A connector does not remove operational responsibility. If the required source log has expired, a controlled resnapshot or reconciliation may be needed.

3. Make the duplicate guard and local effect atomic

A query that checks whether an event was processed, followed by a separate update, races when two consumers receive the same event concurrently. Both can observe “not processed” and apply the effect.

For a local database effect, use a durable unique guard scoped to the consumer's effect, tenant where applicable, and stable event identity. Insert that guard and apply the business change in the same transaction. A rollback must undo both.

In PostgreSQL, INSERT with ON CONFLICT and RETURNING supports an insert-or-ignore guard backed by a unique constraint. The application must branch on whether the guard was inserted; an ignored insert followed by an unconditional business update is still incorrect.

Receive and validate the event identity, tenant, schema, and allowed source.
Begin a database transaction.
Attempt the unique guard insert for this consumer effect and event.
If this transaction inserted the guard:
  Validate the permitted business transition.
  Apply the local state change.
  If needed, write an external-effect intent in this same transaction.
Commit.
If the guard already existed:
  Confirm the existing committed operation has the expected identity and payload.
Acknowledge only after the transaction or duplicate check succeeds.
On transaction failure: roll back and apply the classified retry policy.

This is control-flow pseudocode, not drop-in database code. Test the chosen isolation level, unique constraint, conflict handling, lock timeout, and transaction retry behavior. Compare a stored payload fingerprint or equivalent immutable identity when the same event ID could arrive with different content; quarantine conflicting reuse rather than silently accepting it.

Keep guard retention aligned with the longest permitted redelivery and replay window. Deleting guards while old events can still be replayed reopens the duplicate-effect risk. Different consumer effects need distinct scopes so one consumer cannot suppress another's legitimate work.

4. Separate broker semantics from external side effects

Kafka's delivery-semantics documentation describes transactional processing within Kafka and the cooperation needed from external destinations. Broker transactions do not automatically include a payment API, email service, or unrelated database.

Amazon SQS documents possible duplicate delivery for standard queues. Consumer correctness must match the actual selected delivery mode and failure behavior, not a marketing shorthand.

For an external effect, keep a durable operation ledger with the stable provider idempotency key, request fingerprint, attempt state, provider reference, and reconciliation status. Do not hold an ordinary database transaction open while waiting on the network as a substitute for distributed atomicity.

A timeout after submitting a charge is an unknown result, not proof of failure. Query the provider's operation status or reconcile its records before deciding to repeat. Stripe's idempotency contract documents parameter checks and finite key retention; confirm equivalent details for the provider you use.

If the provider cannot deduplicate or expose a reliable status lookup, document the residual ambiguity and the manual reconciliation path. Do not claim end-to-end exactly-once effects that the boundary cannot support.

5. Choose ordering and scaling from measured workload

Order only the events that require a shared business sequence. An entity key can group related work, but a heavily used entity can still create a hot partition. A globally ordered stream constrains concurrency and may not be necessary.

For each consumer, define how it handles an older version, a missing predecessor, a duplicate, and an unexpected future version. Options include conditional state transitions, a bounded wait for a gap, or reconciliation against an authoritative record. Avoid treating arrival time as authoritative business order.

Load-test representative event sizes, key skew, processing times, and downstream limits. Measure completion age and the time needed to drain an outage backlog while new traffic continues. A fixed event-count alert can mean seconds for one service and days for another.

Set concurrency from the actual broker and consumer model, database connection budget, external rate limits, and handler safety. Increasing workers can make a constrained dependency slower. Bound in-flight work and memory, and test pause or admission controls before a backlog becomes an incident.

6. Classify retries and quarantine without hiding lost work

The consumer owner distinguishes transient dependency failures, malformed or incompatible events, denied access, and invalid business transitions. Use bounded retries with delay and jitter where appropriate. Choose the budget from the completion objective and dependency recovery behavior, not a universal number of attempts.

A quarantine or dead-letter destination is an owned work queue, not successful completion. Record original identity, source position, first and last failure, handler version, error category, and repair status. Restrict access if the payload contains sensitive data.

If ordering is required, define the consequence of moving one event aside. Continuing past it may violate the invariant. Pause the relevant entity or partition, or use an approved reconciliation strategy. Never silently skip a poison event merely to make lag decrease.

Escalate based on age, impact, growth, and recovery capacity. Make sure the alert reaches a team that can inspect and act on the failed work.

7. Evolve schemas across both live and retained events

The contract owner tests new producers against existing consumers and new consumers against retained historical events. “Add an optional field” is not a complete compatibility policy: consumers may reject unknown fields, enum additions may break exhaustive logic, and defaults may differ.

Select the schema registry's compatibility mode for the actual format and reader/writer direction, and verify it with representative payloads. A registry checks its defined rules, not the full business meaning of an event.

Deploy compatible readers and writers in a sequence justified by those tests. Do not make a field required only because current producers send it; retained events may still lack it. Keep an adapter, explicit version path, or migration procedure where needed.

Use a documented envelope. For example:

{
  "eventId": "example-event-001",
  "eventType": "order.confirmed",
  "schemaVersion": 1,
  "source": "order-service",
  "occurredAt": "2026-09-20T10:00:00Z",
  "tenantId": "example-tenant",
  "entityId": "example-order",
  "entityVersion": 4,
  "correlationId": "example-operation",
  "data": { "orderId": "example-order" }
}

This is an illustrative internal contract, not a claim of conformance to an external event standard. Define field meaning, sensitivity, uniqueness, and validation before adopting it.

8. Rehearse replay, compensation, and recovery

Test concurrent duplicates, crash before commit, crash after commit before acknowledgment, relay retry after publication, dependency timeout after an external effect, out-of-order delivery, schema mismatch, consumer restart, and backlog recovery. Verify both local state and external records.

For a multi-service saga, define compensation as a new business action, not database rollback. A refund is not the same as erasing a charge; shipped goods may not be recallable. Compensation can also fail and needs its own ownership, retries, and reconciliation.

On a bad consumer release, stop the affected processing scope, preserve source positions, and identify completed effects. Restore a compatible handler or deploy a forward fix. Do not reset offsets and replay until the owner has resolved schema, duplicate-guard, and external-effect consequences.

9. Prove the invariant with a concurrent-delivery rehearsal

Use an illustrative order-payment workflow with a sandbox provider. The expected outcome is one approved payment operation for one order, not one message delivery. The consumer owner prepares a committed order event, two simultaneous deliveries carrying the same event identity, and an unchanged request fingerprint. The database owner verifies that the duplicate guard, state transition, and payment intent share the same local transaction. The provider executor remains a separate boundary.

Run these cases before increasing concurrency. Capture the final order state, guard row, intent record, acknowledgment history, and provider records for every case. Never run a duplicate-charge test against an actual customer account without separately authorized scope.

| Failure injection | Local assertion | External assertion | | --- | --- | --- | | Both deliveries race for the guard | Only the winning transaction applies the transition and creates its intent | One business operation identity is offered to the executor | | Winner fails before local commit | Its guard, transition, and intent are absent together | No executor can act on an uncommitted intent | | Winner commits but acknowledgment is lost | Redelivery recognizes committed work without another transition | Existing intent retains its original identity | | Provider accepts but executor loses the response | Intent becomes unresolved, not falsely failed | Reconciliation finds the original effect before deciding on retry | | Same event identity arrives with different content | Conflicting reuse is quarantined for review | No new effect is authorized by the conflicting payload |

The test must distinguish transient database errors from a confirmed duplicate. A lock timeout or serialization failure is not evidence that the first transaction committed. Retry the transaction under the chosen policy and inspect durable state. Likewise, an acknowledgment failure should not erase the business evidence that makes redelivery safe.

If two different event IDs can represent the same payment operation, an event-ID guard alone is insufficient. Add a business-operation uniqueness rule at the authoritative boundary, with semantics agreed by the process owner. Repeated payment attempts after an explicit failure may be legitimate; they need a new authorized operation rather than a random new identifier used to bypass an unresolved earlier attempt.

This rehearsal is complete when each injected failure has an explained durable result and every unresolved provider operation has an owner. It does not establish protection beyond the tested provider contract, retention window, or deployment configuration.

10. Calculate a recovery budget before adding workers

The following is a hypothetical planning calculation, not a throughput result. Suppose new work arrives at 80 events per second and the tested consumer completes 100 per second under the intended recovery configuration. With a backlog of 7,200 events, spare capacity is 20 events per second. The idealized drain time is 360 seconds, or six minutes. Dividing the backlog by total consumer throughput would incorrectly ignore new arrivals.

That estimate assumes a stable arrival rate, uniform work, no retries, and sustained completion capacity. Real recovery may be slower because event sizes differ, a partition is hot, a database is constrained, or external providers throttle requests. Count completed business work, not messages pulled from the broker. If completion capacity is no greater than arrivals, there is no finite drain time under those assumptions.

The platform owner converts the estimate into a controlled experiment. Use the actual key distribution and retained schemas. Record oldest pending age, successful completions, retries, quarantine growth, database saturation, and downstream rejections as concurrency changes. Stop increasing workers when a dependency limit or the agreed service-health boundary is reached. The recovery plan may instead need temporary admission control, prioritized work, or additional downstream capacity.

Keep recovery headroom distinct from normal utilization. A system that only matches normal arrivals can look healthy until an outage leaves work it cannot catch up on. Conversely, idle workers do not prove spare capacity if an ordered hot partition or provider quota is the real bottleneck.

11. Authorize replay with a bounded manifest

Before replay, the operator prepares a manifest identifying source positions or event IDs, tenant scope, failure class, handler revision, expected effect, and excluded records. Include the duplicate-guard policy, allowed external capabilities, replay rate, stop condition, and reconciliation owner. Store a stable manifest revision so the operator can prove which scope was approved. Do not log sensitive payloads merely to make the manifest convenient.

For a projection rebuild, direct output to an isolated target where practical and disable customer-facing effects. Compare expected entity counts and selected business invariants with the authoritative source before switching readers. For a payment or notification recovery, dry-run classification is not permission to repeat the action. The process owner must approve the specific effect and resolve unknown prior outcomes first.

During replay, account separately for completed, duplicate-suppressed, quarantined, and unresolved records. These categories should reconcile to the authorized input set without treating quarantine as success. A changing source may require a cutoff or recorded revision for a meaningful comparison. Preserve a resumable checkpoint and prove the stop control works before expanding the rate.

Close the manifest only when the business owner accepts the reconciliation and all remaining exceptions have owners. The next action is one small recovery rehearsal, not a bulk offset reset. Retention limits, deleted personal data, expired provider keys, and changed business rules may make some historical effects inappropriate or impossible to repeat.

12. Keep an event recovery record and completion checklist

Workflow, invariant, and accountable owner:
Producer, consumer effect, schema, and broker configuration:
Event identity and duplicate-guard retention:
Local transaction boundary and external provider contract:
Ordering scope, maximum pending age, and recovery objective:
Failure class, retry budget, quarantine owner:
Replay manifest, permissions, and approved effects:
Handler revision, rate limit, stop signal, and operator:
Expected and observed business outcomes:
Uncertain or compensated operations and reconciliation owner:
Evidence links, unresolved gaps, and final decision:

"The business invariant and pending, failed, and compensated outcomes are defined.", "State and publication intent share a local transaction where agreement is required.", "Concurrent duplicates cannot repeat the local effect, and acknowledgment follows durable completion.", "External-effect ambiguity has a stable operation record and a tested reconciliation path.", "Retention, ordering, concurrency, and alert thresholds come from the actual workload.", "Schema tests include retained events and mixed producer and consumer versions.", "Quarantined work remains owned, visible, and controlled during replay.", "Recovery tests cover both crash boundaries and business outcomes, not only broker delivery." ]} />

The final handoff is a tested workflow and recovery contract. Broker configuration, database behavior, security review, and domain acceptance remain implementation-specific gates; this guide does not certify a production deployment.