Event-Driven Architecture: Reliable Delivery and Safe Migration

Design reliable asynchronous flows with explicit payload contracts, transactional outboxes, atomic consumer effects and a migration plan that avoids duplicate...

Decide what may complete later

An event-driven flow lets a producer record a fact and other components react without requiring them all to be available at the same instant. That can isolate failures and support independent consumers. It also creates a period in which different parts of the system know different things.

Start with the business boundary. An order may be accepted while its notification is still pending. It may not be acceptable to tell the customer that stock is reserved before that reservation exists. Define what the initial response promises and how the user sees later success or failure.

An event describes a fact, such as an accepted order. A command asks an owner to do something, such as reserve stock. A query asks for information. All can travel through messaging systems; transport alone does not determine their meaning.

This guide covers reliable delivery between components. It does not require events to be the source of truth for domain state. That separate decision belongs in event sourcing and CQRS.

Choose the boundary before the broker

Use asynchronous delivery when consumers can work later, independently or at a different rate. Keep a direct call when the caller needs the result now and can handle that dependency within its deadline. Keeping related changes in one database transaction may be simpler than introducing a distributed workflow.

For a multi-step business process, write down the states, deadline, owner and action when progress stops. A compensating action is new business work, not a rollback of time. Refunding a payment does not mean the original charge never happened.

Select infrastructure against the actual requirements: retained replay, routing, ordering scope, throughput, access control, recovery objectives and operating cost. Avoid a generic “best broker” table. Queue behavior, log retention and ordering depend on product configuration, consumer behavior and failure conditions.

A broker acknowledgement is also not evidence of a completed business action. RabbitMQ distinguishes publisher confirms from consumer acknowledgements: the two mechanisms cover different parts of the path. A production check must follow the outcome as well as message acceptance.

Write the event contract around consumer needs

Give each event a stable identity, producer, event type, schema identifier and the data required for its stated purpose. A correlation ID helps investigation but is not a substitute for an event ID or business operation key. In the CloudEvents specification, the combination of source and ID identifies a distinct event.

Choose the payload deliberately:

| Shape | Useful when | Cost or limit | | --- | --- | --- | | Selected state | Consumers need the relevant facts as they were at the event. | Repeats data; requires privacy and retention review. | | Delta | Consumers own a prior state and can enforce the required order. | Missing or out-of-order changes need recovery. | | Reference | Consumers should fetch authorized current data from its owner. | Adds a dependency; current data may differ at replay time. |

“Selected state” does not mean copying an entire customer record into every event. For an order notification, identifiers and necessary order details may be enough. Keep secrets and unnecessary personal data out of payloads and metadata. Broker retention, dead-letter storage and logs can create additional copies.

Document whether a timestamp describes business occurrence, persistence or publication. Do not use wall-clock timestamps as a total ordering guarantee. If per-order order matters, define an order key and sequence policy; concurrent consumers and retries still require enforcement.

Test schema changes against actual consumers. Adding an optional field can be compatible, but not if a consumer rejects unknown fields or the new producer changes the meaning of an existing value. Keep representative old events as fixtures and define how long old versions must remain readable.

Exactly once has a transaction boundary

At-least-once delivery allows duplicates. At-most-once processing can lose work when acknowledgement precedes a failed effect. “Exactly once” must name the effect and the system boundary, not imply that a message can never appear twice.

Apache Kafka's 4.0 design documentation explains how transactions can combine consumed offsets with output records in Kafka. Appropriate isolation and client behavior matter. Writing to an external database or calling a payment provider requires coordination with that destination; Kafka transactions do not automatically include it.

The practical question is: if the process crashes between saving the effect and acknowledging the message, what will a retry do? A separate “check processed IDs” call followed by a write is not sufficient. Two workers can both pass the check, or a crash can leave the marker and effect inconsistent.

Worked example: an order and a stock reservation

Consider a hypothetical order service and stock service with at-least-once delivery. The order service first commits the order and an outbox record in the same database transaction. The record carries a stable event ID. A relay publishes committed outbox records and records its progress only after the broker confirms acceptance.

This follows the transactional outbox pattern. It closes the gap between a local state change and the intent to publish. The relay can still publish a duplicate if it crashes after the broker accepts the record but before progress is saved.

The stock consumer handles one event in a local transaction:

1. Claim a unique inbox key scoped to this consumer, event source and event ID.
2. If a committed claim already exists, treat the delivery as a duplicate.
3. Otherwise, validate the order and reserve stock with concurrency-safe inventory checks.
4. Save the reservation result, including a refusal when stock is unavailable, and any outgoing event in the same transaction as the inbox claim.
5. Commit, then acknowledge (ack) the incoming delivery.

Use database-enforced uniqueness and transactional state changes. A preliminary read alone does not provide that guarantee. Distinct orders can still compete for stock, so deduplication does not replace the inventory concurrency rule.

| Failure point | Durable state | Safe next action | | --- | --- | --- | | Before order commit | Neither order nor outbox exists | Retry the original command with its business operation key. | | After order commit, before publish | Order and outbox exist | Relay publishes the pending record. | | After publish, before relay progress | Broker may already have it | Publish again with the same event identity. | | Before stock transaction commits | Neither inbox claim nor effect commits | Redelivery can attempt the transaction again. | | After stock commit, before ack | Inbox claim and result both exist | Redelivery sees the claim and does not reserve twice. |

This example assumes the inbox and business effect share a transactional store. It does not make an external email, payment or warehouse API call atomic. For such an effect, persist an outgoing operation, use the destination's supported idempotency mechanism where available, and reconcile an uncertain response before issuing a potentially duplicate action. If the destination offers neither deduplication nor reliable status lookup, state that residual risk explicitly.

Retain deduplication records for the supported redelivery and replay horizon. Deleting a marker while its event can return can re-enable the effect.

Plan for stalled and invalid work

Retry only failures that may succeed later, with bounded attempts, backoff and attention to the operation deadline. A permanent schema error needs repair or quarantine, not an endless retry loop.

A dead-letter queue needs an owner, access controls, investigation evidence and a replay procedure. Moving an event there can unblock throughput but break a required sequence. Decide whether to pause the affected key, stop the partition or allow a documented gap. Do not call skipped business work successfully processed.

Watch the age of the oldest pending business operation, not just queue depth. A small queue can contain one critical order that has been stuck for hours. Combine that with processing errors, publish failures, consumer lag and a comparison of accepted orders with terminal reservation outcomes.

Migrate without activating two paths

Do not publish an event and keep an active synchronous side effect for the same operation unless both paths coordinate the business identity and ownership. Otherwise a temporary migration can create duplicate stock reservations or payments.

Use a controlled transition:

| Stage | Required evidence | | --- | --- | | Observe | Publish through a reliable local commit path; a shadow consumer produces comparisons but no external effects. | | Compare | Match source operations with shadow results, including retries, invalid inputs and late events. | | Switch | Assign one active handler per operation or cohort using an explicit routing rule and cutover record. | | Verify | Reconcile in-flight work and outcomes before retiring the old path. | | Recover | Pause new work or return routing safely; do not re-run completed effects as rollback. |

Keep a shared business operation key across old and new paths where overlap is possible. Drain or account for pre-cutover work. If the new consumer is behind, reverting the application does not erase messages already accepted or effects already performed. The recovery plan must name those cases.

Before production, test a lost acknowledgement, a consumer crash after commit, a duplicate event, an out-of-order event and a replay beyond the normal retry window. Bring the resulting state trace and unresolved boundaries to Ampity's system architecture and design service when reviewing an event-driven migration.