Microservices Communication: Contracts, Delivery Boundaries and Recovery

A decision framework and worked order-reservation model for choosing synchronous calls, asynchronous messages, idempotency boundaries and compensating actions.

Decision brief

Audience: backend leads, staff engineers and service owners responsible for an operation that crosses independently deployed systems.

Decision: which parts of an operation must finish before the caller receives a response, which may progress asynchronously, and who repairs an operation that stops between those boundaries.

Thesis: a protocol choice cannot establish a business invariant. First define acceptance, completion and recovery. Then choose the transaction and communication mechanisms that can implement those meanings under failure.

This paper develops an illustrative order-and-reservation workflow. It is not a customer engagement, performance benchmark, compliance assessment or account of Ampity delivery. The output is a contract ledger, failure-test matrix and small executable state model. The model is deliberately narrower than a deployable messaging system.

An API gateway governs an ingress boundary. This paper governs work after ingress, including the period when no caller is connected. The companion API gateway architecture paper owns routing and policy decisions, not the order's business completion state.

1. Start with the invariant, not the transport

Assume a product accepts an order only after durably recording its identity and intent. Acceptance does not promise available inventory. A later reservation decision either confirms the order or records a rejection. A cancellation must not silently turn an already shipped order into an unfulfilled promise.

These are proposed requirements for the teaching scenario, not universal commerce rules. A product that promises inventory at checkout might need a synchronous reservation or a different transaction boundary. A product that cannot expose a pending state may be a poor fit for the asynchronous design below.

Before splitting services, ask whether the relevant data and ownership can remain in one transactionally managed application. A modular monolith can still have unreliable dependencies; a microservice can still use local ACID transactions. Deployment topology alone establishes neither reliability nor consistency.

Record the owner who can change each invariant. Product owns what accepted means to a customer. The inventory owner defines reservation expiry and contention behavior. Operations owns detection and recovery procedures, with authority bounded by the business policy. No broker setting resolves a disagreement among these owners.

2. Write the contract ledger

Use a ledger before an interface specification. It ties a message's technical shape to an observable product state.

| Boundary | Meaning of success | Ambiguous result | Required recovery evidence | | --- | --- | --- | --- | | Submit order | Order and outbound intent are committed together | Caller times out after commit | Lookup by stable request identity returns the existing outcome | | Publish intent | Configured broker confirms its responsibility | Connection fails before confirmation is observed | Relay retries the same event identity | | Reserve inventory | Reservation and deduplication record commit together | Worker commits, then fails before acknowledging | Redelivery returns the stored result without reserving again | | Notify order owner | Reservation outcome is recorded in order state | Reply event is delayed or repeated | Reconciliation finds a pending operation and correlates durable state | | Cancel order | Cancellation policy reaches a recorded terminal state | Release action fails or shipment already started | Explicit exception state and authorized resolution |

An idempotency key needs a scope, retention period and request fingerprint. Reusing the same key for a different item or quantity must be a conflict, not a silent replay. Retention shorter than the retry or replay window creates a second-execution risk. Deleting deduplication records therefore belongs in the contract, not merely in a storage-cleanup job.

The contract must also cover authorization. An authenticated caller cannot reuse another tenant's request identity to discover an order. Bind identity lookup and conflict responses to the same tenant and permissions as the original operation.

3. Choose synchronous calls for the answer the caller needs

A synchronous eligibility read can be useful when its answer is needed immediately and the dependency can meet the caller's budget. Propagate a remaining deadline, bound concurrency and decide what a timeout means. A timeout is uncertainty about the result, not proof that the downstream operation did nothing.

gRPC supports unary and streaming call forms. Its deadline behavior must be configured and propagated appropriately; server work should respond to cancellation. These are API capabilities, not evidence that one application will outperform another. gRPC core concepts and deadline guidance describe the relevant mechanisms.

REST is an architectural style, not a serialization format or an HTTP-version restriction. A JSON-over-HTTP endpoint, a binary RPC and an HTTP streaming endpoint have different contracts and operational tooling. Avoid comparisons that label all REST interfaces as non-streaming or predict a universal latency multiplier.

For a proposed protocol change, hold business logic, authentication, payload content and test environment constant. Record connection reuse, compression, concurrency, response size, CPU saturation, warm-up and failure behavior. Compare distributions and errors, not just average successful latency. A result belongs to that workload and configuration. It should not become a general performance claim.

4. Separate local commits from message delivery

The order service commits an order and an outbox entry in one local transaction. A relay publishes committed entries. This avoids a success response for an order whose publication intent was never recorded, provided the transaction and durability settings meet the stated requirements.

The relay can publish twice if it crashes after publication but before recording progress. The consumer must handle that possibility. The transactional outbox addresses one atomicity boundary; it does not create an end-to-end exactly-once business operation. AWS transactional outbox guidance describes this pattern and duplicate-handling concern.

A publisher confirmation and a consumer acknowledgement establish different boundaries. Broker durability, queue type, replication, retention, routing and acknowledgement settings still matter. Inspect failed routing, expired messages and exhausted storage explicitly. RabbitMQ's confirmation documentation distinguishes publisher responsibility from consumer processing acknowledgements.

5. Test duplicate identity before adding infrastructure

The following dependency-free JavaScript model is a teaching fixture. Run it as an ES module with Node.js. It demonstrates duplicate handling, conflicting identity and a release transition. Maps are not durable storage and the function is not a concurrency-safe transaction implementation.

~~~js

const inventory = { available: 2, requests: new Map() };

function release(key) { const result = inventory.requests.get(key); if (!result) throw new Error("unknown reservation"); if (result.state === "reserved") { inventory.available += result.quantity; result.state = "released"; } return result.state; }

assert.equal(reserve("order-1", "item-A", 1).state, "reserved"); // Model a lost acknowledgement: repeat after successful processing. assert.equal(reserve("order-1", "item-A", 1).state, "reserved"); assert.equal(inventory.available, 1); assert.throws(() => reserve("order-1", "item-A", 2), /identity conflict/); assert.equal(release("order-1"), "released"); assert.equal(release("order-1"), "released"); assert.equal(inventory.available, 2); assert.equal(reserve("order-1", "item-A", 1).state, "released"); ~~~

The final assertion is important: replaying the original request does not recreate a reservation after release. A new business attempt needs a new permitted identity. In a real system, state transitions need concurrency control, authorization, persistent deduplication and a transaction that prevents stock updates from becoming detached from request records.

This fixture does not cover parallel deliveries, process crashes during commit, distributed ordering or storage corruption. A passing fixture is decision evidence about a small state machine, not production evidence about a queue.

6. Make compensation a business decision

If a later step fails, releasing a reservation may be appropriate. Refunding a settled payment, reversing a shipment and releasing stock are different actions with different owners. Some actions cannot restore the original state. Compensation can itself fail and require escalation. Microsoft's compensating transaction pattern makes these limitations explicit.

Define states such as pending, reserved, rejected, cancellation-requested, released and manual-review. Name the transitions that are forbidden after shipment begins. Store the reason, actor and correlation identity for an exception without logging payment secrets or unnecessary personal data.

A workflow deadline should trigger investigation or a defined cancellation policy. It should not automatically erase evidence of unfinished work. Repeated compensation is also an idempotency problem. The recovery interface needs its own authorization and audit trail because it can change real business state.

Two-phase commit is not impossible across services. PostgreSQL supports prepared transactions for external transaction managers. The coordinator, locks, failure recovery and availability implications must be evaluated. PostgreSQL 17 prepared transactions documents the mechanism and operational cautions. Keeping one local transaction may be simpler than either a distributed transaction or a saga.

7. Bound retries and operational load

Retry only when the operation's semantics and remaining deadline permit it. Decide which layer owns retries so a caller, gateway, client library and worker do not multiply attempts. Use backoff, jitter and explicit concurrency limits appropriate to the dependency and workload.

A circuit breaker may reduce calls to a failing dependency, but it does not repair previously accepted work. A dead-letter queue makes some failures visible, but it is not a completed recovery process. Assign a response owner, inspection permissions, retention policy and replay contract. Quarantine an invalid schema instead of endlessly cycling it.

Budget capacity for replay and repair as well as normal traffic. A large backlog can compete with current orders for the same database connections. Test a bounded replay rate and stop conditions using realistic contention. Include broker storage, duplicate processing, cross-region transfer and operational staffing in the cost decision. “Asynchronous” does not mean inexpensive or unconstrained.

8. Treat a mesh as infrastructure with its own boundary

A mesh may centralize parts of identity, encryption, routing and telemetry. It does not automatically establish application authorization, request idempotency or a correct reservation policy. Failure of certificate issuance or policy distribution can create a new availability problem.

Istio documents both sidecar and ambient data-plane modes. Traffic interception and policy placement depend on the chosen mode; applications do not universally send requests to localhost. Review the deployed release, interception exclusions, identity binding, certificate rotation, protocol support and fail-open or fail-closed behavior. Istio data-plane modes.

Compare a mesh with simpler client-library and platform controls. A useful evaluation records the maintenance owner, debugging path, resource overhead and response to partial rollout. Adopt the tool only when its boundary is understood and its operating burden has an owner.

9. Build the failure-test matrix

| Injected failure | Required observation | Stop or repair action | | --- | --- | --- | | Timeout after order commit | Same identity resolves to one accepted order | Return existing status; investigate missing acknowledgement | | Relay restart after publication | Duplicate event does not create duplicate stock use | Inspect deduplication and progress checkpoints | | Consumer crash before commit | No partial reservation survives | Redeliver under the tested transaction configuration | | Consumer crash after commit | Replayed event returns committed outcome | Acknowledge only after durable processing | | Outcome arrives after cancellation | Versioned transition rejects an invalid state change | Reconcile or escalate without losing the event | | Broker retention expires during outage | Missing range is detectable | Rebuild from retained authoritative intent or declare recovery gap | | Authorization changes before replay | Recovery respects current approved policy | Hold for an authorized decision, not silent privileged execution |

Run these tests against the actual driver, database, broker, replication and deployment versions. Correlate order identity, event identity and attempt identity. A trace can help explain a request path, but sampling, asynchronous gaps and retention mean traces are not a complete ledger of business state.

Record test data, timing, observed states and the operator action required. A successful final count without intermediate-state evidence can conceal duplicate effects that were later manually corrected.

10. Reusable decision record and rollout gates

For each operation, complete this record:

Customer-visible acceptance and completion definitions.
Authoritative state owner and local atomicity boundary.
Deadline, retry owner, idempotency scope and conflict response.
Delivery acknowledgement, retention and replay assumptions.
Ordering requirements and late-event policy.
Compensation authority, irreversible action and manual-review path.
Security, cost and capacity constraints for normal and recovery traffic.
Failure-test evidence, open questions and named decision owner.

Introduce the design on a bounded operation with observable status. Test old and new producers against the contract before changing traffic. Keep the old path only if it can safely interpret the resulting state. Routing traffic back does not undo reservations or externally completed actions.

At each gate, compare pending age, rejected transitions, duplicate conflicts and reconciliation differences with an agreed baseline. Thresholds must be chosen for the product, traffic pattern and support capacity. This paper supplies no universal success-rate or latency target.

11. Compare the complete communication options and trade-offs

Do not begin with “REST or events.” Compare the transaction and operating boundary required by the business operation. The same product can use a local transaction for one invariant, a synchronous call for a required immediate answer, and asynchronous work for a process that can remain pending.

| Option | Strong fit | Main trade-off to own | Evidence before adoption | | --- | --- | --- | --- | | One local transaction | Data and invariant can share one authority | Coupling inside one deployment and store | Transaction, lock, recovery and module-boundary tests | | Synchronous service call | Caller needs an immediate answer within a bounded deadline | Dependency latency, availability, admission and unknown outcomes | Deadline, retry, overload and post-timeout status tests | | Durable work queue | Work can remain pending and buffering protects a constrained dependency | Backlog, duplicate execution, expiry, status and replay operation | Crash-point, burst, outage, drain and reconciliation tests | | Published domain event | A committed fact has independent consumers | Schema lifecycle, consumer inventory, retention and replay | Version compatibility, duplicate, ordering and consumer recovery evidence | | Orchestrated saga | One owner must coordinate a multi-step process and compensations | Coordinator state, participant contracts and failed compensation | State-machine, timeout, compensation and manual-resolution exercises | | Choreographed events | Participants react without one central workflow owner | Emergent coupling, difficult global status and replay consequences | Consumer map, invariant ledger and end-to-end recovery ownership | | Distributed transaction | Participants and infrastructure support the required atomic protocol | Availability, locks, coordinator recovery and operational complexity | Version-specific failure, recovery and capacity tests |

Apply hard constraints before preferences. If the customer cannot be told “pending,” an asynchronous submission may not meet the product contract. If an action is irreversible, a compensation narrative cannot make it reversible. If a single database can enforce the invariant with acceptable change and scale, splitting it may add failure states without adding useful independence.

Then compare operating work. Count schemas, credentials, queues or topics, replay tooling, dead-letter handling, reconciliation, dashboards, certificates, upgrades, on-call paths and support procedures. A lower request latency or simpler code sample can be outweighed by a recovery model the team cannot sustain.

Evaluate mixed designs deliberately. The order service can return a durable pending identity synchronously, publish processing intent asynchronously, call an inventory read synchronously and record the reservation outcome through an event. Each boundary needs one meaning and owner. Avoid using several mechanisms for the same business transition without an explicit authority rule.

The decision output is not a protocol winner. It is a map showing which user promise, invariant and failure response each mechanism implements, plus the evidence that the organization can operate it.

12. Treat identity and authorization as part of every message contract

Authentication at an API gateway does not authorize a downstream service to act on every order. Define the trusted identity presented to each service, how tenant and user context is bound, which service owns the resource decision, and which actions require current authorization rather than the permission captured at submission.

For commands, derive scope from authenticated context and verify object and action permissions before durable acceptance. Do not let a user-supplied tenant ID select a partition or authorization filter without trusted binding. For events, distinguish who is permitted to publish the fact from who may consume its fields. A valid broker credential is not evidence that the publisher can create that business fact.

Delayed execution raises a policy question. A user may submit an export and lose access before the worker runs. Decide whether execution uses the authorization captured at acceptance, rechecks current access, or requires a privileged service policy. Record that choice for the specific action. Sensitive exports and destructive changes often require current authorization or an explicit service-owner decision.

Keep secrets, access tokens and unnecessary personal data out of messages and correlation labels. Encrypt the transport and protected storage according to the data class, but remember that authorized consumers see plaintext. Limit retention, dead-letter access, replay exports and diagnostic copies. A replay environment should not become an uncontrolled secondary data store.

Operator and repair paths need stronger controls. Replaying a message, changing workflow state, releasing a reservation or issuing compensation can create real customer and financial effects. Require exact scope, reason, permitted transitions, actor identity and retained outcome. Where dual approval is necessary, make it part of the tool rather than an informal chat instruction.

Test negative cases through the deployed path:

another tenant reuses a valid operation identity;
an old event carries permissions that are now revoked;
a support role tries to invoke a customer-only transition;
a worker receives a message for an unauthorized region or account;
a replay attempts to execute an already completed external effect; and
audit delivery fails while an operator action succeeds.

Contain an authorization failure by stopping the affected consumer or route while preserving evidence. Do not broadly disable policy to restore throughput. Reconcile any operations processed under the wrong scope and invoke the relevant incident process.

13. Version schemas and business meaning together

Schema compatibility tools can show that a field remains parseable. They cannot establish that its meaning, default, authorization or timing remains safe. Maintain a contract record for each message or API: owner, version, producers, consumers, field meaning, invariants, retention, ordering scope and retirement conditions.

Adding an optional field can still break a consumer that rejects unknown properties or assumes a closed enumeration. Removing a field can break a delayed event retained for replay. Changing a default can alter writes without changing the schema. Test supported producer and consumer versions against representative behavior, including errors and late delivery.

Use an additive transition when feasible:

consumers tolerate and understand the new representation;
producers begin supplying it while preserving old semantics;
deployed consumer evidence shows the old representation is no longer required;
retained messages and replay windows are assessed; and
the old field or version is removed under a separate retirement decision.

For state changes, include an entity version or equivalent concurrency rule where the domain needs it. A late “reserved” outcome must not overwrite a later cancellation or shipment. The receiving owner decides whether to reject, reconcile or escalate the stale transition. Broker order alone may not protect records that travel through different partitions or recovery paths.

Maintain a consumer inventory with deployed versions and owners. Runtime observation helps, but a quiet monthly job or disaster-recovery consumer can be invisible during a short window. Retirement requires ownership, contract and retention evidence, not only declining traffic.

When a schema or semantic error reaches production, stop incompatible publication or isolate the consumer, preserve the invalid record, and use a reviewed repair. Do not mutate historical messages in place without provenance. Repaired or republished messages need a new attempt identity linked to the original.

14. Rehearse an uncertain outcome from detection to reconciliation

Consider the order scenario at the hardest point: inventory commits a reservation, the outcome event is delayed, and the order owner remains pending. A retry of the reservation command arrives while support is investigating. The architecture must make the real state discoverable without reserving stock twice.

Run this as an operator exercise. Give the responder the aged-order alert, order identity, broker evidence and normal runbook. They should locate the authoritative reservation, prevent duplicate execution, choose republish or reconciliation, and document the result. Do not provide direct database edit instructions as the default repair.

Measure the time to a trustworthy state, not merely the time to restart the consumer. Record missing permissions, inaccessible logs, ambiguous identifiers and replay controls that are too broad. These are operating-model gaps that must close before the workflow is accepted.

Repeat the exercise with a stale outcome that arrives after cancellation. The order service should reject or escalate the invalid transition while retaining evidence. The inventory owner then decides whether release is permitted. This proves that message delivery and business-state acceptance are separate gates.

15. Operate the workflow with service-level evidence

Define indicators for the business process: accepted operations, completion latency, oldest pending age, terminal rejection, unknown outcomes, duplicate conflicts, compensation backlog and reconciliation age. Broker depth and consumer CPU explain mechanisms but do not replace the workflow view.

Assign alerts to decisions. An old pending order may require reconciliation. A rising duplicate-conflict rate may indicate client misuse or lost response behavior. A growing compensation backlog may require stopping new intake. A broker storage alert may require capacity or admission action before retention expires.

Monitor the telemetry pipeline itself. Missing correlation, failed audit delivery or sampled traces can make the workflow appear healthy. The durable operation records remain authoritative for business state. Use traces and logs to investigate, not as the only ledger of accepted work.

Review cost across normal and recovery traffic: broker storage, replicas, cross-region transfer, retries, dead-letter retention, replay capacity, databases, telemetry and on-call work. A design that reduces caller latency by creating an expensive permanent backlog may not meet the product objective.

Run a periodic reconciliation that compares order and inventory terminal states, not just message counts. Every discrepancy has an age, owner and permitted repair. A queue emptied by deleting messages is not evidence that the business work completed.

16. Use a production-readiness checklist and a narrow next step

"Acceptance, completion, rejection, cancellation and unknown outcomes have product-owned meanings.", "Every local transaction and durable handoff has one authoritative state owner.", "Synchronous calls have deadline, admission, timeout and post-timeout status behavior.", "Asynchronous delivery has durable intent, duplicate handling, ordering scope, expiry and replay ownership.", "Operation identities are tenant-bound, fingerprinted and retained through the retry and replay window.", "Schema and semantic compatibility are tested across deployed and retained versions.", "Compensation, manual repair and replay are privileged, audited business actions.", "Failure exercises cover commit gaps, duplicate delivery, stale outcomes, broker retention and failed compensation.", "Workflow indicators expose aged pending and unknown operations, not only infrastructure health.", "A receiving operator can reconcile one uncertain outcome without undocumented specialist access." ]} />

Begin with one consequential operation. Complete the contract ledger, mark every commit and acknowledgement boundary, choose the smallest viable communication design, and run the uncertain-outcome exercise. Do not standardize a broker, mesh or saga framework across every service until this one operation has demonstrated an operable recovery model.

17. Preserve a communication decision packet

The final packet should let a new reviewer reconstruct why the boundary exists and how it fails. Include the user promise, business invariant, authoritative stores, request and event contracts, producer and consumer versions, operation identity rules, local commit points, acknowledgement meaning, deadlines, retry ownership, ordering scope, retention, replay, compensation, security controls and operating owners.

Attach the options comparison and rejected alternatives. State whether a local transaction, synchronous call, queue, published event, saga or distributed transaction failed a requirement or merely carried a less attractive tradeoff. Record the condition that would justify revisiting it. This prevents architecture style from replacing workload evidence.

Include at least one successful trace and one failed operation from the exercises. The failed example should show intermediate durable states, not only the final repair. Link test configuration, driver, broker, database and deployment versions so the result is not generalized beyond its context.

Name the last safe rollback point and the recovery after it. If new state or external effects make the old path stale, say that the path is forward recovery or reconciliation. Do not keep a “rollback” button that cannot preserve current business state.

Close every temporary component: shadow topics, comparison stores, debug sampling, expanded credentials, adapters and dual-running capacity. Where retention or recovery requires one to remain, record its owner, access, cost and removal gate.

The accountable product, service, data, security and operations owners review the parts they control. Editorial approval does not substitute for that system-specific acceptance. Set the next review trigger against changed semantics, new consumers, repeated unknown outcomes, schema churn, support burden or a failed recovery exercise.

At that formal review, replay the hardest known production-like failure with current deployed versions and normal operator access. If the team can no longer independently locate authoritative business state, contain duplicate execution, or reconcile external effects, pause further expansion and formally reopen the boundary decision before adding more consumers.

Limitations, evidence and next step

The scenario and fixture are illustrative engineering material. They establish no customer association, measured improvement or compliance result. Primary sources were checked on 21 September 2026; living product documentation still requires a deployment-specific version review.

Publication remains blocked on an accountable author and distributed-systems reviewer, environment-specific failure tests, source verification and authorized editorial approval. No downloadable engagement report or validated benchmark is attached. The source-migration metadata is not proof of authorship.

For a scoped backend systems and API review, bring one operation's contract, dependency map, anonymized failure traces and current recovery runbook. The first deliverable should be a boundary and test plan with explicit owners, not a promised performance improvement.