Microservices Communication: Contracts, Deadlines and Delivery

Choose synchronous calls, queues and streams by their completion contract. Work through acknowledgments, one retry budget, duplicate handling and service-mesh tradeoffs.

Before choosing REST, gRPC or a broker, finish this sentence: when the caller receives success, the system has done what?

The answer might be “returned a current price,” “durably accepted a report job” or “completed a transfer.” Those are different contracts. Synchronous and asynchronous mechanisms can support them, but neither makes a business outcome reliable without application-level state and recovery.

This guide covers the communication decision between existing services. If the problem is unclear data ownership or repeated joint releases, review the service boundaries first.

Choose by what the caller needs to know

| Need | Candidate mechanism and obligation | | --- | --- | | A current answer is required before continuing | Request/response, with a deadline and an explicit unavailable result | | Work can finish after the interaction | Durable job submission, status lookup and a terminal outcome | | Several consumers react independently | Published events with separate consumer progress and recovery | | Consumers must replay retained history | A log with defined retention, ordering scope and replay rules | | A continuous exchange is required | Streaming with cancellation, bounded buffers and backpressure |

A queue can absorb a temporary burst, but it also adds waiting time. A retained log can support replay, but replaying an email or payment side effect is not automatically safe. Choose the semantics before comparing product features.

Specify payload ownership, schema compatibility, authentication, authorization and trace context for every mechanism. Moving a call to a broker does not remove coupling to the event's meaning.

Compare HTTP APIs and gRPC against the workload

An HTTP API can fit public clients, browser integration and established HTTP tooling. Caching only helps when the operation and its cache controls permit reuse. A personalized response cannot be placed in a shared cache without a correct key and access boundary.

gRPC provides an interface definition, generated clients and streaming capabilities. Its performance depends on message shape, serialization, connections, proxies and client implementation. Test representative payloads, concurrency and failure behavior instead of assuming a universal latency advantage.

For either protocol, measure the whole request path. Serialization savings may be immaterial if the operation waits on a database lock. Include cold connections, large responses, cancellation and overloaded dependencies in the comparison.

For streaming, bound application queues and define when a slow peer is disconnected or paused. gRPC flow control governs receiver capacity for streaming RPCs, but application logic must still consume messages and avoid deadlock. A successful write to a stream buffer is not proof that the peer completed the business action.

Distinguish acceptance from completion

Consider a fictional report service. The API validates the request, creates a durable job and a dispatch record, then returns an accepted response with a job identifier. A worker later builds the report and records its result.

The following sequence shows a successful exchange. The API lifeline represents the submission and status service, including its durable state. The broker and dispatch relay are omitted; the dispatch arrow can arrive more than once.

HTTP 202 Accepted does not mean processing has completed. Define pending, running, succeeded and failed states, plus cancellation behavior where supported. Protect the status endpoint with the same ownership rules as submission.

If the submission response is lost, the client must recover the original job through a stable request key or lookup mechanism. Creating another job on every timeout can duplicate expensive work. Scope an idempotency key to the caller and operation, and reject reuse with a conflicting payload.

A transactional outbox can keep the job and intent to publish in the same local transaction. The relay still needs retry handling, and consumers still need to tolerate duplicate deliveries.

Put acknowledgments at the correct boundary

RabbitMQ distinguishes publisher confirms from consumer acknowledgments. A publisher confirm is evidence about broker acceptance, not evidence that a worker finished. A consumer acknowledgment depends on where the application places it.

For the report worker, persist the completed result before acknowledging the delivery. If the worker crashes after persistence but before acknowledgment, a redelivery should find the existing result and avoid publishing another artifact. Couple the deduplication record with the local state change wherever the storage model permits it.

For an external side effect, such as sending a report to a partner, local deduplication alone cannot make the remote action atomic. Use the remote system's supported idempotency or reconciliation contract. A timeout after submission can mean the partner accepted it.

Classify failures. Retry a transient dependency problem within a bounded policy. Quarantine an invalid message with its reason and minimal safe diagnostic context. Give quarantined work an owner, retention rule and reviewed redrive procedure. Blindly returning a poison message to the same queue can consume all worker capacity.

Give one owner the end-to-end retry budget

Use a caller deadline that includes connection setup, queueing, attempts, backoff and response work. gRPC deadline guidance describes propagation and cancellation responsibilities; verify the behavior of the language runtime you operate.

Here is an illustrative 800 ms budget for a read operation, not a recommended timeout:

| Stage | Maximum allocated time | | --- | --- | | Local validation and preparation | 100 ms | | First downstream attempt | 250 ms | | Backoff before one retry | 80 ms | | Second downstream attempt | 250 ms | | Response work | 50 ms | | Unallocated margin | 70 ms |

The allocations total 800 ms. Before a retry, check the remaining deadline and reserve response time. If the first stage runs longer than planned, the second attempt must shrink or be skipped. The per-attempt limit must include connection and transport behavior, not only server execution.

Retry only when the failure and operation permit it. A validation error is not made useful by repetition; a timed-out write is not proven uncommitted. Use cancellation to stop avoidable work, but do not treat cancellation as rollback.

Inventory retries in the application, SDK, proxy and mesh. Three layers each making up to three attempts can produce up to 27 lowest-level attempts for one call. Select one policy owner and test the effective behavior, including transparent retries described in the gRPC retry documentation. Record attempts separately from logical operations.

Add a mesh only for a stated operating need

A service mesh can centralize selected traffic and identity controls. It also adds configuration, resource use, upgrade responsibilities and another place where failures can be introduced.

Istio's traffic-management documentation explains timeout and retry configuration and warns about interaction with application policies. Inspect effective settings rather than assuming that installing a mesh supplies the desired deadline.

A mesh cannot determine whether repeating a charge is safe. It cannot define the job's terminal business state or replace consumer deduplication. For a small service estate, application libraries and platform controls may satisfy the need with fewer moving parts. Evaluate the operational workload alongside feature coverage.

Test the contract under interrupted communication

| Test | Evidence required | | --- | --- | | Response lost after a write | Caller recovers the same operation without repeating its effect | | Worker fails before acknowledgment | Redelivery completes or finds the existing result | | Invalid message arrives repeatedly | Work is isolated and other messages progress | | Dependency slows or becomes unavailable | Deadline, retry budget and resource limits hold | | Queue grows during a burst | Age and completion delay remain visible; admission can be limited | | Old and new payload versions overlap | Supported producers and consumers still interoperate |

Use the event-driven architecture guide for event contracts and the resilience guide for overload and recovery controls.

Bring one completed communication contract and its failure test results to Ampity's backend systems and API service. Include what success means, who owns retries and how an uncertain outcome is resolved.