When Payment Webhooks Arrive Out of Order

Handle out-of-order payment webhooks with scoped event identities, guarded business transitions and a separate fulfilment record. Reconcile gaps without creating...

Keep payment evidence separate from the order decision

An older payment notification arriving after a newer one must not overwrite the order's current decision. Store the provider observation, correlate it to the right account and payment attempt, then apply a transition that the business permits. Keep fulfilment and later financial adjustments in their own records so a delayed event cannot silently ship another order or erase a refund.

Consider a hypothetical marketplace selling a reserved coaching session. The buyer completes payment, but the success notification reaches the application before an earlier processing notification. A handler that copies each incoming status into the order changes paid back to pending. A support operator sees the stale state and offers another payment link. The transport delay has now changed the customer's experience and may lead to another charge.

This is an illustrative integration design, not an Ampity customer incident or a complete accounting specification. It assumes the marketplace has an agreed reservation and fulfilment policy. The engineering team must adapt the transitions to its payment methods, provider contracts and cancellation rules. A transport-level delivery receipt cannot define those policies.

The proposed boundary is narrow: one accepted purchase should produce at most one intended fulfilment action, while every unresolved payment observation remains inspectable. A later refund or dispute must remain visible even if the initial payment completed. Decide which service owns that boundary before selecting a queue or writing the webhook handler.

Record identities before interpreting a status

Retain the provider account context, environment, event identifier, event type, object identifier, API representation and receipt time. Associate them with the marketplace's payment attempt and order through an existing authorized mapping. Test and live observations must not share a business ledger. A connected account's object cannot be attached to an order merely because its identifier looks familiar.

For each valid receipt, distinguish receipt identity from processing identity and business-operation identity. Redelivery of one event is a transport duplicate. Two observations about the same payment can still carry different evidence. A worker retry is another execution attempt. None should create a new purchase identity or replace the original order-to-payment mapping.

Stripe's webhook documentation says delivery order is not guaranteed and distinct snapshot events can share a creation timestamp. It describes duplicate deliveries, event-ID tracking and cases with separate event objects. It also requires signature verification against the raw request body and documents account scopes. Treat these as provider-specific constraints, not a universal event-ordering protocol.

Use the supported verification mechanism before treating a payload as evidence. Restrict accepted event types and validate the expected account, environment and object relationship. An authenticated provider event can still be irrelevant to this order. Unknown mappings belong in a restricted investigation queue; they should not create a customer or attach payment authority based on free-text metadata.

Store only the fields required for correlation, recovery and the approved audit purpose. Protect any retained payload with access and retention controls. Avoid putting full payment objects, customer addresses or secrets into ordinary application logs. Operators need a traceable observation, not unrestricted access to every field the provider sent.

Accept the receipt without claiming processing is complete

For this design, the endpoint durably records a validated receipt and recoverable work before acknowledging acceptance. Keep the transaction short. A database inbox with a unique scoped event key is one option; a durable queue with an equivalent recovery contract is another. State what happens if storage fails and which supported provider retry behavior the endpoint relies on.

An acknowledgement means the receiver accepted responsibility for the receipt. It does not mean the payment-to-order transition, session reservation or customer notification succeeded. Track those steps separately. A dashboard that calls every acknowledged webhook processed conceals the gap between ingress and business completion.

The receipt and worker dispatch must not leave an unobserved handoff. If the application stores the receipt but crashes before enqueueing work, a dispatcher or reconciliation scan should discover the pending receipt. If dispatch repeats, the worker must handle the same processing identity safely. Define this recovery path against the chosen storage and queue guarantees rather than assuming two successful calls form one transaction.

Prevent two workers from independently applying the same order transition. Use the application's supported transaction or conditional-update mechanism to claim work and compare the current order revision. Keep the attempt result and transition evidence. A unique receipt key alone does not protect an external fulfilment call after the worker crashes midway through its execution.

Apply a business transition, not a global status ranking

Define which evidence permits which change for each payment attempt and order state. For the coaching example, fulfilment requires a verified completed payment flow, a current reservation and no existing fulfilment operation for that purchase. If the reservation has expired, the payment owner needs the approved exception path. Payment success cannot recreate an unavailable session silently.

Stripe's payment-status guidance distinguishes PaymentIntent states requiring processing, capture or customer action. It also explains that subsequent refunds and disputes appear on the Charge even when the PaymentIntent remains succeeded. Inspect the relevant objects for the financial question being answered; one field cannot represent every later outcome.

Keep refund, dispute, seller-transfer and fulfilment records separate where they have distinct lifecycles. A refund may require an operational cancellation, but it does not prove that a coach received the cancellation or that a seller transfer reversed. Those effects need their own approved rules, receipts and reconciliation. Do not rank all these states on one numeric success scale.

When prerequisites are missing, retain the observation and fetch the supported authoritative object under the correct account context. Record the observation and retrieval separately. A later read shows what was observed at retrieval time; it does not prove the exact state when a delayed notification was generated. If the read fails or the objects conflict, hold the consequential transition and assign reconciliation.

Compare the local revision when applying the decision. A successful provider lookup followed by an unconditional update can still overwrite a concurrent cancellation. Recheck the reservation, existing effect and permitted transition within the local commit boundary. If an external action is required, create a durable intent for that action and settle it separately from the database change.

Work through five arrival and recovery cases

The following worksheet uses synthetic orders and inert fulfilment adapters. It specifies proposed application behavior, not new provider statuses. Retain each observation even when it causes no current order change. An operator should be able to explain why the handler held, applied or reconciled a transition.

| Arrival or failure | Evidence to inspect | Proposed business response | | --- | --- | --- | | Processing arrives after success | Same account, payment attempt and current order revision | Retain the observation; do not regress the accepted order decision | | The same event is delivered twice | Scoped event identity and prior processing result | Resume unfinished work or return its recorded outcome; do not fulfil twice | | Success has no local reservation | Authorized order mapping and current reservation evidence | Hold fulfilment and assign the approved payment exception path | | A refund follows fulfilment | Refund evidence and separate fulfilment and cancellation records | Record the financial adjustment; settle required operational effects separately | | The fulfilment call times out | Stable operation identity and authoritative external receipt | Mark the outcome unknown; reconcile before issuing another effect |

A timestamp sort would not settle the missing reservation or timed-out fulfilment. Event deduplication would not settle a legitimate refund. Review each row against the actual operation boundary and include open exceptions in the report. Counting only completed workers can make a stalled inbox appear healthy while paid buyers have no accepted booking.

Reconcile missing evidence and external effects

Assign pending receipts and unknown outcomes to an owner with a review deadline. Inspect the provider's supported current-state or event-recovery mechanism alongside the local order, effect intent and any external receipt. Record which source established each result. A support note saying retried is not evidence that a reservation or cancellation completed.

Keep the external operation identity stable across recovery attempts where the external contract supports it. If fulfilment timed out after submission, inspect the existing effect before sending another request. Changing the key can make a retry look like a new booking. If the external service provides neither duplicate protection nor authoritative lookup, retain the uncertainty and use the agreed manual exception process.

Design reconciliation to cover more than the received inbox. A missing event has no failed local worker. Compare the relevant provider population to mapped local attempts within a recorded observation window, with pagination, scope and data-lag limits stated. Distinguish an unmatched payment, a deliberately excluded object and a lookup that could not finish. A partial scan cannot prove that no gaps remain.

An AI assistant may summarize redacted exceptions for an authorized operator and cite the relevant records. It must not infer a successful payment from a customer's message, choose a refund, change an account mapping or close an unknown external effect. The payment owner makes those decisions under the approved policy; model confidence does not add financial evidence.

Rehearse reordering before changing the handler

Build a synthetic fixture with one order, a reserved occurrence, scoped payment observations and an inert fulfilment adapter. Specify the expected order decision and effect count independently of the handler. Deliver success before processing, duplicate the success receipt and repeat the worker after it records the intent. Confirm that the current decision stays accepted and the fixture creates only the one intended fulfilment action.

Then break the prerequisites. Remove the reservation, change the account context, fail the authoritative read and cancel the order between lookup and local commit. Inject a timeout after the inert adapter records the effect but before it returns. Recovery must discover that receipt or retain an assigned unknown outcome, rather than assuming failure and issuing another booking.

Run a refund-after-fulfilment fixture and verify both financial and operational records. Assert the application exposes a pending cancellation when its adapter has not completed. Restore the old handler against records written by the candidate and inspect compatibility before calling the release reversible. Keep the source revision, fixture input, expected results and observed receipts in the acceptance record.

For the wider failure model, read distributed systems in production. For action recovery, use the idempotent write-tool discussion. Ampity's backend systems review and DevOps and SRE review can support the relevant integration work. Reading this article does not require submitting contact details.

Start the next review with the five-case worksheet and the synthetic timeout receipt for this payment handler. Assign the payment owner to resolve the missing-reservation policy before accepting fulfilment logic that depends on it.