Replay Marketplace Webhooks Without Repeating the Purchase

Rehearse duplicate, delayed and concurrent webhook deliveries against order invariants, durable receipts and unknown external effects in an isolated drill.

trigger="An integration change or delivery incident makes webhook replay necessary, but orders already contain accepted decisions and external effects." owner="The marketplace service owner accountable for the order invariants and drill acceptance." participants={['Payment integration engineer', 'Order-domain owner', 'Security reviewer', 'Independent test observer', 'Reconciliation operator']} prerequisites={['Approved order and payment transition rules', 'A pinned candidate and event representation', 'A synthetic scoped fixture set', 'An isolated durable inbox and order ledger', 'Inert payment, fulfilment and notification adapters']} outputs={['A scoped replay manifest', 'An independently specified invariant matrix', 'A concurrent and crash-boundary result log', 'An external-effect reconciliation ledger', 'A release decision with owned holds']} doneWhen={['Every delivery has a recorded disposition', 'Permitted order transitions occur without duplicate fulfilment', 'Account and environment boundaries remain intact', 'Unknown external effects stay held until reconciled', 'Restart and rollback preserve accepted evidence']} />

Rehearse the business boundary, not just event delivery

Replaying a webhook is another delivery attempt, not a new purchase. Keep the provider observation, the worker attempt, the marketplace decision and the external effect as distinct records. This drill tests whether those boundaries survive duplicate deliveries, reordered observations, competing workers and interruption. A receiver returning success is not evidence that fulfilment completed, and a queue showing no backlog is not evidence that every order is correct.

Consider a hypothetical marketplace reserving coaching sessions. Order O17 maps to payment attempt P4 in one test account. A success observation arrives twice, a processing observation arrives later, and a fulfilment adapter completes an action before its response is lost. The required behavior is defined by the marketplace's approved reservation policy, not by arrival order. A second delivery must not book another session, overwrite a refund record or turn an uncertain adapter result into a new action.

Run this procedure only with synthetic identities and inert destinations. It is a proposed rehearsal, not an Ampity customer incident or an accounting specification. The payment and business owners must approve the actual transitions and permitted operations. This playbook does not authorize replaying live events, issuing refunds or changing provider settings. Its completion establishes evidence for the pinned test candidate and stated failure boundaries, not universal exactly-once processing.

1. Agree the invariant and the replay scope

Owner: service owner with order-domain owner. Output: approved drill boundary. Name one order workflow and the operation it must protect. For the example, one accepted purchase may produce at most one authorized session booking, while later financial adjustments remain separately visible. Define what qualifies as accepted and which record is authoritative. An “order paid” flag is insufficient if payment attempts, reservation expiry and cancellations have independent rules.

Choose the test account, environment, tenant, event types and observation window. List excluded workflows and external destinations. Keep test and live identities separate even if their payloads look similar. Record the maximum deliveries, concurrency, execution duration and storage budget so a replay experiment cannot become an uncontrolled load test. The independent observer must know how to stop admission and pause effect dispatch.

Specify the stop conditions before running anything. Stop on an unexpected external connection, a cross-account mapping, an unapproved state transition, a missing durable receipt or an unexplained effect count. Preserve the evidence when the drill stops. Do not reset the fixture database merely to obtain a later clean run; the failure is part of the acceptance record until its cause and correction have been demonstrated.

2. Pin the receiver, worker and event representation

Owner: integration engineer with security reviewer. Output: candidate identity register. Record the receiver and worker releases, schema, transition policy, configured event representation and adapter revisions. Retain the fixture bytes and their integrity references. A replay evaluated by one parser and dispatched by another worker is not a single identified candidate. Include configuration and mappings that change behavior without changing the source commit.

Stripe's webhook guidance documents duplicate delivery, lack of guaranteed delivery order, raw-body signature verification and differences between event representations. Its manual resend behavior does not replace automatic retries. Treat these as specific provider constraints, not a promise for every integration. Our drill recommendation is to pin the applicable representation and verify transport separately from business acceptance, with unsupported shapes receiving an explicit disposition.

Keep provider-generated deliveries distinct from locally constructed fixtures. A fixture signed with a test secret proves only that the local verifier accepts that fixture under its test configuration. It does not establish provider provenance. Never disable production verification to make historical bytes replayable. If a separate replay ingress is needed, define its authenticated operator, bounded manifest and audit path, and ensure it cannot be mistaken for the public webhook receiver.

3. Specify expected results before observing the candidate

Owner: independent observer with order-domain owner. Output: fixture and invariant matrix. Give each case a starting order revision, permitted observations, arrival schedule, expected decision and expected effect count. Agree results from the business rules before running the handler. Include duplicate success, processing after success, success after cancellation, refund after payment and an unrelated account whose object identifier resembles an existing mapping.

Separate ordinary duplicates from different observations about the same object. Two event identifiers can describe the same relevant business fact, but two refund objects can also represent distinct permitted adjustments. An object identifier plus event type is not a universal key for every business action. Define the operation identity and allowable state transition at the domain boundary rather than dropping every later event of the same type.

Add negative controls. Deliberately use a candidate that repeats fulfilment or copies the last arriving status directly into the order. The observer must show that the expected-result checks fail for that candidate. An AI assistant may suggest additional event permutations, but it must not decide refund policy, infer missing payment evidence or approve its own expectations. Retain the independently approved matrix with the test results.

4. Verify transport without admitting unauthorized business work

Owner: security reviewer with receiver engineer. Output: ingress result log. Exercise valid test delivery, altered bytes, wrong secret, missing verification metadata, unacceptable freshness and malformed payload. Record the receiver's response, durable receipt and effect count for each case. The untrusted envelope must not be allowed to select a trusted tenant or payment mapping merely by supplying an order identifier.

In this proposed receiver contract, validate the request before persisting an accepted event. Store the permitted evidence under the correct provider, environment and account scope, then acknowledge only after that durable acceptance is confirmed. Complex order processing occurs afterward. If persistence fails or its result is uncertain, do not claim the event was safely accepted. Observe redelivery and recovery against the retained receipt rather than assuming a response timeout means nothing was stored.

Do not confuse delivery freshness with business authority. A valid fresh resend can concern an old purchase that may no longer permit fulfilment. A previously accepted receipt can still require authenticated, scoped operator replay without pretending its old transport signature is a current public delivery. Keep both routes visible in the fixture matrix, and test that neither can bypass the order transition or current effect-authorization rules.

5. Test durable identity under concurrent duplicate admission

Owner: receiver engineer with database reviewer. Output: duplicate-admission evidence. Define the accepted event identity using provider, environment, account and event identifier at the documented scope. Record delivery attempts separately. Changing the destination endpoint or replay run must not create another business event identity. Preserve representation and payload integrity evidence without silently using a new decoder version to evade duplicate detection.

PostgreSQL's INSERT documentation describes unique-conflict handling and atomic upsert outcomes. In a PostgreSQL implementation, use the appropriate constraint as part of the receiver design, then inspect its actual transaction outcome. A conflict handler is not proof that a worker claim, order update or external call is atomic. Do not overwrite the retained accepted payload merely because the later delivery uses the same key.

Deliver the same event concurrently from two test clients. Require one accepted inbox identity and traceable attempt dispositions, with no second fulfilment intent. Repeat with the same identity but changed business payload; the proposed policy is to quarantine that inconsistency for investigation rather than mutate the original evidence. Test different accounts and environments too, so a valid independent event is not incorrectly suppressed by an overly broad global key.

6. Guard the order transition against stale and competing work

Owner: order-domain engineer with observer. Output: transition result record. Process each permitted observation against the authoritative order and payment mapping. Verify object identity, account, currency and amount where they are prerequisites of the approved transition. Record the source observation, evaluated order revision, policy revision, decision and reason. Do not derive the order state from a timestamp alone or make an event name sufficient authority to fulfil an expired reservation.

Run workers concurrently against the same starting revision. The transition must reject or re-evaluate a stale claim under the implementation's concurrency contract. Retrying a database conflict is allowed only after reading and evaluating the resulting authoritative state. A successful worker lease acquisition does not establish that another path cannot update the order. Include customer cancellation or another legitimate domain action in the race, not only two identical workers.

Permute arrival schedules and compare final invariants, not identical intermediate logs. Some orders of observation can require an explicit hold or authoritative lookup. Define that expectation before the run. Preserve later refunds and disputes separately from initial payment completion. Do not force every financial lifecycle into an increasing numeric status to make out-of-order tests appear simple; the business conditions may be incomparable rather than earlier or later.

7. Commit the decision and effect intent together

Owner: order-domain engineer with dispatch owner. Output: committed decision and intent receipts. Where the design uses a transactional outbox, commit the permitted order transition and its identified effect intent in the same local transaction. Give the business operation a stable identity that survives event redelivery and worker retry. Record when no intent should be produced, such as an already satisfied transition, denied mapping or held reservation.

AWS's transactional outbox guidance describes the database/message dual-write problem and the need to handle duplicate messages. The outbox coordinates the local state change with a durable notification intent. It does not make an external booking, payment API and local receipt one transaction. Keep those effects and their settlement evidence separate from local commit success.

Interrupt the candidate before transaction commit, after commit but before dispatch and after dispatch acknowledgement. On restart, verify the intended committed rows rather than counting worker executions. A rolled-back transition must have no dispatchable effect intent; a committed transition must retain its pending intent. Test publication failure and duplicate dispatch explicitly. A broker acknowledgement cannot close the business effect unless the separate destination contract and receipt justify that conclusion.

The local transaction covers only the order decision and effect intent. Receipt acceptance, external execution and settlement are separate boundaries. Blue arrows show an admitted path; the orange branch retains an unknown external result. Reconciliation supplies evidence to the receipt record, not a shortcut that repeats the external action.

8. Preserve unknown effects and retry identities

Owner: dispatch owner with reconciliation operator. Output: external-effect ledger. Make the inert adapter complete its simulated action but withhold its response. The dispatcher now has an unknown result, not a proven failure. Require the ledger to preserve the operation identity, attempted parameters, destination scope and uncertainty. Replaying the input event must not mint a new operation and invoke the action again.

Stripe's idempotent-request reference explains that repeated keys return a retained first result, including some failures, and that a key reused after pruning can produce a new request. Its contract is provider-specific and bounded. Our proposed dispatcher must preserve the original operation and parameters, check the applicable retry contract and refuse a new effect when the outcome cannot be safely established.

Test an unchanged retry while the adapter's accepted identity remains available, a changed-parameter retry and a retry after simulated identity expiry. Require explicit outcomes for each. Do not assume that a provider key establishes permanent application deduplication. If the destination cannot safely find or replay the original operation, keep it held for an approved investigation. An operator deciding that a second action is justified needs a separately recorded business decision, not a disguised retry.

9. Crash after receipt recording and after acknowledgement loss

Owner: integration engineer with independent observer. Output: interruption matrix. Kill the worker after it records a known destination receipt but before marking its attempt complete. Restart and redeliver the event. The retained receipt must prevent another business action even though the worker attempt appears interrupted. Separately, simulate a local receipt-write timeout after a successful external response; read back the operation record before deciding whether it is missing.

Keep missing evidence distinct from confirmed absence. A failed read, incomplete destination search or unavailable receipt store cannot justify another dispatch. Where a readback proves the recorded operation, validate identity and parameters before settling it. Where it proves a safe retry under the destination contract, retain that basis. Where neither is possible, keep the action unknown with an owner and next investigation step.

Test the observer's own evidence loss too. If the fixture adapter's action ledger is inaccessible, the drill cannot claim no duplication merely because the application counter stayed unchanged. Restore access or classify the result as inconclusive. Preserve both positive and negative observations with their coverage. A zero count from an unavailable or incorrectly scoped query is not an acceptance result.

10. Reconcile the full replay manifest

Owner: reconciliation operator with observer. Output: disposition and business-state reconciliation. List every intended fixture delivery and its ingress result. Connect accepted event identities to worker attempts, order decisions, effect intents and verified adapter receipts. Include ignored event types, duplicates, quarantined conflicts, held mappings and unknown effects. The manifest is the denominator; counting successful jobs alone hides the fixtures that never reached a worker.

Compare the final order and financial records with the independently agreed expectations. Check booking count, accepted order revision, retained cancellation or refund evidence and the absence of cross-scope writes. An expected no-op should still have an explainable disposition. A correctly held unknown should be reported as held, not as completed work or a disappeared error.

Repeat the same manifest without resetting accepted evidence. Require the same business result and no new external action for already settled operations. Then add one genuinely new, permitted adjustment with its own identity; it must not be suppressed by an overbroad deduplication rule. Keep the replay run identifier in the test log only. It is a useful investigation handle, not a reason to change the purchase or effect identity.

11. Stop and recover without deleting obligations

Owner: service owner with release engineer. Output: recovery and restart record. Pause replay admission and effect dispatch when a stop condition occurs. Preserve inbox records, committed decisions, pending intents and uncertain effects. Returning to the previous worker release is safe only if it can read that retained state and apply the accepted rules. Hold unsupported representations rather than letting an older parser silently discard their evidence.

Rollback does not undo an already accepted booking or financial adjustment. Reconcile each completed or unknown operation before deciding on a compensating action. A compensation is a new authorized business operation with its own receipt and limits. In the drill, demonstrate the bookkeeping through inert adapters; do not turn the exercise into a live refund or cancellation procedure.

Only reset disposable fixtures after the observer exports the result and the owner accepts the disposition of every hold. The reset must target the exact isolated environment, not a shared queue or production table. Retain the failed candidate and corrected result as separate runs. Restart with the same durable identities where that is the tested recovery condition, rather than making duplicates disappear by erasing their receipts.

12. Accept the candidate using retained evidence

Owner: service owner with independent observer. Output: scoped acceptance decision. Review the acceptance criteria below against actual receipt and invariant evidence. Name the candidate, fixture matrix, concurrency schedules, interruption points and adapters observed. Passing a serial happy path does not prove the competing-worker case. An unavailable effect ledger leaves that part unverified even when all application assertions pass.

Replay acceptance checklist

• Every intended delivery has an explicit transport and processing disposition at the correct account and environment scope.
• Concurrent duplicates preserve one accepted event identity; conflicting payloads do not overwrite retained evidence.
• Arrival permutations preserve the approved order invariants, including cancellations and later financial adjustments.
• A committed transition retains one identified intent; rollback creates no dispatchable intent.
• Settled effects remain settled across worker restart and manifest replay. Unknown results cannot produce a new operation automatically.
• Changed parameters, expired retry support and unavailable readbacks have tested holds with owners.
• Reconciliation accounts for no-ops, rejected mappings, pending intents and unresolved external effects.
• Recovery preserves accepted state and does not imply that stopping a worker reverses external actions.

Assign a next action to every failed or inconclusive case. Record what change invalidates the result, such as a parser upgrade, transition policy change or new destination contract. Use the payment webhook ordering article for domain boundaries and the marketplace transaction playbook for the broader decision model. Bring one completed replay manifest to a backend systems review or DevOps and SRE review. Downloads require no email address; requesting contact remains optional.