Recover Business Tasks from a Dead-Letter Queue

Classify failed messages by business outcome, preserve operation identity and authorize bounded replay with evidence that customer work has been resolved.

Find the business task behind the failed message

A dead-letter queue isolates messages that could not be processed under the configured delivery policy. Recovery still requires a decision about the business task: whether it already completed, remains authorized, needs repair or must end without another attempt. Assign an owner and preserve evidence of that disposition. Moving a message back to a source queue establishes another delivery opportunity, not customer completion.

Consider a hypothetical service that submits approved supplier updates to an external system. Some messages failed before submission. Others timed out after the external system may have accepted them. A third group refers to approvals that expired while the messages waited. Replaying all three groups after fixing a network issue can duplicate accepted changes and execute obsolete instructions.

This article proposes a recovery review for event-driven application owners. The supplier example is illustrative, not a customer case study or a report of Ampity production operations. Queue products have different delivery and retention contracts. Verify the actual broker, queue type, client and consumer configuration before using this procedure. An AI-generated classification can assist review, but it does not establish permission to replay a consequential task.

Establish why the message was isolated

AWS's SQS dead-letter queue guidance describes isolating unsuccessful messages for diagnosis and a receive-count policy for moving them from a source queue. Our recommendation is to record the reason a particular message reached that boundary, including the consumer revision and last observed processing stage. A receive count alone does not explain the business result.

Inspect whether the consumer failed to parse the payload, lacked permission, lost a dependency or completed a write before acknowledgement failed. Check processing duration and the broker's delivery settings. A message that repeatedly becomes available while a slow consumer is still working needs a different correction from an invalid payload that cannot be processed by that consumer version.

Preserve a controlled diagnostic record before changing configuration. Include the business-operation identifier, payload revision, original event time, source, error classification and relevant execution receipts. Restrict raw payload access to authorized reviewers. Do not paste supplier records, credentials or personal data into a broadly shared incident channel merely to make diagnosis easier.

Group messages by observed cause and outcome evidence. A shared exception string can conceal different states, such as a request never sent and a request accepted without a returned receipt. If the original logs cannot distinguish those states, mark the outcome unknown and assign reconciliation. Do not convert missing evidence into an assumption that nothing happened.

Classify tasks before granting replay permission

Separate the authority to operate the broker from the authority to execute the enclosed business action. An administrator may be able to move messages without being entitled to change a supplier record. Bind the recovery decision to a named task, revision, destination and permitted action. For a large cohort, document a selection rule that the reviewer can inspect and test.

The worksheet below is a proposed classification, not a broker feature or an automatic approval policy. Each task needs a current disposition. If later evidence changes that disposition, record who changed it and why rather than silently placing the task in another replay batch.

| Business state | Proposed disposition | Evidence required | | --- | --- | --- | | No effect occurred; cause fixed; authority remains valid | Eligible for a bounded replay | Failure stage, tested correction and current approval | | Original task already completed | Close the recovery item without another business action | Authoritative completion receipt tied to the original operation | | External effect remains unknown | Hold for reconciliation | Destination lookup or an owned exception when lookup is unavailable | | Task expired, was cancelled or lost authority | Record final non-execution | Current task state and the owner's decision on customer communication | | Payload needs a material correction | Prepare a separately reviewed revision | Original evidence, changed fields and renewed execution authority |

Replay eligibility is temporary. Recheck it at the execution boundary because a user can cancel the request or another worker can complete it after review. A spreadsheet exported yesterday does not establish today's permission. Keep the original operation identity for a repeat of the same task. If a correction creates a different business instruction, define its relationship to the original and review whether the original could still execute.

Preserve task identity when transport identity changes

AWS's SQS redrive documentation states that redriven messages receive new message identifiers and enqueue times. It also describes interleaving replayed messages with new arrivals, and notes that native redrive cannot filter or modify messages. Our recommendation is to reconcile by a stable business-operation identity carried by the application, rather than treating a transport identifier as the customer task.

A new broker identifier must not automatically mean a new supplier update. Determine how the destination checks whether the original action already happened, and how long that check remains effective. If duplicate prevention expires before the proposed recovery window, the owner needs another reconciliation method or a narrower recovery boundary. Record that limitation before admitting old tasks.

Inspect ordering at the domain level. An older supplier address change replayed after a newer approved change can overwrite the current address. A FIFO transport configuration cannot decide which revision is authoritative for the business record. Check the required predecessor, current revision and allowed transition in the consumer, and preserve an explicit rejected or superseded outcome when the old action no longer applies.

Choose a recovery mechanism that can enforce the reviewed cohort. If a native move operation releases everything in a mixed queue, it cannot implement a selective business decision by itself. Use an approved selective processing path or an execution-time eligibility gate that safely handles every released task. Test the mechanism's behavior before assuming the console's redrive action matches the reviewer's selection.

Measure recovery as resolved tasks, not queue movement

Maintain a recovery ledger with one row per unique business operation in scope. Broker messages and execution attempts can appear as linked evidence beneath that row. Record disposition, owner, decision time, original revision, replay authority, destination receipt and outstanding uncertainty. Define which outcomes count as resolved and keep completed work separate from authorized non-execution.

For an illustrative ledger, suppose 145 unique tasks have been classified into 30 eligible for replay, 40 already completed, 20 expired, five needing correction and 50 awaiting outcome evidence. Those disjoint groups total 145. The queue may contain more than 145 messages if several refer to the same task. Do not use message count as the denominator for a customer completion percentage.

After a bounded replay of the 30 eligible tasks, suppose 24 have confirmed completion, four are held by a fresh eligibility check and two have an uncertain external result. Record the new evidence for each of those 30 tasks. Moving all 30 messages out of the dead-letter queue does not justify marking all 30 tasks complete. The original 50 unknown tasks also remain open until their evidence is reconciled.

Compare remaining ledger items with broker state after each tranche. A disappearing message can reflect processing, movement, expiry or deletion. Require a disposition before deleting a recovery item, and preserve necessary evidence under the organization's access and retention policy. Do not retain sensitive payloads indefinitely merely because a technical incident remains open. Assign retention decisions to the appropriate data owner.

Bound replay and prepare for another failure

Test the correction with synthetic fixtures in an isolated environment before production recovery. Include a duplicate, an already-completed operation, an expired approval, a payload requiring correction and a write whose response is deliberately lost. Verify that the consumer produces the expected business disposition and preserves uncertainty when it cannot establish an outcome.

Admit replay at a rate supported by the receiving system while new work continues. Check useful completion rate, downstream pressure, oldest eligible task age and repeated failure reasons. Assign an operator who can stop further admission and an independent observer for unexpected effects. Define the stop conditions before starting; a growing rejected cohort or unknown-outcome count can justify stopping even while the queue is shrinking.

AWS documents that cancelling an in-progress SQS move does not remove messages already moved to the destination. Our operational recommendation is to record both the move task and the destination's execution state when stopping. Pausing transport movement may still leave admitted work executing. Do not claim rollback unless the business system supports an authorized corrective action and its result is verified.

Exercise a second dependency failure during the test replay. Can the team identify which tasks started, completed, remained queued or became uncertain? Does the recovery gate survive a consumer restart? If restarting the worker loses the reviewed disposition or creates a fresh operation identity, repair that gap before widening the batch. Automation should preserve the ledger, not replace it with a success-looking move status.

Assign an owner and a deadline to every unresolved task

A queue alarm should lead to a named recovery owner, not an instruction to everyone to retry. Assign application diagnosis to the consumer maintainer, business eligibility to the relevant process owner and replay operation to an authorized operator. Name the person responsible for communicating a delayed or final non-execution outcome to affected users. These may be different people.

Use an evidence deadline for unknown external effects. A destination lookup may be unavailable or may return incomplete history. Preserve that limitation and escalate through the approved business process. Repeatedly resubmitting the task to obtain a receipt can create an additional effect while leaving the original uncertainty unresolved. A timeout in the investigation path is not new execution authority.

Keep broker retention and business expiry distinct. AWS describes different SQS retention timestamp behavior for standard and FIFO dead-letter queues. Review the actual queue policy before planning investigation time, and carry the original business deadline independently. A transport retention reset does not extend an expired supplier approval or customer instruction.

If AI assists classification, constrain it to proposing a reason and pointing to the supporting evidence. Do not let a fluent explanation substitute for the destination receipt, current authorization or named-owner decision. Use reviewed rules for deterministic eligibility checks where possible, and retain exceptions for human review. The recovery record should reveal which evidence supported each action.

Bring one reconciled cohort to the recovery review

Start the next review with a bounded cohort of unique business operations. Attach the original failure evidence, current eligibility, selected recovery mechanism, duplicate-prevention boundary, measured replay limit and stop procedure. Include the unresolved tasks and their owners alongside completions. A broker dashboard screenshot is useful supporting evidence, but it cannot supply those business decisions.

Use the retry-recovery article to check aggregate admission pressure and the timed-out action article for uncertain effects. Bring the cohort ledger to a reliability review or DevOps and SRE review before approving a wider recovery run. These resources can be read without providing an email address or asking Ampity to contact you.