Data Pipeline Architecture: Replay, Late Data and Safe Publication

Design recoverable data pipelines with stable event keys, bounded replay, explicit late-data policy and atomic publication. Includes a worked correction and failure...

Start with the published result

A pipeline accepts data from systems it does not control, transforms it and publishes results that people or applications use. Its architecture needs to explain what happens when a run only partly succeeds.

Write the consumer contract before selecting an orchestrator. Specify the meaning of a row, the decision it supports, tolerated delay, required completeness and recovery owner. A morning operations report and an automated inventory action can need different release policies even when they use the same source.

Keep four responsibilities visible: capture reproducible input, transform it under a known version, validate a candidate and publish an identifiable result. One tool may perform several jobs, but its success status must not conceal which responsibility failed.

This guide focuses on recoverability and observable data delivery. Whether a dataset is suitable and authorized for an AI use case is a separate question, covered by a data-readiness assessment.

Define source and destination contracts

“Orders table” does not explain whether one row represents an order, item, event or current state. That distinction determines the key and replay behavior.

Record these agreements:

| Contract element | Decision to record | |---|---| | Identity and grain | Stable source and business keys; whether rows are immutable events or mutable entities. | | Time | Event time, arrival time, source time zone and the cutoff used for a published window. | | Changes | How updates, cancellations and deletions are represented and ordered. | | Publication | Output revision, completeness status and the interface consumers read. | | Recovery | Retained input, replay boundary, transformation version and responsible owner. | | Access | Authorized producers and consumers, sensitive fields and approved retention. |

A schema check can detect a missing column. It cannot establish whether a timestamp changed from carrier acceptance to label creation. Keep domain-approved examples beside structural tests.

For mutable sources, a snapshot must align with the change-capture position used afterward. Otherwise a backfill can miss or repeat changes at the boundary. If historical source state cannot be reconstructed, retain the permitted source evidence needed for the agreed replay period and disclose the limit.

Make retries deterministic within a named boundary

A retry should use the same input manifest, transformation version and output identity. Reading “the latest files” on each attempt changes the question halfway through recovery.

Airflow's task guidance recommends repeatable task outcomes, bounded partitions and care with duplicate-producing writes. An orchestrator can retry work; it cannot make an arbitrary destination idempotent.

Choose a sink mechanism deliberately:

For immutable events, enforce a stable event key and detect conflicting payloads under the same key.
For mutable entities, apply a defined source version or ordering rule so an old update cannot overwrite a newer one.
For a bounded snapshot, build and validate a candidate revision, then switch the consumer reference through a supported atomic operation.
For external effects, use an accepted idempotency key or reconciliation process. A database transaction cannot roll back a notification already sent elsewhere.

Checkpoints should describe durable progress. Advancing an offset before the output commits can lose data. Committing output before a separately stored offset advances can cause replay. The latter is recoverable only if replay does not duplicate the effect.

Worked replay: a shipment-count correction

Consider a hypothetical daily report of net dispatched units. The source uses immutable events; a correction is a new adjustment event, not an edit to an earlier event. The business contract permits a negative adjustment but rejects an unexplained duplicate identity with a different payload.

The input fixture is deliberately small:

| Arrival | Stable event ID | Unit delta | |---|---|---:| | Initial delivery | dispatch-A | 3 | | Initial delivery | dispatch-B | 2 | | Retry of the first delivery | dispatch-A | 3 | | Late correction for the same day | adjustment-C | -1 |

The first manifest contains three deliveries but only two unique events. Its total is 3 + 2 = 5 units. Counting deliveries would incorrectly produce eight.

For this example, accepted event rows, output revisions, a publication pointer and the publication ledger live in one transactional database. That is a design assumption, not a guarantee for every warehouse or object store.

A bounded publish procedure is:

Pin the source manifest, reporting day and transformation version in the run record.
Deduplicate by source plus event ID. Verify that repeated IDs carry the same payload; quarantine conflicts.
Build a candidate revision from the accepted events identified by the pinned manifest for that reporting day. Keep it separate from the active revision.
Verify keys, allowed adjustments and the five-unit control total.
In one transaction, record the published revision and advance the active reference and publication ledger. Serialize or condition this update so an older concurrent run cannot replace a newer correction.
On retry, inspect the ledger. If the same manifest and transformation already published, verify that result instead of creating another business effect.

PostgreSQL's transaction tutorial explains the all-or-nothing boundary used by this example. If the output and pointer live in different systems, document their actual commit protocol and recovery states instead of assuming the same guarantee.

An upstream checkpoint outside that transaction may lag behind it. Replaying the input is safe here because the event identity and publication record resolve duplicate work. It is not a claim of universal end-to-end exactly-once processing.

Give late data an explicit correction path

Suppose adjustment-C arrives after the daily revision has been published. Its event date belongs to that day, while its arrival time is later. Publishing time alone cannot tell a consumer how complete the source was.

For this illustrative report, the owner allows automatic revisions for two days after the reporting day. Later events enter an exception workflow. Two days is a consumer agreement to test against source behavior, not a default for other pipelines.

The late event produces a new manifest and revision with 3 + 2 - 1 = 4 units. The original five-unit revision remains identifiable under the retention policy. Consumers receive the affected date, previous revision, replacement revision and reason, so they can refresh or preserve a reproducible historical report.

Do not append a replacement total to the same unversioned output. A consumer summing both rows would count both the original and corrected result.

In streaming systems, Beam's watermark and late-data guidance separates event time from processing time and describes lateness handling. A watermark estimates event-time progress. It does not prove that the source will never send another correction. Configure allowed lateness and retained state with a separate path for data that can no longer be handled automatically.

Test the interrupted states

Use the fixture above to exercise the sink, not only the transformation function.

| Injected failure | Required observation | |---|---| | Crash while building a candidate | The active revision remains unchanged; incomplete candidate data is not exposed. | | Crash before the publication transaction commits | No partial pointer or publication-ledger update becomes visible. | | Connection drops after commit but before acknowledgement | Retry finds the committed run and does not duplicate the published effect. | | Conflicting payload arrives under dispatch-A | The conflict is visible to an owner; it is not silently accepted as a duplicate. | | Older run finishes after the correction | It cannot replace the newer active revision. | | Input needed for replay has expired | Recovery reports the missing evidence instead of presenting an incomplete rebuild as success. |

The arithmetic can be checked without infrastructure. Atomicity, concurrency and failure recovery must also be tested against the chosen storage and orchestration configuration before production use.

The main limitation is that a successful transformation test does not prove publication atomicity, consumer compatibility or replay completeness. If the retained input cannot reconstruct a required revision, stop presenting rebuild as a recovery option and document the remaining data-loss boundary.

Quarantine is appropriate only when consumers can tolerate the excluded data. Record rejected counts, affected outputs and who resolves the exception. If excluding a record breaks a required reconciliation, hold the entire candidate.

Observe the consumer contract

Track job execution separately from data freshness, completeness and delivery. A completed run can publish an empty or stale result.

For each revision, retain a lineage record containing input identifiers, source progress, code and schema versions, checks, rejected records and downstream dependencies. Avoid copying sensitive payloads into broadly accessible logs.

Define service objectives with an explicit denominator, such as the proportion of scheduled report windows published by their agreed deadline with required checks satisfied. Mark windows with missing source data separately; do not hide them by reporting the newest event timestamp.

Alert on the affected product and the action required. The source owner resolves missing data, the pipeline owner resolves transformation and publication failures, and the consumer owner decides whether a stale revision remains usable.

Choose the next recovery exercise

Take one published dataset and rehearse the failed-acknowledgement case, a duplicate event and a late correction. Confirm which revision each consumer reads and how the operator identifies incomplete work.

Bring the input contract, run ledger and test results to a system architecture review. They establish the recovery boundary before adding another scheduler, queue or processing engine.