Moving Retail ETL from Overnight Jobs to Incremental Data Products

An anonymous retail-analytics migration using Kinesis, Lambda, managed Flink and S3, with state, reconciliation and downstream publication treated separately.

Freshness mattered at the consumer, not just inside the job

A retail analytics company depended on overnight Spark jobs running on EC2. Delayed or failed processing affected when customers could use partner data. Adding a faster processor alone would not change daily aggregations or the meaning of the results exposed to existing queries.

Ampity redesigned the pipeline around Kinesis Data Streams, Lambda transformations, S3 storage and Athena query access. Stateful windowed computation also formed part of the design through managed Flink. The migration needed to preserve data quality, lineage and downstream compatibility while the operating team learned a different processing model.

Select the processing responsibility before the service label

The choices included extending the Spark approach, a serverless streaming path and a broader managed Kafka/Flink arrangement. The selected design combined record transformations with separate stateful aggregation responsibilities. It was not a requirement that every operation become stateless.

Managed Flink appears in the stateful design even though the broader Kafka/Flink option was not selected. The precise split and connection between the Lambda and Flink paths remains unspecified. A comparable decision needs the actual transformations, joins and retained state before choosing that division.

A scheduled incremental job may still be sufficient when it meets the consumer's deadline. Conversely, a continuously running processor cannot make an upstream partner publish data sooner. Include capture, waiting, transformation, sink publication and query refresh in the freshness boundary.

Keep state recovery separate from published correctness

The storage design used time-partitioned Parquet data in S3, with Athena providing query access. Daily aggregations had to become incremental calculations rather than being assumed equivalent after a change of engine.

Define the event identity, time basis, late-arrival rule and the meaning of an updated aggregate. If a repeated input appends the same contribution twice, a successful processor restart can still leave the business total wrong. The connector settings, deduplication horizon, checkpoint configuration and publication protocol are not specified by this case.

For a comparable implementation, retain an approved input range and transformation revision, reconcile candidate results and identify the version visible to each consumer. A checkpoint or an object arriving in storage is not, on its own, permission to present a dataset as complete.

Compare the same business data during migration

The migration approach kept batch and streaming paths in parallel, reconciled their outputs and moved simpler sources before feeds with custom transformations. Existing queries, data-handling requirements and customer commitments remained constraints.

Parallel outputs need a common comparison boundary. A provisional stream and a closed batch can legitimately differ because one includes a later correction. Compare defined input ranges and finality rules before treating every difference as an engine defect or silently accepting it.

Review one partner feed before moving another

Start with authorized read-only inspection of permitted feed definitions, output schemas and existing reconciliation records. This worksheet is proposed guidance for a comparable migration, not a completed exercise from this engagement.

Any later rehearsal needs an authorized isolated environment, synthetic or approved sanitized data, restricted identities and separate non-delivering outputs. Prevent changes to production datasets, customer dashboards and downstream actions. Bound replay volume, runtime and spend, and stop on unexplained differences or a prohibited destination.

Restoring the previous query view changes future reads. It does not retract reports already exported or reverse decisions made from an incorrect aggregate. Keep those corrections owned separately.

Scope without a freshness or savings benchmark

This case covers the pipeline design, incremental-model change and parallel migration approach. It does not publish measured freshness, cost reductions, engineering-productivity gains, a no-loss result or sales attributed to the work.

Start with the feed whose correction rule is understood

Choose one source and name the consumer, its decision deadline, when results become final and how a correction reaches it. Use the batch-versus-stream processing playbook for the broader mode-selection and recovery process, rather than treating this service combination as the default for every feed.

Technical reference

AWS's Lambda and Kinesis guidance identifies duplicate-processing risk in event source mappings. Its managed Flink fault-tolerance guidance describes checkpoint and snapshot configuration. These mechanisms have different scopes; neither service name establishes the correctness of this migration's external dataset or consumer decisions. Verify the actual connector and recovery settings before reuse.