Serverless Data Pipeline for Real-Time ETL

Replaced batch ETL with serverless streaming pipeline, reducing data latency from 24 hours to 5 minutes while cutting infrastructure costs by 65%.

[Overview] System Overview

A retail analytics company processed data from 500+ retail partners to provide market intelligence. Their batch ETL processed 10TB daily, but the 24-hour processing cycle meant insights were always stale. Competitors offering real-time analytics were winning deals.

We redesigned the pipeline using serverless components: AWS Lambda for transformation, Kinesis for streaming, and S3 for storage with Athena for queries. The result was 5-minute data freshness at 65% lower cost than the previous batch infrastructure.

The transformation required not just technical changes but rethinking how data was modeled. Batch processing could accumulate state across hours of data; streaming required stateless transformations or careful state management in the streaming layer.

[Context] Situation before

The existing ETL ran on a cluster of EC2 instances running Apache Spark overnight. Jobs started at midnight and completed by 6 AM. Any failure during the window delayed data availability to customers.

##### Operational Burden

The ETL team spent 40% of time troubleshooting failed jobs. Spark cluster tuning was a constant effort. Memory issues caused OOM failures regularly.

##### Scaling Challenges

Adding new data sources required sizing the cluster for peak load. The cluster ran all night but only needed peak capacity for 2 hours. Resources were wasted.

##### Customer Complaints

Enterprise customers demanded real-time data. Sales was losing deals worth $2M annually to competitors with streaming capabilities.

[Diagnosis] The real problem

Batch processing was not just slow-it was architecturally wrong for the use case. Retail decisions needed fresh data. Stock-out detection delayed 24 hours was useless; the stock-out had already affected sales.

The data model assumed batch. Aggregations were computed across entire day datasets. Converting to streaming required decomposing aggregations into incremental updates-a significant redesign effort.

Understanding the root cause changed everything.

[Boundaries] Working constraints

Data pipeline transformation had to maintain data quality and downstream compatibility.

Constraints are features, not obstacles. They guide design decisions.

[Decisions] Key tradeoffs

Architecture decisions balanced streaming benefits against complexity.

status="rejected" title="Managed Spark Streaming" description="EMR with Spark Structured Streaming" reasoning="Still requires cluster management. Cost structure similar to batch. Learning curve lower but operational burden remains." /> status="chosen" title="Serverless Streaming Stack" description="Lambda + Kinesis + S3" reasoning="True pay-per-use eliminates idle cluster costs. No infrastructure management. Scales automatically with load." /> status="rejected" title="Managed Kafka + Flink" description="MSK with managed Flink" reasoning="More powerful but more complex. Overkill for current requirements. Team would need significant training." />

[Design] Architecture design

Data flows from sources through Kinesis Data Streams. Lambda functions consume records and perform transformations. Processed data lands in S3 in Parquet format partitioned by time. Athena provides query access.

For aggregations requiring state, we use Kinesis Data Analytics (managed Flink) for windowed computations. Results feed into downstream Kinesis streams and ultimately to S3.

"type": "architecture", "title": "Serverless ETL Architecture", "description": "Event-driven data processing with Lambda and Kinesis", "components": [ "name": "Data Sources", "type": "external", "description": "500+ retail partners" }, "name": "API Gateway", "type": "service", "description": "Data ingestion" }, "name": "Kinesis Streams", "type": "queue", "description": "Event streaming" }, "name": "Lambda Transform", "type": "service", "description": "Stateless ETL" }, "name": "Kinesis Analytics", "type": "service", "description": "Aggregations" }, "name": "S3 Data Lake", "type": "database", "description": "Parquet storage" }, "name": "Athena", "type": "service", "description": "Query engine" } ], "connections": [ "from": "Data Sources", "to": "API Gateway", "label": "Push data" }, "from": "API Gateway", "to": "Kinesis Streams", "label": "Ingest" }, "from": "Kinesis Streams", "to": "Lambda Transform", "label": "Process" }, "from": "Lambda Transform", "to": "S3 Data Lake", "label": "Write" }, "from": "Kinesis Streams", "to": "Kinesis Analytics", "label": "Aggregate" }, "from": "Kinesis Analytics", "to": "S3 Data Lake", "label": "Results" }, "from": "S3 Data Lake", "to": "Athena", "label": "Query" } ], "boundaries": [ "name": "Streaming Platform", "contains": [ "Kinesis Streams", "Lambda Transform", "Kinesis Analytics" ] } ] }

{/* Diagram: data-flow - Incremental Aggregation Pattern */}
{/* ```json
{
  "type": "data-flow",
  "title": "Incremental Aggregation Pattern",
  "description": "Windowed aggregation for streaming data",
  "components": [
    {
      "name": "Raw Events",
      "type": "queue"
    },
    {
      "name": "Window Function",
      "type": "service"
    },
    {
      "name": "Partial Aggregate",
      "type": "database"
    },
    {
      "name": "Final Aggregate",
      "type": "database"
    }
  ],
  "connections": [
    {
      "from": "Raw Events",
      "to": "Window Function",
      "label": "5-min tumbling window"
    },
    {
      "from": "Window Function",
      "to": "Partial Aggregate",
      "label": "Window results"
    },
    {
      "from": "Partial Aggregate",
      "to": "Final Aggregate",
      "label": "Roll up hourly"
    }
  ]
}

[Approach] Execution model

Migration ran streaming and batch in parallel for 3 months. Daily reconciliation compared streaming outputs to batch. Discrepancies were investigated and streaming logic refined.

We migrated data sources progressively-low-complexity sources first, building confidence. High-complexity sources with custom transforms migrated last.

[Results] Measurable outcomes

Data latency dropped from 24 hours to 5 minutes. Customers could see retail activity in near real-time, enabling same-day business decisions.

Infrastructure costs decreased 65%. The serverless architecture eliminated idle compute. Pay-per-use meant costs scaled directly with data volume.

Engineering productivity increased. The team spent 80% of time on new features versus 40% previously. Infrastructure was no longer a concern.

Sales closed $1.8M in new enterprise deals citing real-time capability as the differentiator.

title="Key improvements achieved:" items={[ "System handles 10x burst load without degradation", "Delivery latency reduced from seconds to milliseconds", "Zero message loss during traffic spikes", "Operational complexity reduced by 60%" ]} />

[Anti-patterns] What could have gone wrong

Competitors would have continued winning deals. The 24-hour latency was a fundamental limitation that no batch optimization could solve. Market position was eroding.

Cluster costs would have grown with data volume. The batch infrastructure cost structure scaled linearly with data, eating into margins.

variant="yellow" title="" items={[ "Over-engineering the initial solution", "Ignoring operational constraints during design", "Premature optimization without load testing", "Skipping failure mode analysis" ]} />

[Applicability] Where this pattern applies

Serverless streaming applies wherever data freshness matters more than batch economics. IoT analytics, fraud detection, real-time dashboards, and operational alerting all benefit.

The pattern is less suitable for large-scale machine learning training or complex multi-step transformations that benefit from batch frameworks.

Every system is different. Context always matters.