Serverless Data Processing Platform for IoT Analytics

Built serverless data pipeline processing 10B daily events from 5M IoT devices, achieving sub-second analytics while reducing infrastructure costs 70% compared to...

[Overview] System Overview

An industrial IoT company needed to process telemetry from 5 million sensors across manufacturing facilities worldwide. Their Kafka-Spark architecture was expensive to operate and struggled with variable load patterns.

We designed a serverless-first architecture using AWS Kinesis, Lambda, and DynamoDB. The system processes 10 billion events daily with sub-second latency for real-time anomaly detection.

Infrastructure costs dropped 70%. More importantly, the system auto-scales from quiet periods (weekends) to peak manufacturing hours without manual intervention.

The architecture includes edge processing for initial filtering, real-time stream processing for anomaly detection, and batch analytics for historical analysis.

[Context] Situation before

The existing architecture was a traditional Kafka-to-Spark pipeline. It worked but was expensive and operationally heavy.

##### Cost Pressure

Kafka cluster: $45K/month. Running 24/7 whether data flowed or not.

Spark cluster: $60K/month. Sized for peak load, idle most of the time.

Data engineering team: 50% time on infrastructure maintenance.

##### Scaling Challenges

Weekend device volume was 10% of weekday peaks. Infrastructure costs were flat.

New facility onboarding required capacity planning weeks in advance.

Holiday shutdowns meant paying for infrastructure processing near-zero data.

[Diagnosis] The real problem

The surface problem was high infrastructure cost. The reflexive solution was "optimize Spark/Kafka."

But the real problem was the architectural mismatch. IoT data is inherently bursty-sensors report more during active manufacturing. Traditional infrastructure prices for peak capacity.

Serverless pricing matches workload patterns. You pay for events processed, not capacity provisioned.

Understanding the root cause changed everything.

[Boundaries] Working constraints

IoT migration must not lose device telemetry.

Constraints are features, not obstacles. They guide design decisions.

[Decisions] Key tradeoffs

Serverless migration traded familiar tools for cost efficiency.

status="rejected" title="Optimize Existing (Kafka + Spark)" description="Right-size clusters, use spot instances" reasoning="Would achieve 20-30% savings max. Doesn't address the fundamental mismatch. Still requires operational overhead." /> status="rejected" title="Managed Streaming (MSK + EMR)" description="AWS-managed Kafka and Spark" reasoning="Reduces operational burden but still capacity-priced. Doesn't solve cost-scaling mismatch." /> status="chosen" title="Serverless (Kinesis + Lambda)" description="Event-driven, pay-per-use" reasoning="Cost scales with usage. Zero idle cost. Auto-scales instantly. Trades Spark expertise for Lambda learning curve." />

[Design] Architecture design

The architecture is fully serverless: Kinesis for ingestion, Lambda for processing, DynamoDB for hot storage, S3 for cold storage, and Athena for ad-hoc analytics.

Edge processing at the gateway filters and aggregates before cloud ingestion-reducing data volume 80% for normal operation.

Anomaly detection runs in real-time Lambda functions with sub-second latency. ML models are deployed as Lambda layers.

"type": "architecture", "title": "Serverless IoT Data Platform", "description": "Event-driven processing with auto-scaling", "components": [ "name": "IoT Devices", "type": "external", "description": "5M sensors" }, "name": "IoT Gateway", "type": "service", "description": "Edge processing" }, "name": "Kinesis", "type": "queue", "description": "Stream ingestion" }, "name": "Lambda", "type": "service", "description": "Stream processing" }, "name": "DynamoDB", "type": "database", "description": "Hot storage" }, "name": "S3", "type": "database", "description": "Cold storage" }, "name": "Athena", "type": "service", "description": "Analytics queries" } ], "connections": [ "from": "IoT Devices", "to": "IoT Gateway", "label": "MQTT" }, "from": "IoT Gateway", "to": "Kinesis", "label": "Filtered events" }, "from": "Kinesis", "to": "Lambda", "label": "Trigger" }, "from": "Lambda", "to": "DynamoDB", "label": "Real-time" }, "from": "Lambda", "to": "S3", "label": "Archive" }, "from": "S3", "to": "Athena", "label": "Query" } ], "boundaries": [ "name": "Serverless Core", "contains": [ "Kinesis", "Lambda", "DynamoDB" ] } ] }

{/* Diagram: data-flow - Edge to Cloud Flow */}
{/* ```json
{
  "type": "data-flow",
  "title": "Edge to Cloud Flow",
  "description": "Data reduction through edge processing",
  "components": [
    {
      "name": "Sensor",
      "type": "external"
    },
    {
      "name": "Gateway (Edge)",
      "type": "service"
    },
    {
      "name": "Cloud (Kinesis)",
      "type": "queue"
    },
    {
      "name": "Lambda",
      "type": "service"
    }
  ],
  "connections": [
    {
      "from": "Sensor",
      "to": "Gateway (Edge)",
      "label": "1000 events/min"
    },
    {
      "from": "Gateway (Edge)",
      "to": "Cloud (Kinesis)",
      "label": "200 events/min (80% filtered)"
    },
    {
      "from": "Cloud (Kinesis)",
      "to": "Lambda",
      "label": "Process"
    }
  ]
}

[Approach] Execution model

Migration proceeded facility-by-facility over 6 months.

Month 1-2: Serverless infrastructure, shadow processing (parallel to existing).

Month 3-4: First facility fully migrated. Validation of accuracy vs legacy.

Month 5-6: Remaining facilities migrated. Legacy Kafka/Spark decommissioned.

[Results] Measurable outcomes

Cost: $105K/month → $32K/month (70% reduction).

Scaling: Automatic. Peak loads handled without pre-provisioning.

Latency: Sub-second anomaly detection maintained.

Operations: Data team reclaimed 50% of time from infrastructure work.

Flexibility: New facility onboarding reduced from weeks to hours.

title="Key improvements achieved:" items={[ "System handles 10x burst load without degradation", "Delivery latency reduced from seconds to milliseconds", "Zero message loss during traffic spikes", "Operational complexity reduced by 60%" ]} />

[Anti-patterns] What could have gone wrong

Optimizing existing architecture would have achieved 20-30% savings-leaving significant cost on the table.

As device count grows to 10M+, traditional architecture costs would scale linearly. Serverless costs scale sub-linearly.

Continued operational burden on data team would limit capacity for analytics improvements.

variant="yellow" title="" items={[ "Over-engineering the initial solution", "Ignoring operational constraints during design", "Premature optimization without load testing", "Skipping failure mode analysis" ]} />

[Applicability] Where this pattern applies

Serverless data processing applies when: workload is bursty, cost efficiency matters more than ultimate performance, and team capacity for infrastructure is limited.

The pattern is less suitable for: ultra-low-latency requirements (<50ms), extremely high throughput sustained loads, or workloads requiring stateful processing.

Every system is different. Context always matters.