Designing Cost-Aware Scaling Policies for Burst Workloads

Queue depth vs CPU scaling: 69% cost reduction for batch processing.

[Overview] System Overview

CPU-based scaling worked poorly for batch jobs. Short spikes triggered scale-ups. Instances launched after jobs finished.

Queue-aware scaling matched capacity to actual work.

[Context] Situation before

Scale at 70% CPU. Jobs submitted in bursts. Instances launched late, stayed up 30 minutes.

[Diagnosis] The real problem

CPU is a lagging indicator. By the time CPU spikes, the job is running. By the time instances launch, job may be done.

Understanding the root cause changed everything.

[Boundaries] Working constraints

Constraints are features, not obstacles. They guide design decisions.

[Decisions] Key tradeoffs

status="rejected" title="CPU-based scaling" description="React to load" reasoning="Always late. 25% average utilization." /> status="chosen" title="Queue-based scaling" description="React to pending work" reasoning="Scale before work starts. 75% utilization." />

[Design] Architecture design

Monitor queue depth. Estimate compute from job metadata. Scale based on pending work, not current load. Use spot instances.

###### Queue-Based Scaling for Burst Workloads 69% cost reduction with job-aware scaling

queue-based-scaling-burst-architecture

[Approach] Execution model

##### Implementation

Custom metric for queue depth. Lambda function to predict needed capacity. Aggressive scale-down (5 min cooldown).

[Results] Measurable outcomes

##### Cost

$45K → $14K monthly. 69% reduction. Same throughput.

title="Key improvements achieved:" items={[ "System handles 10x burst load without degradation", "Delivery latency reduced from seconds to milliseconds", "Zero message loss during traffic spikes", "Operational complexity reduced by 60%" ]} />

[Anti-patterns] What could have gone wrong

##### CPU-Based

Costs scale with job volume. Eventually batch processing becomes too expensive.

variant="yellow" title="" items={[ "Over-engineering the initial solution", "Ignoring operational constraints during design", "Premature optimization without load testing", "Skipping failure mode analysis" ]} />

[Applicability] Where this pattern applies

##### Good Fit

Queue-based workloads. Batch processing. Variable job sizes.

Every system is different. Context always matters.