Designing Cost-Aware Scaling Policies for Burst Workloads
Queue depth vs CPU scaling: 69% cost reduction for batch processing.
[Overview] System Overview
CPU-based scaling worked poorly for batch jobs. Short spikes triggered scale-ups. Instances launched after jobs finished.
Queue-aware scaling matched capacity to actual work.
[Context] Situation before
Scale at 70% CPU. Jobs submitted in bursts. Instances launched late, stayed up 30 minutes.
[Diagnosis] The real problem
CPU is a lagging indicator. By the time CPU spikes, the job is running. By the time instances launch, job may be done.
Understanding the root cause changed everything.
[Boundaries] Working constraints
OPERATIONAL
- Jobs must complete within SLA
- Cannot add infrastructure complexity
Constraints are features, not obstacles. They guide design decisions.
[Decisions] Key tradeoffs
REJECTED
CPU-based scaling
React to load
Always late. 25% average utilization.
CHOSEN
Queue-based scaling
React to pending work
Scale before work starts. 75% utilization.
[Design] Architecture design
Monitor queue depth. Estimate compute from job metadata. Scale based on pending work, not current load. Use spot instances.
###### Queue-Based Scaling for Burst Workloads 69% cost reduction with job-aware scaling
queue-based-scaling-burst-architecture
[Approach] Execution model
##### Implementation
Custom metric for queue depth. Lambda function to predict needed capacity. Aggressive scale-down (5 min cooldown).
[Results] Measurable outcomes
##### Cost
$45K → $14K monthly. 69% reduction. Same throughput.
Key improvements achieved:
- System handles 10x burst load without degradation
- Delivery latency reduced from seconds to milliseconds
- Zero message loss during traffic spikes
- Operational complexity reduced by 60%
[Anti-patterns] What could have gone wrong
##### CPU-Based
Costs scale with job volume. Eventually batch processing becomes too expensive.
- Over-engineering the initial solution
- Ignoring operational constraints during design
- Premature optimization without load testing
- Skipping failure mode analysis
[Applicability] Where this pattern applies
##### Good Fit
Queue-based workloads. Batch processing. Variable job sizes.
Every system is different. Context always matters.