ML Inference Pipeline at Scale Without GPU Sprawl

Batch inference architecture reduced ML infrastructure costs by 60% while improving throughput predictability.

[Overview] System Overview

Machine learning inference is the production phase of ML systems. Models trained offline must serve predictions in real time. The gap between research environments and production requirements causes most ML projects to fail.

This case examines an ML platform serving 40+ models across multiple product lines. Document classification. Fraud detection. Personalization. Each model had different latency requirements and traffic patterns.

When inference infrastructure grows organically, costs compound. GPU instances run underutilized. Teams cannot predict capacity needs. The platform becomes a cost center that engineering leadership cannot defend.

[Context] Situation before

The platform had grown from 3 models to 40 over two years. Each model deployment followed a similar pattern: request GPU instances, deploy model, hope utilization stays reasonable.

##### What Worked

Model serving infrastructure was functional. SageMaker endpoints handled traffic.

Data scientists could deploy models without infrastructure expertise.

Latency requirements were met for synchronous prediction use cases.

##### What Quietly Did Not Work

GPU utilization averaged 18% across the fleet. Most instances sat idle.

Each model ran on dedicated infrastructure. No resource sharing.

Scaling decisions were reactive. Traffic spikes caused latency degradation before auto-scaling responded.

No visibility into model-level costs. Total ML spend was known; per-model spend was not.

[Diagnosis] The real problem

The visible symptom was high infrastructure costs. The instinctive response was right-sizing. Review each endpoint, reduce instance sizes, achieve savings.

Right-sizing helps temporarily. It does not address the underlying architecture. Each model still runs independently. Resource sharing remains impossible. The next 10 models will recreate the same problem.

##### The Deeper Issue

The platform treated every inference request identically. Real-time prediction for fraud detection and batch classification for document processing used the same architecture.

Not all predictions need real-time response. Document classification can tolerate 30-second latency. Personalization recommendations can be pre-computed. Treating everything as synchronous wastes capacity.

Understanding the root cause changed everything.

[Boundaries] Working constraints

OPERATIONAL

• Cannot break existing API contracts for real-time use cases.
• Data scientists must retain deployment autonomy.
• No new ML frameworks. TensorFlow and PyTorch only.

COST

• Target: 50% cost reduction without degrading SLAs.
• New infrastructure must pay for itself within 6 months.
• No multi-year reserved instance commitments.

TEAM

• ML platform team: 2 engineers.
• No dedicated ML infrastructure specialists.
• Data scientists have varying infrastructure knowledge.

MIGRATION

• Existing models cannot be retrained.
• API schemas for inference endpoints are fixed.
• Monitoring and alerting must continue working.

IMMUTABLE

• AWS is the cloud provider.
• SageMaker is the serving infrastructure.
• Model artifacts stored in S3.

Constraints are features, not obstacles. They guide design decisions.

[Decisions] Key tradeoffs

REJECTED

Right-size existing endpoints

Analyze traffic patterns and reduce instance sizes

Achieves 15-20% savings. Does not address utilization problem. Requires ongoing management.

REJECTED

Move to serverless inference

Use SageMaker Serverless for all models

Cold start latency breaks real-time SLAs. Per-request pricing expensive at current volume.

CHOSEN

Multi-model endpoints with batch processing tier

Consolidate models on shared infrastructure. Route latency-tolerant requests to batch tier

Enables resource sharing. Maintains real-time capability. Costs scale with actual usage.

[Design] Architecture design

The architecture introduced two processing tiers: real-time and batch. A routing layer directed requests based on latency requirements.

ML Inference Pipeline Architecture

Distributed inference with GPU optimization and intelligent caching

ml-inference-pipeline-optimization-architecture

"type": "system-context", "title": "Two-Tier Inference Architecture", "description": "Shows how requests are routed to appropriate processing tier based on latency requirements.", "components": [ "name": "API Gateway", "type": "service", "description": "Request ingestion" }, "name": "Router", "type": "service", "description": "Latency-based routing" }, "name": "Real-time Tier", "type": "boundary", "description": "Multi-model endpoints" }, "name": "Batch Tier", "type": "boundary", "description": "Queue-based processing" }, "name": "Model Registry", "type": "database", "description": "Model metadata" }, "name": "Response Cache", "type": "database", "description": "Prediction caching" } ], "connections": [ "from": "API Gateway", "to": "Router", "label": "Inference request" }, "from": "Router", "to": "Real-time Tier", "label": "Low latency" }, "from": "Router", "to": "Batch Tier", "label": "High latency OK" }, "from": "Router", "to": "Response Cache", "label": "Cache check" }, "from": "Both Tiers", "to": "Model Registry", "label": "Model lookup" } ], "boundaries": [ "name": "Real-time Tier", "contains": [ "Multi-model Endpoint 1", "Multi-model Endpoint 2" ] }, "name": "Batch Tier", "contains": [ "Job Queue", "Batch Workers", "Result Store" ] } ] }

{/* Diagram: data-flow - Batch Processing Flow */}
{/* ```json
{
  "type": "data-flow",
  "title": "Batch Processing Flow",
  "description": "Shows how batch requests are queued, processed, and returned.",
  "components": [
    {
      "name": "Batch API",
      "type": "service"
    },
    {
      "name": "SQS Queue",
      "type": "queue"
    },
    {
      "name": "Batch Workers",
      "type": "service"
    },
    {
      "name": "Multi-model Endpoint",
      "type": "service"
    },
    {
      "name": "Result Store",
      "type": "database"
    },
    {
      "name": "Callback Service",
      "type": "service"
    }
  ],
  "connections": [
    {
      "from": "Batch API",
      "to": "SQS Queue",
      "label": "Enqueue"
    },
    {
      "from": "SQS Queue",
      "to": "Batch Workers",
      "label": "Pull"
    },
    {
      "from": "Batch Workers",
      "to": "Multi-model Endpoint",
      "label": "Batched inference"
    },
    {
      "from": "Batch Workers",
      "to": "Result Store",
      "label": "Store result"
    },
    {
      "from": "Batch Workers",
      "to": "Callback Service",
      "label": "Notify",
      "style": "dashed"
    }
  ],
  "boundaries": []
}

[Approach] Execution model

##### Phase 1: Model Classification (2 weeks)

Analyzed all 40 models for latency requirements. Categorized into real-time (must respond in under 200ms) and batch-eligible (can tolerate up to 60 seconds). Result: 12 real-time, 28 batch-eligible.

##### Phase 2: Multi-model Consolidation (4 weeks)

Grouped models by framework and resource requirements. Created 4 multi-model endpoints instead of 40 individual endpoints. Real-time models consolidated first.

##### Phase 3: Batch Tier Implementation (3 weeks)

Built queue-based processing for batch-eligible models. Implemented request batching for GPU efficiency. Added callback mechanism for async completion.

##### Phase 4: Traffic Migration (3 weeks)

Migrated traffic incrementally. Started with lowest-risk models. Monitored latency and accuracy throughout. Shadow mode for batch tier before full cutover.

[Results] Measurable outcomes

##### Cost Reduction

ML infrastructure costs decreased 60%. From $180K/month to $72K/month. Savings came from GPU consolidation and batch processing efficiency.

##### Utilization Improvement

Average GPU utilization increased from 18% to 67%. Multi-model endpoints and request batching enabled efficient resource sharing.

##### Throughput Predictability

Batch tier handles traffic spikes without latency degradation. Queue depth provides early warning for capacity planning.

##### Model Deployment

Data scientists deploy to shared infrastructure. No per-model capacity planning required. Platform team manages 4 endpoints instead of 40.

Key improvements achieved:

• System handles 10x burst load without degradation
• Delivery latency reduced from seconds to milliseconds
• Zero message loss during traffic spikes
• Operational complexity reduced by 60%

[Anti-patterns] What could have gone wrong

##### Continued Organic Growth

Each new model would add dedicated infrastructure. Costs would scale linearly with model count. By 100 models, infrastructure would be unmanageable.

##### Right-sizing Only

Short-term savings would be achieved. Utilization would still average under 25%. The next traffic spike would require scaling back up.

• Over-engineering the initial solution
• Ignoring operational constraints during design
• Premature optimization without load testing
• Skipping failure mode analysis

[Applicability] Where this pattern applies

##### Good Fit

Organizations running 10+ ML models in production.

Platforms with mixed latency requirements.

Teams without dedicated ML infrastructure engineers.

Cost pressure on ML infrastructure.

##### Poor Fit

All models require sub-100ms latency.

Traffic is constant with no batching opportunity.

Fewer than 5 models in production.

Every system is different. Context always matters.