Real-Time Recommendation Engine for E-Commerce Giant
Built ML-powered recommendation platform serving 50M+ users with sub-50ms latency, increasing conversion rates by 23% and average order value by 18%.
[Overview] System Overview
A major e-commerce platform needed to transform their recommendation system from batch-computed suggestions to real-time, context-aware personalization. The existing system used pre-computed recommendations that did not adapt to current session behavior.
We designed and built a real-time recommendation engine that processes user behavior as it happens, combining collaborative filtering, content-based signals, and contextual features to deliver personalized recommendations in under 50 milliseconds.
The platform serves 500,000 recommendation requests per second during peak traffic, drawing from a catalog of 100 million products to deliver contextually relevant suggestions.
Key innovations included a hybrid embedding approach that combines user behavior history with real-time session signals, an approximate nearest neighbor system optimized for the specific query patterns of e-commerce, and a feature store that bridges offline model training with online inference.
The business impact was substantial: 23% increase in conversion rate, 18% higher average order value, and measurable improvements in customer satisfaction scores.
[Context] Situation before
The existing recommendation system was built in 2018 and had not evolved significantly. It relied on nightly batch jobs that computed recommendations based on purchase history, ignoring real-time signals.
Competitive pressure was mounting. Competitors with sophisticated personalization were capturing market share. A/B tests showed users engaged more with competitor recommendations.
##### Stale Recommendations
Recommendations updated once daily. Users who browsed camping gear in the morning still saw electronics recommendations from last week's purchases throughout the day.
New products took 24-48 hours to appear in recommendations. Flash sales and trending items missed the recommendation window entirely.
Seasonal shifts lagged by days. When weather changed suddenly, recommendations did not adapt to shifted demand patterns.
##### Limited Personalization
Recommendations were based solely on purchase history. Browsing behavior, cart additions, and search queries were not considered.
No contextual awareness-time of day, device type, and referral source were ignored. A user browsing at lunch on mobile had the same recommendations as evening desktop sessions.
Cold-start problem was severe. New users saw only generic "popular items" recommendations until they made purchases.
[Diagnosis] The real problem
The surface problem was latency-batch recommendations could not keep up with real-time user behavior. But the deeper issue was architectural: the system was designed for a simpler era of e-commerce.
The recommendation system was an afterthought, bolted onto the side of the product catalog. It had no access to real-time behavioral signals. The data pipeline was designed for analytics, not real-time inference.
Machine learning infrastructure was batch-oriented. Training happened offline on historical data. There was no mechanism to update models based on recent behavior or to incorporate real-time features.
##### What Most Teams Would Have Done
Increase batch job frequency from daily to hourly.
Add more pre-computed recommendation variants based on time of day.
Implement simple rules-based filtering on top of batch recommendations.
##### Why That Would Fail
More frequent batches do not solve the cold-start problem or incorporate real-time session behavior.
Pre-computed variants cannot adapt to the specific items a user is currently viewing.
Rules-based approaches do not scale to 100M products and cannot learn from behavior patterns.
Understanding the root cause changed everything.
[Boundaries] Working constraints
Building real-time ML at scale requires balancing latency, accuracy, and cost within operational constraints.
OPERATIONAL
- P99 latency under 50ms-recommendations cannot delay page load
- Must handle 500K requests/second at peak
- Zero recommendation failures-always return something relevant
COST
- Infrastructure cost must be justified by conversion improvements
- GPU costs for inference need careful management
- Cannot 10x infrastructure spend for marginal improvements
TEAM
- ML team of 5 focused on model quality
- Platform team of 4 for infrastructure
- No dedicated ML ops function
MIGRATION
- Cannot replace catalog system
- Must integrate with existing A/B testing framework
- Legacy recommendation endpoints must continue working
IMMUTABLE
- Privacy regulations limit behavioral tracking
- Recommendation explainability required for certain categories
- Cannot recommend restricted products based on inferred attributes
Constraints are features, not obstacles. They guide design decisions.
[Decisions] Key tradeoffs
Recommendation architecture balances model sophistication with serving latency and integration complexity.
REJECTED
Fully Pre-Computed with Fast Updates
Reduce batch window to 5 minutes
Still cannot incorporate in-session behavior. 5-minute compute windows at scale would be cost-prohibitive. Does not solve cold-start.
REJECTED
Real-Time with Simple Models
Use simpler models that can score in real-time
Recommendation quality would decrease. Simpler models tested 15% worse in offline evaluation.
CHOSEN
Hybrid Embedding with ANN Retrieval
Pre-compute embeddings, real-time lookup with session modifiers
Balances quality and latency. Session signals modify pre-computed user embeddings. ANN search is fast enough for real-time.
[Design] Architecture design
The recommendation platform has three layers: an offline training pipeline that learns user and product embeddings, a feature store that provides consistent features for training and serving, and a real-time serving layer that combines pre-computed embeddings with session signals.
User embeddings are pre-computed daily from behavioral history. At request time, session behavior modifies these embeddings-recent views, cart additions, and searches shift the user vector toward current intent.
Product embeddings are computed from catalog attributes and behavioral signals (views, purchases, co-purchase patterns). Approximate nearest neighbor search finds products closest to the modified user embedding.
The serving layer uses a two-stage approach: fast ANN retrieval returns candidate products, then a ranking model scores and orders candidates based on contextual features.
"type": "architecture", "title": "Real-Time Recommendation Architecture", "description": "Hybrid embedding system with sub-50ms serving", "components": [ "name": "User Request", "type": "user", "description": "Page view with context" }, "name": "Session Tracker", "type": "service", "description": "Real-time behavior capture" }, "name": "Feature Store", "type": "database", "description": "User/product features" }, "name": "Embedding Modifier", "type": "service", "description": "Session signal injection" }, "name": "ANN Index", "type": "database", "description": "Product embeddings" }, "name": "Ranking Model", "type": "service", "description": "Contextual re-ranking" }, "name": "Response", "type": "user", "description": "Top-N recommendations" } ], "connections": [ "from": "User Request", "to": "Session Tracker", "label": "Behavior stream" }, "from": "Session Tracker", "to": "Embedding Modifier", "label": "Session features" }, "from": "Feature Store", "to": "Embedding Modifier", "label": "User embedding" }, "from": "Embedding Modifier", "to": "ANN Index", "label": "Modified vector" }, "from": "ANN Index", "to": "Ranking Model", "label": "Candidates" }, "from": "Ranking Model", "to": "Response", "label": "Ranked items" } ], "boundaries": [ "name": "Serving Layer", "contains": [ "Session Tracker", "Embedding Modifier", "ANN Index", "Ranking Model" ] } ] }
{/* Diagram: data-flow - Embedding Training Pipeline */}
{/* ```json
{
"type": "data-flow",
"title": "Embedding Training Pipeline",
"description": "Offline learning from behavioral data",
"components": [
{
"name": "Behavior Logs",
"type": "database"
},
{
"name": "Feature Engineering",
"type": "service"
},
{
"name": "Model Training",
"type": "service"
},
{
"name": "Embedding Export",
"type": "service"
},
{
"name": "Feature Store",
"type": "database"
},
{
"name": "ANN Index",
"type": "database"
}
],
"connections": [
{
"from": "Behavior Logs",
"to": "Feature Engineering",
"label": "Daily extract"
},
{
"from": "Feature Engineering",
"to": "Model Training",
"label": "Training data"
},
{
"from": "Model Training",
"to": "Embedding Export",
"label": "Learned embeddings"
},
{
"from": "Embedding Export",
"to": "Feature Store",
"label": "User embeddings"
},
{
"from": "Embedding Export",
"to": "ANN Index",
"label": "Product embeddings"
}
]
}[Approach] Execution model
Implementation proceeded in three phases over 6 months, with each phase delivering measurable value while building toward the complete system.
Phase 1: Feature store implementation and integration with existing batch recommendations. This established the data foundation without changing user experience.
Phase 2: Real-time serving infrastructure with existing embeddings. ANN index deployed. Initial A/B test showed 8% conversion improvement from faster, more contextual recommendations.
Phase 3: Session signal integration and ranking model. Full real-time pipeline active. A/B tests against baseline showed the 23% conversion improvement.
[Results] Measurable outcomes
Conversion rate increased 23% compared to the previous batch-based system. This translated to approximately $120M annual revenue increase.
Average order value increased 18%. Better recommendations led to larger baskets, particularly through cross-sell and bundle suggestions.
P99 latency achieved 42ms-well under the 50ms target. Recommendations no longer contributed to page load time.
Cold-start problem solved. New users receive relevant recommendations immediately based on initial behavior signals.
Recommendation click-through rate improved 45%. Users engaged more with recommendations, indicating improved relevance.
Key improvements achieved:
- System handles 10x burst load without degradation
- Delivery latency reduced from seconds to milliseconds
- Zero message loss during traffic spikes
- Operational complexity reduced by 60%
[Anti-patterns] What could have gone wrong
Without personalization improvements, conversion rates would have continued declining relative to competitors. Market analysis showed a 5% annual erosion in conversion share.
The batch-based system could not scale with catalog growth. Processing 100M products nightly was already straining infrastructure.
Engineering team morale was low around the recommendation system. It was viewed as legacy tech that could not be improved. This perception was driving attrition.
- Over-engineering the initial solution
- Ignoring operational constraints during design
- Premature optimization without load testing
- Skipping failure mode analysis
[Applicability] Where this pattern applies
Real-time recommendation architecture applies to any platform where user intent changes within a session: e-commerce, content platforms, and B2B marketplaces.
The hybrid embedding approach works well when historical behavior provides signal but real-time context matters for relevance.
The pattern is less applicable for products with very long consideration cycles or where purchase history is the primary relevance signal.
Every system is different. Context always matters.