Personalized Recommendation Engine at Streaming Scale

Built real-time ML recommendation system serving 500M+ daily recommendations with 15ms p99 latency, increasing user engagement by 34%.

[Overview] System Overview

A video streaming platform with 80 million active users needed personalized content recommendations to drive engagement. Users were overwhelmed by the content catalog-finding relevant content was the primary friction point. Better recommendations meant more viewing time and better retention.

The recommendation system needed to serve personalized recommendations across multiple surfaces: home page, search, end-of-video suggestions, email campaigns, and push notifications. Each surface had different latency requirements and ranking criteria. The system processed 500 million recommendation requests daily.

We built a two-stage recommendation architecture: candidate generation produced thousands of potentially relevant items, then ranking models scored and ordered candidates based on user context and business objectives. Real-time feature computation enabled immediate responsiveness to user behavior.

[Context] Situation before

The existing system used collaborative filtering computed in batch. Recommendations updated once daily, meaning new users saw generic content for 24 hours. New content was invisible until the next batch run.

##### Cold Start Problems

New users received poor recommendations based only on demographics. It took 2-3 weeks of usage before recommendations became relevant. Many users churned before reaching this point.

##### Lack of Context Awareness

Recommendations ignored viewing context. Watching on mobile versus TV suggested different content preferences. Time of day and day of week patterns were not captured.

[Diagnosis] The real problem

Batch processing could not capture the real-time nature of user preferences. A user who just finished an action movie has immediate preference signals that should influence the next recommendation. Waiting 24 hours lost this signal value.

The recommendation quality was limited by feature availability. Powerful features like session behavior, real-time popularity, and fresh content signals were not available to models trained on batch data.

Understanding the root cause changed everything.

[Boundaries] Working constraints

Recommendation system constraints spanned latency, scale, and model complexity.

OPERATIONAL

• P99 latency must be under 20ms for home page
• Must serve 10K+ QPS per region
• Recommendations must be explainable for editorial review

COST

• Infrastructure costs capped at $150K monthly
• Cannot use GPUs for real-time inference at our scale

TEAM

• Data science team of 5 must be able to iterate on models
• ML platform must support A/B testing without engineering support

MIGRATION

• Cannot replace existing system at once-must run parallel
• Editorial team must approve recommendation changes

IMMUTABLE

• Content licensing restrictions must be enforced
• Parental controls must be respected absolutely

Constraints are features, not obstacles. They guide design decisions.

[Decisions] Key tradeoffs

Recommendation architecture trade-offs center on model complexity versus latency.

REJECTED

Single End-to-End Model

One model from user to ranked recommendations

Cannot meet latency requirements with model complex enough for our catalog size. Would need to score millions of items per request.

CHOSEN

Two-Stage Retrieval + Ranking

Fast candidate retrieval, then complex ranking

Separates concerns: retrieval can use efficient approximate methods, ranking can use complex models on smaller candidate sets. Industry-proven pattern.

REJECTED

Cache-Heavy Approach

Pre-compute recommendations, serve from cache

Cannot capture real-time signals. Personalization is limited to what can be pre-computed. Cache invalidation is complex with changing preferences.

[Design] Architecture design

The system implements a two-stage architecture. Candidate generation uses approximate nearest neighbor search on learned embeddings, retrieving 1000 candidates in under 5ms. The ranking model scores candidates using user features, item features, and context features.

Real-time features are computed from Kafka streams and served from Redis. Batch features from user history are computed in Spark and loaded to feature stores. The ranking model accesses both feature types during inference.

"type": "architecture", "title": "Two-Stage Recommendation Architecture", "description": "Candidate generation with ranking pipeline", "components": [ "name": "User Request", "type": "user" }, "name": "API Gateway", "type": "service" }, "name": "Candidate Generator", "type": "service", "description": "ANN search" }, "name": "Embedding Store", "type": "database", "description": "Item embeddings" }, "name": "Ranking Service", "type": "service", "description": "ML scoring" }, "name": "Feature Store", "type": "database", "description": "Real-time + batch" }, "name": "Business Rules", "type": "service", "description": "Filtering" } ], "connections": [ "from": "User Request", "to": "API Gateway", "label": "Get recs" }, "from": "API Gateway", "to": "Candidate Generator", "label": "1. Retrieve" }, "from": "Candidate Generator", "to": "Embedding Store", "label": "ANN query" }, "from": "API Gateway", "to": "Ranking Service", "label": "2. Rank candidates" }, "from": "Ranking Service", "to": "Feature Store", "label": "Get features" }, "from": "Ranking Service", "to": "Business Rules", "label": "3. Filter" } ], "boundaries": [ "name": "ML Platform", "contains": [ "Candidate Generator", "Embedding Store", "Ranking Service", "Feature Store" ] } ] }

{/* Diagram: data-flow - Feature Pipeline */}
{/* ```json
{
  "type": "data-flow",
  "title": "Feature Pipeline",
  "description": "Real-time and batch feature computation",
  "components": [
    {
      "name": "User Events",
      "type": "queue",
      "description": "Kafka"
    },
    {
      "name": "Stream Processor",
      "type": "service",
      "description": "Flink"
    },
    {
      "name": "Real-time Store",
      "type": "database",
      "description": "Redis"
    },
    {
      "name": "Batch Processor",
      "type": "service",
      "description": "Spark"
    },
    {
      "name": "Batch Store",
      "type": "database",
      "description": "Feature store"
    }
  ],
  "connections": [
    {
      "from": "User Events",
      "to": "Stream Processor",
      "label": "Consume"
    },
    {
      "from": "Stream Processor",
      "to": "Real-time Store",
      "label": "Update"
    },
    {
      "from": "User Events",
      "to": "Batch Processor",
      "label": "Daily load"
    },
    {
      "from": "Batch Processor",
      "to": "Batch Store",
      "label": "Compute"
    }
  ]
}

[Approach] Execution model

Implementation proceeded in three phases. Phase 1 built infrastructure: feature stores, model serving, and A/B testing platform. Phase 2 deployed initial models trained on historical data. Phase 3 introduced real-time features and advanced models.

A/B testing ran continuously. Each model change was validated against control before full rollout. Guardrail metrics prevented degradation of user experience.

[Results] Measurable outcomes

User engagement increased 34% as measured by viewing hours. Users found relevant content faster and watched more. The home page completion rate doubled.

Session length increased 28%. Users explored more content when recommendations were relevant. Previously, users would give up searching and leave.

New user retention improved 22%. Real-time adaptation meant new users saw good recommendations within minutes instead of days. First-week retention is a leading indicator of long-term subscription.

P99 latency achieved 15ms, well under the 20ms target. The two-stage architecture efficiently balanced quality and performance.

Key improvements achieved:

• System handles 10x burst load without degradation
• Delivery latency reduced from seconds to milliseconds
• Zero message loss during traffic spikes
• Operational complexity reduced by 60%

[Anti-patterns] What could have gone wrong

Without improved recommendations, user engagement would have continued declining. Competitors with better personalization were gaining market share. User surveys consistently cited content discovery as the primary frustration.

The batch system could not scale to new recommendation surfaces. Adding push notifications or email personalization required new batch jobs with inconsistent models. Real-time recommendations unified all surfaces.

• Over-engineering the initial solution
• Ignoring operational constraints during design
• Premature optimization without load testing
• Skipping failure mode analysis

[Applicability] Where this pattern applies

Two-stage recommendation architecture applies to any catalog too large for single-model scoring. E-commerce product recommendations, news article ranking, ad targeting, and social media feeds all use similar patterns.

Real-time feature computation matters wherever user behavior provides immediate preference signals. The patterns translate across domains with different content types.

Every system is different. Context always matters.