Designing Embedding-Based Recommendations
A full walkthrough for a recommendation feed serving 100M users over 10M items, covering the event and feature pipeline, two-tower embeddings, ANN candidate generation, multi-stage ranking, real-time user signals, cold start with content embeddings, where LLMs and generative retrieval fit, feedback loops, integrity and evaluation.
Lesson 10 of 10 16 min
Recommendation systems were the original "embeddings at scale" systems, long before RAG. They're a good design problem because they combine everything: streaming data, feature stores, ANN search, several ML models under a tight latency budget, and product decisions about what "good" even means. GenAI adds a new layer: LLMs and multimodal encoders that understand content deeply, which helps most exactly where classic recommenders are weakest: new items, sparse data and explanations.
1. Clarify the task and the quality bar
Questions to ask:
- What content? Assume a content platform with articles and short videos from creators: 10M active items, ≈ 500K new each day.
- Which surfaces? A personalised home feed, "more like this" on each item, and notifications later.
- Scale? 100M users, ≈ 20 feed requests per user per day.
- What is the objective? Not just clicks. Clicks reward clickbait; the product wants satisfied users who come back, and a healthy creator ecosystem. So the objective combines engagement, satisfaction signals (likes, completion, "not interested") and long-term retention.
- Constraints? Content-safety settings, regional rights, already-seen items, and in some regions a legal requirement to offer a non-personalised feed.
Functional requirements
- Personalised home feed and item-to-item "more like this"
- New items recommendable within minutes of publishing
- Filters: region, language, safety settings, seen items, blocked creators
- User controls: "not interested", "show less of this topic", reset recommendations
Non-functional requirements
- 100M users, 10M active items
- 2B feed requests a day, p99 under 150 ms for the recommendation service
- Interest changes reflected within a session
- Continuous improvement through A/B testing
2. Choose the operating point
Latency and throughput are hard constraints at ≈ 70K QPS peak. Relevance and long-term satisfaction are the objective within them. LLMs and large encoders sit offline and near-line (content understanding, embeddings, labels), not in the per-request path, for the same reasons as in semantic product search.
3. Estimates
| Quantity | Value |
|---|---|
| Feed requests per day | 100M × 20 = 2B, ≈ 23K QPS average, ≈ 70K peak |
| Item embeddings (256-dim, fp16) | 10M × 512 B ≈ 5.1 GB, which fits in RAM on every node, so replicate rather than shard |
| HNSW graph (M = 16) | 10M × 128 B × 1.1 ≈ 1.4 GB |
| ANN QPS per node | Several thousand at small dimensions, so ≈ 20–30 nodes for 70K QPS with headroom |
| Ranking | 70K requests × 1,000 candidates = 70M candidate scores per second across stages: the real compute cost |
| Interaction events | 2B requests × ≈ 20 items shown → ≈ 40B impressions a day, plus clicks and dwell: a large streaming pipeline |
| New items per day | ≈ 500K, enriched and embedded within minutes |
| LLM content understanding | 500K items × ≈ 2K tokens ≈ 1B tokens a day: ≈ $250 a day with a small model |
4. Architecture
Online
Feed request
user_id, context
User embedding
user tower on recent activity
Candidate generation
ANN top 1,000 + other sources
Filter
seen, region, safety
Rankers
light → heavy, top 50
Re-rank
diversity, freshness, rules
Offline and near-line
Event stream
impressions, clicks, dwell
Feature pipelines
batch + streaming
Train two-tower + rankers
daily / continuous
Item embeddings
batch + streaming for new items
ANN index + feature store
incremental updates
Components
| Component | Responsibility |
|---|---|
| Event pipeline | Impression, click, dwell, like, hide and share events via Kafka into streaming and batch stores |
| Feature store | One definition per feature, materialised offline for training and online (low-latency key-value) for serving |
| Content pipeline | For new items: LLM and multimodal understanding, safety classification, content embeddings |
| Two-tower model | User and item encoders trained on interactions |
| ANN service | Replicated HNSW index of item vectors with incremental inserts and filtering |
| Candidate sources | ANN, item-to-item, follow graph, trending, fresh content |
| Rankers | Light and heavy ranking models, served on GPU or CPU with batching |
| Re-ranker | Diversity, freshness, integrity and business rules |
| Experimentation | A/B assignment, metrics, long-term holdouts |
5. Deep dives
Deep dive A: two-tower candidate generation
- A user tower encodes user features and recent interaction history into a vector. An item tower encodes item features and content. They are trained so the dot product predicts engagement, using a contrastive loss with in-batch negatives.
- Negatives matter: in-batch negatives over-represent popular items, so apply a sampling-bias (logQ) correction and add hard negatives (shown but skipped).
- Item vectors are precomputed and indexed with ANN. The user vector is computed per request in a few milliseconds, or near-line after each interaction.
- Multiple candidate sources are merged: two-tower ANN, "similar to recently engaged items" (item-to-item ANN), creators the user follows, trending by region, and fresh content. This improves coverage and makes the system robust when one source degrades.
- Filtering during retrieval for hard constraints (region, language, safety settings), and a bloom filter or recent-history set for already-seen items.
See Embeddings for system designers and HNSW explained.
Deep dive B: the ranking funnel
| Stage | Candidates | Model | Budget |
|---|---|---|---|
| Candidate generation | 10M → 1,000 | ANN dot product | ≈ 10–20 ms |
| Light ranker | 1,000 → 300 | Small model on cheap features | ≈ 10 ms |
| Heavy ranker | 300 → 50 | Deep model with cross features | ≈ 30–50 ms |
| Re-rank | 50 → 20 shown | Diversity (MMR), freshness, business rules | ≈ 5 ms |
- Multi-task ranking: the heavy ranker predicts several outcomes (click, dwell time, like, share, hide, report). A value model combines them into one score with weights that encode product goals. Changing those weights is a product decision with A/B tests, not a modelling detail.
- Calibration: predicted probabilities must be accurate, not just well-ordered, because they're combined across tasks.
- Feature fetching is often the latency bottleneck: hundreds of features for 300 candidates. Batch lookups, cache item features locally, and precompute expensive cross features.
Deep dive C: real-time signals and training-serving consistency
- Session signals: someone who just watched three cooking videos should see more cooking now, not tomorrow. Update the user embedding near-line from the event stream, or use a sequence model (a transformer over the last N interactions) as the user tower.
- Streaming features: counts over sliding windows (item CTR in the last hour, user's topic mix today) computed by a stream processor.
- Training-serving skew is the most common silent bug: a feature computed one way in training and another in serving. Define each feature once in the feature store and log the served feature values with each impression, then train on those logs.
- Model freshness: retrain daily at minimum; some rankers update continuously on streaming data.
Deep dive D: cold start and freshness
- New items have no interactions. Their item-tower input includes content embeddings (a text encoder on title and description, an image or video encoder on frames, or LLM-generated summaries and tags), so they land near similar items immediately. Insert them into the ANN index incrementally within minutes.
- Exploration: reserve a small share of impressions for new or uncertain items, using bandit-style strategies, so they collect signal quickly. Creators care intensely about this "first distribution".
- New users: onboarding topic choices, popular-by-region items, and session signals that reshape the user vector within the first few interactions.
Deep dive E: where LLMs and GenAI help
| Use | Where | Value |
|---|---|---|
| Content understanding: topics, entities, format, quality and safety signals, summaries | Near-line at ingestion | Better features and content embeddings for every item, especially new ones |
| Multimodal embeddings (text + image + video frames) | Near-line | Cold start and "more like this" that understands what content is actually about |
| Synthetic relevance labels (user interest ↔ item) | Offline | Augments sparse feedback; calibrated against real engagement |
| User interest summaries ("follows home cooking, Italian food, budget recipes") | Near-line | Readable profiles for features, explanations and user controls |
| Explanations ("Because you watched…") | Offline or cheap templating | Trust and transparency |
| Conversational discovery ("something like this but shorter") | Separate surface | Queries the same embedding index with seconds of latency budget |
| Generative retrieval with semantic IDs | Emerging | Items encoded as short token sequences that a sequence model generates directly, instead of ANN |
The LLM never sits in the 150 ms request path. Its outputs become features and embeddings that fast models use.
Deep dive F: feedback loops, integrity and user control
- Feedback loops: models trained on their own recommendations amplify what they already show, especially popular content. Mitigate with exploration traffic, inverse propensity weighting in training, and diversity constraints.
- Integrity: borderline content (clickbait, sensationalism, misinformation) is down-ranked using classifier scores from the content pipeline, and violating content is removed upstream. See the content moderation design.
- User control: "not interested" and "show less of this" must take effect immediately (a negative signal in the session and a filter), not after tomorrow's retrain.
- Privacy: users can view and reset their interest profile; deleting history removes it from features and triggers user-embedding recomputation.
6. Evaluate, safeguard, operate
Offline evaluation
- Candidate generation: recall@K on time-split held-out interactions (train on the past, test on the future) to avoid leakage.
- Ranking: AUC and NDCG per task; calibration curves.
- Off-policy evaluation: estimate how a new policy would have performed on logged data, using propensity scores from the logging policy. It's cheaper than an A/B test but noisy.
Online evaluation
- A/B tests on engagement (CTR, dwell), satisfaction (likes, hides, reports, survey responses), and creator-side distribution.
- Long-term holdouts: a small group kept on an older system for months, because short-term engagement gains can hurt retention.
- Guard metrics: latency, diversity per user, share of fresh content, integrity metrics.
Operations
- Dashboards per stage: candidate recall proxies, feature freshness and missing-feature rates, ranker latency, index size and insert lag.
- Fallbacks: if personalisation fails, serve trending-by-region; if the heavy ranker times out, use the light ranker's order. A feed must never be empty.
- Precompute feeds for inactive users near-line, and cache the first page for a short time to absorb app-open spikes.
Failure modes
| Failure | Mitigation |
|---|---|
| Feature store latency spike | Default values plus local caches; light ranker only |
| Stale ANN index | Insert-lag alerts; fresh-content candidate source as a backstop |
| Training-serving skew | Log served features; skew monitors comparing distributions |
| Bad model push | A/B guardrails, canary, instant rollback |
| Event pipeline lag | Session features degrade gracefully; batch features still serve |
7. How the design evolves
- v1: popularity by region plus item-to-item collaborative filtering; a simple ranker.
- v2: two-tower retrieval with ANN, a multi-task heavy ranker, a feature store and streaming features.
- v3: multimodal and LLM content understanding for cold start, sequence-based user towers, off-policy evaluation, long-term holdouts, and conversational discovery.
8. Bottlenecks and follow-ups
| Likely question | Answer sketch |
|---|---|
| "Catalogue grows to 1B items?" | Shard the ANN index or use IVF-PQ, route by region or category, and add more candidate sources |
| "Real-time interests (just watched X)?" | Near-line user embedding updates from the event stream, a sequence-model user tower, and item-to-item candidates from recent views |
| "Filter bubbles?" | Diversity re-ranking, exploration, topic spread per user as a metric |
| "Ranking cost too high?" | A light ranker stage, feature caching, distilling the heavy ranker, GPU batching |
| "Why not use an LLM to recommend directly?" | Latency and cost at 70K QPS; use LLMs to understand content and generate features, and fast models to rank |
| "Engagement up, retention down?" | Re-weight the value model toward satisfaction signals; trust long-term holdouts over short-term A/B wins |
Key takeaways
- Recommendations are a funnel. ANN over item embeddings retrieves about 1,000 candidates, rankers score them, and a re-ranker applies diversity, freshness and business rules.
- Two-tower models embed users and items in one space, so item vectors can be precomputed and indexed; the 5 GB index is replicated, not sharded.
- Real-time signals and training-serving consistency (one feature definition for both) matter as much as model architecture.
- Cold start for new items uses content embeddings from text, image or LLM encoders, plus exploration traffic.
- Evaluate offline with time-split recall and NDCG, online with A/B tests and long-term holdouts, and watch for feedback loops and popularity bias.
Go deeper
Finished reading? Mark it done to track your progress.