Designing Semantic Product Search

Product search is the system where GenAI enthusiasm most often meets a hard budget. A shopper who types "waterproof hiking boots for wide feet under $150" expects results before they finish blinking, thousands of times a second. The interesting design question is not "how do we put an LLM in search" but where LLMs create value without sitting in the hot path. The answer: mostly offline, making the index, the embeddings and the ranker smarter for every query.

1. Clarify the task and the quality bar

Questions to ask:

• Catalogue and traffic? Assume 50M products (a marketplace with third-party sellers), 5K QPS at peak.
• Latency budget? p99 under 200 ms for the results page, end to end on the server.
• Query styles? Short keywords ("nike air"), attribute-heavy queries ("red dress size 12"), and natural language ("gift for a dad who likes camping").
• Languages and markets? Assume several markets with local catalogues and languages.
• Business constraints? Sponsored products, merchandising rules, and marketplace fairness between sellers.
• What is "good"? Relevant products high in the list, measured by conversion and revenue per search, with a relevance floor so revenue optimisation can't show irrelevant items.

Functional requirements

• Keyword and natural-language search over 50M products
• Filters and facets (brand, price, size, colour, in stock), sort options, spelling correction
• Personalisation: preferred brands and sizes, past purchases
• Sponsored placements blended with organic results under relevance rules

Non-functional requirements

• 5K QPS peak, p99 under 200 ms
• Inventory and price changes reflected within about a minute; new products searchable within minutes
• Zero-result rate as low as possible; graceful behaviour on nonsense queries
• Revenue-critical: every change goes through an A/B test

2. Choose the operating point

Latency and cost are fixed constraints; quality is the objective within them. At 5K QPS, even a $0.0005 LLM call per query is $2.50 a second, ≈ $6.5M a month, and adds hundreds of milliseconds. So LLMs belong offline (enriching data, labelling, precomputing) and on the long tail of rare queries, through a small, fast model.

3. Estimates

Query frequency is heavy-tailed: the top ~10% of distinct queries make up most of the traffic, and a large share of distinct queries are seen only once a day. Result caching and precomputation handle the head; the tail is where query understanding earns its keep.

4. Architecture

Online

Offline

Components

5. Deep dives

Deep dive A: where LLMs fit

Enrichment in practice. Seller data is messy: "NEW!!! Mens Hiking Boot Waterproof Leather BEST DEAL size 8-13". An LLM with a structured-output schema (brand, material, waterproof: bool, width_options, activity) turns it into clean fields that power filters and facets. Validate outputs against the category's schema, spot-check samples by category, and keep the seller's original text for BM25. See Function calling and structured outputs.

Deep dive B: query understanding

A cascade, cheapest first:

1. Normalise: lowercase, Unicode folding, unit normalisation ("10in" → "10 inch").
2. Cache lookup of precomputed understanding for known queries.
3. Spelling correction trained on the catalogue vocabulary (brand names aren't in dictionaries).
4. Category prediction with a small classifier: "apple" in electronics vs groceries changes everything.
5. Attribute and filter extraction: "under $150" → price < 150, "wide" → width = wide. Only apply a filter automatically when confident; otherwise treat it as a soft ranking signal to avoid zero results.
6. Long tail: for unseen, complex queries, a small LLM parses the query into the same structure in ≈ 30–60 ms, and the result is cached for next time.

Deep dive C: retrieval and embeddings

• Hybrid retrieval. BM25 wins on exact brand names, model numbers and sizes; dense vectors win on descriptive and natural-language queries ("cosy jumper for winter"). Retrieve from both in parallel and fuse (reciprocal rank fusion or a learned blend). See Hybrid search and reranking.
• Fine-tuned two-tower encoders. General-purpose embeddings don't know that "trainers" and "sneakers" are the same thing in your catalogue, or that shoppers searching "iPhone case" buy cases, not phones. Train a query encoder and a product encoder contrastively on (query, purchased product) pairs, with hard negatives (products that were shown and skipped). This is usually the biggest relevance gain after hybrid search.
• Multimodal. Product images carry information the title doesn't (style, colour, pattern). A CLIP-style image embedding index supports "floral summer dress" and visual search ("find similar").
• Index choice. At 50M items, int8 HNSW fits on one node per replica, so replicate rather than shard. At 1B items, move to IVF-PQ or disk-based indexes and route by category. See Sizing and sharding vector indexes.

Deep dive D: ranking

Retrieval finds ≈ 500 candidates; ranking orders them.

• Features: text relevance scores (BM25, vector similarity), query–product attribute matches, popularity and conversion rate, price, ratings and reviews, seller quality, freshness, and personal features (brand and size affinity).
• Model: a learning-to-rank model (LambdaMART-style gradient-boosted trees, or a small neural network) trained on clicks, add-to-carts and purchases, with position-bias correction (items shown higher get clicked more regardless of relevance).
• Optional cross-encoder on the top 20–50 for tail queries, where the budget allows (≈ 20–40 ms on a GPU).
• Business layer: sponsored products blended under a relevance floor, merchandising boosts, deduplication of near-identical listings, and brand and seller diversity.

Deep dive E: latency budget (p99 200 ms)

Each stage has a timeout and a fallback: if the long-tail parser is slow, search without it; if the image index times out, fuse the other two lists.

Deep dive F: filters and freshness

• Filtering inside retrieval. "In stock in the user's region" can eliminate 90% of the catalogue for some queries. Use in-index filtering with a planner that switches to brute force over small filtered sets, rather than post-filtering the top 500.
• Inventory and price updates stream from the inventory service (change data capture) into index metadata within seconds. Vectors change only when product content changes.
• New products are enriched and embedded on creation, and get a freshness boost with a little exploration traffic, so they can collect clicks despite having no history (the cold-start problem).
• Cache invalidation: cached result pages for head queries expire quickly (minutes), and entries are dropped when a top result goes out of stock.

6. Evaluate, safeguard, operate

Offline evaluation

• A judged query set sampled from head, torso and tail, with graded labels (exact, substitute, complement, irrelevant) from human raters. LLM-assisted labelling scales this to tens of thousands of queries once it agrees with raters on a calibration set.
• Metrics: NDCG@10 for ranking, recall@500 for the candidate stage, zero-result rate.

Online evaluation

• A/B tests on conversion, revenue per search, add-to-cart, CTR, zero-result and reformulation rates.
• Interleaving (mixing two rankers' results in one list) detects ranking differences with far less traffic than A/B tests.
• Guard metrics: p99 latency, diversity, sponsored-share, and seller fairness.

Safeguards

• Enrichment can hallucinate attributes ("waterproof" when it isn't), which becomes a customer-trust and returns problem. Validate against seller-provided fields, sample by category, and let sellers correct.
• Queries for restricted items (weapons, drugs) follow policy per market.
• Protect against adversarial sellers stuffing keywords, which LLM-cleaned titles help with.

Failure modes

7. How the design evolves

• v1: BM25 with synonyms, a GBDT ranker, filters and facets.
• v2: LLM catalogue enrichment, dense retrieval with fine-tuned encoders, precomputed query understanding.
• v3: multimodal retrieval, long-tail query parsing, personalisation, and a conversational shopping assistant that uses search as a tool.

8. Bottlenecks and follow-ups