A search index with a billion documents returns thousands of candidates matching a query. Showing all of them ranked by TF-IDF doesn’t produce great results. The ranking signals needed for good results (machine-learned models, user personalization, freshness, domain authority) are expensive to compute. You can’t compute them for a thousand candidates in under 100ms.

The solution: a funnel. Use cheap signals to eliminate most candidates fast, then apply expensive signals to the small remainder.

The Funnel#

Stage 1, Retrieval: inverted index lookup returns top 1000 documents matching the query terms, scored by basic TF-IDF. Fast, takes 10-20ms.

Stage 2, Lightweight ranking: apply fast signals to the 1000 candidates. PageRank score, document freshness, domain whitelist/blacklist, query-title overlap. Prune to top 100. Fast, another 10ms.

Stage 3, Heavyweight ranking: apply expensive signals to the top 100. Machine-learned ranking model with hundreds of features: user click history, semantic similarity between query and document, entity recognition, document quality signals. Prune to top 10-20 for final display. 50-100ms.

Stage 4, Presentation: apply final filters for diversity (not all results from the same domain), ad injection, direct answers. Return to user.

1B documents
  → Retrieval: 1000 candidates    (TF-IDF, 20ms)
  → Lightweight: 100 candidates   (PageRank + freshness, 10ms)
  → Heavy ranking: 20 candidates  (ML model, 80ms)
  → Presentation: 10 shown        (diversity, <5ms)
Total: ~115ms
graph TD A[1 Billion Documents] --> B[Stage 1: Retrieval - 1000 candidates] B --> C[Stage 2: Lightweight ranking - 100 candidates] C --> D[Stage 3: ML ranking - 20 candidates] D --> E[Stage 4: Diversity and presentation - 10 results] style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff

Learning to Rank#

The heavy ranking stage uses a learned model. Training data: queries paired with documents and a label (“was this document clicked? was it rated relevant?”). Features: TF-IDF scores, PageRank, query-document similarity, click-through rate for this query-document pair historically.

The model learns which combinations of features correlate with relevance. Common algorithms: LambdaMART (a gradient boosting model optimized for ranking metrics), neural ranking models (BERT-based models that understand query-document semantic similarity).

The challenge: training data is biased toward what users saw. If a relevant document was never shown, it never got clicks, so the model doesn’t learn it’s relevant. This is the position bias problem: results shown first get more clicks regardless of relevance.

Offline vs Online Features#

Some features can be precomputed offline: PageRank, document quality scores, entity counts. These are stored in a lookup table and retrieved in milliseconds.

Online features require the query: TF-IDF scores for the query against this document, query-document semantic similarity. These are computed at query time and are the expensive part of the heavy ranking stage.

At Salesforce#

Our report search used a two-stage approach: a text search returning up to 200 reports matching the query, then a second stage that re-ranked based on usage frequency (how often had this report been run in the last 30 days?). Frequently-used reports surfaced above rarely-used ones even with lower text relevance scores. This simple two-stage approach significantly improved the “first result quality” metric for power users who repeatedly ran the same reports.

What I’m Learning#

The funnel architecture is a latency-accuracy trade-off at each stage. Spend compute budget on the candidates most likely to be shown, apply cheap filters to eliminate bulk candidates, save expensive signals for the final round. The architecture scales by adding more compute to the stages that need it independently.

Have you built multi-stage filtering or ranking pipelines, and how did you decide where to put the cutoff between stages?