Because they fail differently: embeddings match paraphrase but miss rare exact strings like error codes and part numbers, while BM25 matches those tokens and misses paraphrase. Hybrid search fuses both rankings, usually with a reranker on top.
Card 1 of 292. Answer: Because they fail differently: embeddings match paraphrase but miss rare exact strings like error codes and part numbers, while BM25 matches those tokens and misses paraphrase. Hybrid search fuses both rankings, usually with a reranker on top.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
Combine embeddings with keyword search because they fail on different queries: embeddings match meaning and paraphrase but blur rare exact strings, while BM25 matches exact tokens but misses synonyms. Hybrid search runs both retrievers and fuses their ranked lists into one.
An embedding model splits "E-4012" into subword pieces and compresses the passage into one vector, so one rare token barely moves it; BM25 weights it heavily because few documents contain it. Reciprocal rank fusion scores each document Σ 1/(k + rank), with k = 60 by convention. If BM25 ranks D1, D2, D3 and embeddings rank D3, D1, D4, D1 scores 1/61 + 1/62 ≈ 0.0325, D3 0.0323, D2 and D4 about 0.016.
Why fuse ranks, not scores? BM25 scores have no fixed scale and vary by query, while cosine similarities sit between −1 and 1, so adding them needs normalization and tuning; RRF (Cormack, Clarke and Büttcher, SIGIR 2009) needs neither. A cross-encoder reranker then rescores the top candidates. Hybrid helps most with IDs, code and jargon, and out of domain, where the BEIR benchmark (2021) found BM25 a robust zero-shot baseline.
