Retrieval and RAG·Lesson 55 of 67·expert
Hybrid search: dense plus keyword
Embeddings miss the exact error code, the product name and the rare identifier. BM25 finds them and misses the paraphrase. Run both and fuse the rankings.
In this lesson · 20 min
- →Name the queries dense retrieval systematically fails
- →Implement BM25 and explain each of its terms
- →Fuse two rankings with reciprocal rank fusion
- →Fuse scores directly, and know why that is harder
- →Decide when hybrid search is worth its complexity
Dense retrieval is good at meaning and bad at literals. Keyword search is the reverse. Production retrieval runs both, because real queries contain both.
What dense retrieval misses
Keyword search handles every one of those, because a rare token is exactly what an inverted index is good at — and fails at the thing dense retrieval does best: finding the passage that answers the question in different words.
BM25
BM25 scores a document by how often the query’s terms appear in it, weighted by how rare each term is across the corpus, and adjusted so long documents do not win by length alone.
The three parts are worth naming:
- IDF — a term appearing in one document out of a million is worth far more than “the”.
k1saturates term frequency: the tenth occurrence of a word adds much less than the second.bnormalises by length, so a long document does not score highly merely for being long.
Where each one wins
The first query is a bare identifier: BM25 finds it because E4021 is rare, and the dense side has
almost nothing to work with. The second is a paraphrase sharing barely a word with its answer —
where dense retrieval is supposed to shine, and where a lexical method struggles.
Reciprocal rank fusion
The problem with combining two retrievers is that their scores are not comparable — a cosine similarity of 0.82 and a BM25 score of 7.3 mean nothing to each other. RRF sidesteps that entirely by using only the ranks:
c7 wins: third for the dense retriever, first for the keyword one, and the only document both
rate highly. That is exactly the behaviour you want — agreement between two different methods is
strong evidence.
The constant k controls how much a top rank dominates:
Weighted score fusion
If you do want to combine scores, normalise each retriever’s scores into a comparable range first:
Try it yourself
Write rrf(rankings, k=60): score each id by the sum of 1/(k + rank) across every ranking it appears in, and return (id, score) pairs best first.
Write idf(term, documents) using the BM25 formula, then rarest(query, documents) returning the query term with the highest IDF.
Check yourself
A user pastes 'error E4021' and dense retrieval returns passages about errors in general. Why?
Check yourself
Why is reciprocal rank fusion usually preferred over combining two retrievers' scores directly?
Recap
- Dense retrieval fails systematically on identifiers, rare names, versions, numbers and negation. Keyword search fails on paraphrase. Real queries contain both.
- BM25 weights terms by rarity (IDF), saturates repeated terms (
k1) and normalises for document length (b). - Fuse with reciprocal rank fusion: sum
1/(k + rank)across rankings,k = 60. It needs no comparable scales and no tuning. - Weighted score fusion works but needs per-query normalisation, which is query-dependent and has no sensible floor.
- Hybrid costs one extra index and about ten lines, and removes a whole class of failure.