Retrieval and RAG·Lesson 56 of 67·expert
Re-ranking and context assembly
Retrieve widely, then choose carefully — cross-encoders, maximal marginal relevance, de-duplication, and the context budget that decides what the model actually sees.
In this lesson · 20 min
- →Explain why a second ranking stage beats a better first stage
- →Contrast bi-encoders and cross-encoders
- →De-duplicate near-identical chunks before they reach the prompt
- →Select for diversity with maximal marginal relevance
- →Assemble context within a budget, with citations and deliberate ordering
Retrieval and selection are different jobs. Retrieval should cast a wide net — get the answer somewhere in the top fifty. Selection then decides which five reach the model. Doing both with one ranking function does neither well.
Two stages
Bi-encoder and cross-encoder
The distinction is exactly where the query and the document meet:
That ratio is the whole argument. A cross-encoder sees the query and the passage jointly, so it can notice that “the overlap should be 10-20%” answers “how much overlap?” in a way no independent pair of vectors can. It just cannot look at a million things.
Duplicates
Overlapping chunks and repeated documents mean your top five are often the same passage five times:
An exact hash catches identical text; a similarity threshold catches the near-identical, which is what overlapping chunks actually produce.
Diversity: maximal marginal relevance
Even after de-duplication, the top results often all say the same thing. MMR trades a little relevance for coverage, picking each next result for what it adds:
lambda_ = 1.0 is plain relevance ranking; lowering it buys coverage. For a question with one
answer you want a high lambda; for “what are the options?” you want a lower one.
Assembling the context
The retrieved passages are not the prompt. What you build from them is:
Four decisions are visible there, and each is a real one:
- A score floor. A passage at 0.31 is noise, and noise in the context makes answers worse, not merely longer.
- A token budget. Context is finite and every token is paid for on every turn.
- Skip, do not stop. A long low-ranked passage should not block three short better ones.
- Tags carrying the id. That is what makes a citation checkable in the next lesson.
Order matters
Models attend unevenly across a long context — the middle gets the least attention. If you have ten passages, the best ones should not be in positions five and six:
The full selection stage
Six candidates become three passages: one duplicate removed, one below the floor, one that did not fit the budget.
Try it yourself
Write mmr(query_vector, vectors, k, lambda_): pick k positions, the first by pure relevance and each later one maximising lambda_ * relevance - (1 - lambda_) * max_similarity_to_already_chosen.
Write assemble(passages, budget, minimum): drop anything below minimum, take the rest best first while they fit the token budget, and return (chosen, tokens_used). Skip an oversized passage rather than stopping.
Check yourself
Why retrieve 50 candidates and re-rank to 5, rather than just retrieving 5 with a better embedding model?
Check yourself
A passage is too long for the remaining budget. What should the assembler do?
Recap
- Retrieve widely for recall, then select narrowly for precision. One ranking function cannot do both jobs.
- A bi-encoder embeds query and document separately, so documents are precomputed. A cross-encoder reads them together: much better, much slower, tens of candidates only.
- De-duplicate before assembly — overlapping chunks produce near-identical passages.
- MMR trades relevance for coverage;
lambda_ = 1.0is plain ranking. - Assemble with a score floor and a token budget, skipping rather than stopping, and tag each passage with its chunk id so citations are checkable.
- Log the assembled context. Most “hallucination” reports are really “the answer was not in the context”.