Skip to content

Retrieval and RAG·Lesson 56 of 67·expert

Re-ranking and context assembly

Retrieve widely, then choose carefully — cross-encoders, maximal marginal relevance, de-duplication, and the context budget that decides what the model actually sees.


In this lesson · 20 min

  • →Explain why a second ranking stage beats a better first stage
  • →Contrast bi-encoders and cross-encoders
  • →De-duplicate near-identical chunks before they reach the prompt
  • →Select for diversity with maximal marginal relevance
  • →Assemble context within a budget, with citations and deliberate ordering

Retrieval and selection are different jobs. Retrieval should cast a wide net — get the answer somewhere in the top fifty. Selection then decides which five reach the model. Doing both with one ranking function does neither well.

Two stages

PythonPython not started

Bi-encoder and cross-encoder

The distinction is exactly where the query and the document meet:

PythonPython not started

That ratio is the whole argument. A cross-encoder sees the query and the passage jointly, so it can notice that “the overlap should be 10-20%” answers “how much overlap?” in a way no independent pair of vectors can. It just cannot look at a million things.

Duplicates

Overlapping chunks and repeated documents mean your top five are often the same passage five times:

PythonPython not started

An exact hash catches identical text; a similarity threshold catches the near-identical, which is what overlapping chunks actually produce.

Diversity: maximal marginal relevance

Even after de-duplication, the top results often all say the same thing. MMR trades a little relevance for coverage, picking each next result for what it adds:

PythonPython not started

lambda_ = 1.0 is plain relevance ranking; lowering it buys coverage. For a question with one answer you want a high lambda; for “what are the options?” you want a lower one.

Assembling the context

The retrieved passages are not the prompt. What you build from them is:

PythonPython not started

Four decisions are visible there, and each is a real one:

  • A score floor. A passage at 0.31 is noise, and noise in the context makes answers worse, not merely longer.
  • A token budget. Context is finite and every token is paid for on every turn.
  • Skip, do not stop. A long low-ranked passage should not block three short better ones.
  • Tags carrying the id. That is what makes a citation checkable in the next lesson.

Order matters

Models attend unevenly across a long context — the middle gets the least attention. If you have ten passages, the best ones should not be in positions five and six:

PythonPython not started

The full selection stage

PythonPython not started

Six candidates become three passages: one duplicate removed, one below the floor, one that did not fit the budget.

Try it yourself

Exercise 1

Write mmr(query_vector, vectors, k, lambda_): pick k positions, the first by pure relevance and each later one maximising lambda_ * relevance - (1 - lambda_) * max_similarity_to_already_chosen.

Your turnPython not started
Exercise 2

Write assemble(passages, budget, minimum): drop anything below minimum, take the rest best first while they fit the token budget, and return (chosen, tokens_used). Skip an oversized passage rather than stopping.

Your turnPython not started

Check yourself

Why retrieve 50 candidates and re-rank to 5, rather than just retrieving 5 with a better embedding model?

Check yourself

A passage is too long for the remaining budget. What should the assembler do?

Recap

  • Retrieve widely for recall, then select narrowly for precision. One ranking function cannot do both jobs.
  • A bi-encoder embeds query and document separately, so documents are precomputed. A cross-encoder reads them together: much better, much slower, tens of candidates only.
  • De-duplicate before assembly — overlapping chunks produce near-identical passages.
  • MMR trades relevance for coverage; lambda_ = 1.0 is plain ranking.
  • Assemble with a score floor and a token budget, skipping rather than stopping, and tag each passage with its chunk id so citations are checkable.
  • Log the assembled context. Most “hallucination” reports are really “the answer was not in the context”.