Retrieval and RAG·Lesson 58 of 67·expert
Measuring groundedness and retrieval quality
Recall@k, MRR and nDCG for the retriever; groundedness and citation accuracy for the answer — and the ablation that tells you which stage is actually at fault.
In this lesson · 22 min
- →Build a small evaluation set that catches real regressions
- →Compute recall@k, precision@k, MRR and nDCG
- →Measure groundedness and citation accuracy end to end
- →Ablate the pipeline to find which stage is failing
- →Recognise what an LLM judge can and cannot tell you
“It seems better” is not a result. This lesson turns a RAG system into numbers you can compare between two Tuesdays — and, more usefully, numbers that say which stage to fix.
The evaluation set
Retrieval metrics
The three answer different questions, and confusing them is common:
- Recall@k — did retrieval find the answer at all? This is the ceiling on everything after it.
- Precision@k — how much of what you retrieved was worth sending? This is your context budget.
- MRR — how near the top was the first good one? This is what re-ranking improves.
nDCG
When some results are more relevant than others, a binary label is too coarse. nDCG weights by graded relevance and discounts by position:
The discount is logarithmic, so moving a result from position 1 to 2 costs far more than moving it from 9 to 10 — which matches how people read a list.
End-to-end metrics
Retrieval quality is a means. What you ship is the answer:
Four numbers are worth reporting from that, and they are not interchangeable:
Ablation
When end-to-end quality drops, the useful question is where. Ablate:
LLM as judge
Groundedness — “is this claim actually supported by the cited passage?” — needs a reader. A model can be that reader, with caveats:
Module 8 takes this further, including how to calibrate a judge against human labels. For now the important part is the ordering: the mechanical checks come first. Citation support costs nothing and catches the worst failures; reach for a judge only for what mechanics cannot see.
Try it yourself
Write recall_at_k and mrr. Recall returns None when nothing is relevant; MRR averages the reciprocal rank across a list of (retrieved, relevant) pairs.
Write ndcg_at_k(retrieved, graded, k): discounted cumulative gain over the top k, divided by the gain of the ideal ordering. Return 0.0 when nothing is relevant.
Check yourself
Recall@10 is 0.95 but answers are frequently wrong. Which stage should you look at?
Check yourself
Why include deliberately unanswerable questions in a retrieval evaluation set?
Recap
- Build a small, deliberate evaluation set, labelled with the relevant chunk ids, and include questions that should be refused.
- Recall@k is the ceiling, precision@k is your context budget, MRR is what re-ranking moves, and nDCG handles graded relevance with a positional discount.
- End to end, report answer rate, refusal precision and recall, citation support and groundedness — never one blended “accuracy”.
- Ablate by stage: never retrieved, retrieved but not selected, selected but refused, or cited something that was not supplied.
- Do the mechanical checks first; a judge is for what mechanics cannot see, and needs calibrating against human labels before you trust it.