Skip to content

Retrieval and RAG·Lesson 58 of 67·expert

Measuring groundedness and retrieval quality

Recall@k, MRR and nDCG for the retriever; groundedness and citation accuracy for the answer — and the ablation that tells you which stage is actually at fault.


In this lesson · 22 min

  • →Build a small evaluation set that catches real regressions
  • →Compute recall@k, precision@k, MRR and nDCG
  • →Measure groundedness and citation accuracy end to end
  • →Ablate the pipeline to find which stage is failing
  • →Recognise what an LLM judge can and cannot tell you

“It seems better” is not a result. This lesson turns a RAG system into numbers you can compare between two Tuesdays — and, more usefully, numbers that say which stage to fix.

The evaluation set

PythonPython not started

Retrieval metrics

PythonPython not started

The three answer different questions, and confusing them is common:

  • Recall@k — did retrieval find the answer at all? This is the ceiling on everything after it.
  • Precision@k — how much of what you retrieved was worth sending? This is your context budget.
  • MRR — how near the top was the first good one? This is what re-ranking improves.
PythonPython not started

nDCG

When some results are more relevant than others, a binary label is too coarse. nDCG weights by graded relevance and discounts by position:

PythonPython not started

The discount is logarithmic, so moving a result from position 1 to 2 costs far more than moving it from 9 to 10 — which matches how people read a list.

End-to-end metrics

Retrieval quality is a means. What you ship is the answer:

PythonPython not started

Four numbers are worth reporting from that, and they are not interchangeable:

PythonPython not started

Ablation

When end-to-end quality drops, the useful question is where. Ablate:

PythonPython not started

LLM as judge

Groundedness — “is this claim actually supported by the cited passage?” — needs a reader. A model can be that reader, with caveats:

PythonPython not started

Module 8 takes this further, including how to calibrate a judge against human labels. For now the important part is the ordering: the mechanical checks come first. Citation support costs nothing and catches the worst failures; reach for a judge only for what mechanics cannot see.

Try it yourself

Exercise 1

Write recall_at_k and mrr. Recall returns None when nothing is relevant; MRR averages the reciprocal rank across a list of (retrieved, relevant) pairs.

Your turnPython not started
Exercise 2

Write ndcg_at_k(retrieved, graded, k): discounted cumulative gain over the top k, divided by the gain of the ideal ordering. Return 0.0 when nothing is relevant.

Your turnPython not started

Check yourself

Recall@10 is 0.95 but answers are frequently wrong. Which stage should you look at?

Check yourself

Why include deliberately unanswerable questions in a retrieval evaluation set?

Recap

  • Build a small, deliberate evaluation set, labelled with the relevant chunk ids, and include questions that should be refused.
  • Recall@k is the ceiling, precision@k is your context budget, MRR is what re-ranking moves, and nDCG handles graded relevance with a positional discount.
  • End to end, report answer rate, refusal precision and recall, citation support and groundedness — never one blended “accuracy”.
  • Ablate by stage: never retrieved, retrieved but not selected, selected but refused, or cited something that was not supplied.
  • Do the mechanical checks first; a judge is for what mechanics cannot see, and needs calibrating against human labels before you trust it.