Retrieval and RAG·Lesson 57 of 67·expert
A complete RAG pipeline
Ingest, chunk, embed, index, retrieve, re-rank, assemble, answer, cite — every stage from this module wired together, with the contract that makes an answer checkable.
In this lesson · 24 min
- →Wire the ingest and query paths into one working system
- →Write an answer contract the model can be held to
- →Validate that every citation refers to a chunk that was actually supplied
- →Refuse to answer when retrieval found nothing good enough
- →Trace a query end to end so a bad answer can be diagnosed
Every piece exists now. This lesson assembles them and adds the part that makes the output trustworthy: a contract, and a check that the contract was kept.
The two paths
Ingest
Query
The third query returns nothing, which is exactly right — and it is the case most pipelines get wrong, because retrieval happily returns k results however irrelevant.
The answer contract
The prompt is where the system’s honesty is decided:
Rule 3 is the one that matters. Without an explicit refusal phrasing, a model asked a question its context cannot answer will produce a plausible answer from training — which is the failure everyone calls hallucination.
Validating the answer
Never trust the citations. Check them:
A citation pointing at c9 when only c1 and c2 were supplied is a fabrication you can detect
mechanically, with no judge and no second model call.
The whole thing
Three questions, three different outcomes — answered, answered, refused — and the trace explains each one without guessing.
Failure modes
Try it yourself
Write check_citations(answer, supplied_ids): return a dict with cited, unsupported, refused and ok. An answer is not ok if it cites nothing, or cites an id that was not supplied.
Write select_context(candidates, minimum, budget): drop anything below the score floor, fit the rest into the token budget best first, and return (passages, used, outcome) where outcome is answer or refuse.
Check yourself
A user asks something your corpus does not cover. What should a well-built RAG system do?
Check yourself
Why check citations mechanically rather than relying on the prompt's instruction to cite?
Recap
- Two paths: ingest offline (load, chunk, embed, index) and query online (embed, retrieve, re-rank, assemble, generate, validate).
- The answer contract belongs in the system prompt: answer only from the context, cite every claim, and refuse in a fixed phrase when the context does not contain the answer.
- Validate citations against the ids you actually supplied — a citation to an unsupplied chunk is mechanically detectable fabrication.
- A similarity floor is what turns “always returns k things” into “found nothing, and said so”.
- Trace every query: retrieved, selected, cited, unsupported, outcome. Most bad answers are retrieval problems in disguise.