Production AI Systems·Lesson 62 of 67·expert
Building an evaluation harness
A runnable eval set, graders that check properties, results recorded with the prompt and model that produced them, and a comparison that tells you exactly which examples changed.
In this lesson · 22 min
- →Design an eval set that catches real regressions
- →Write graders, from exact match to property checks
- →Run an evaluation concurrently, within a cost budget
- →Record results so two runs can be compared example by example
- →Split the set so prompt tuning does not overfit it
An evaluation harness is the instrument that makes every other decision in this module possible. Without one, “the new prompt is better” is a feeling.
The eval set
JSONL again, for the same reasons as module 2: append a case without rewriting the file, and read a large set without loading it.
Graders
A grader turns an answer and an expectation into a pass or a fail. Compose several:
Each grader answers one question, so a failure says which property broke — which is the difference between a number that drops and a number you can act on.
Running it
One example fails — the system answered a question it should have refused, and cited a chunk to do it. That is precisely the failure an eval set exists to catch.
Record everything
A result without its provenance cannot be compared to anything:
Comparing two runs
The aggregate number tells you whether something changed. The per-example diff tells you what:
A change that fixes three examples and breaks two shows as “+1” on the summary. The regression list is what stops you shipping it without looking.
Do not tune on the test set
The prompt-overfitting warning from module 5, applied here:
A regression gate
Try it yourself
Write grade(answer, expect, context_ids) returning (checks, passed), where checks has boolean refusal, contains, citations and length entries.
Write compare(before, after): given two dicts of id → passed, return improved, regressed and unchanged id lists plus both pass rates. Ignore ids missing from either run.
Check yourself
A prompt change raises the pass rate from 27/30 to 28/30. Is it safe to ship?
Check yourself
Why hash the eval set and record the hash with every run?
Recap
- Keep the eval set as JSONL: inputs plus the properties an acceptable answer must have. Twenty curated examples beat two hundred generated ones.
- Write one grader per property, so a failure names which property broke.
- Run concurrently with a semaphore and report cost alongside the pass rate.
- Record the model, prompt version and hash, retrieval settings and the eval-set hash with every run.
- Compare per example, not just on the aggregate — improved, regressed, unchanged.
- Split into train, dev and test so prompt tuning does not quietly overfit the set.