Production AI Systems·Lesson 63 of 67·expert
LLM-as-judge, and its failure modes
A model can grade what no assertion can reach — and it has biases towards length, towards itself, and towards agreeing with whatever it just read. Calibrate it against humans before you trust a number it produced.
In this lesson · 22 min
- →Decide when a judge is warranted and when a mechanical check is enough
- →Write a judge prompt that returns a stable label
- →Measure agreement between a judge and human labels
- →Name the known biases and design around them
- →Use pairwise comparison when absolute scores will not hold still
Some questions have no assertion: is this answer actually supported by the passage, is it helpful, is it better than the previous version. A model can answer those. It can also be confidently wrong in patterned ways, and the patterns are known.
Only when mechanics cannot
A judge prompt
Ask for a label from a fixed set, not a score out of ten:
Three properties make that prompt work: a fixed label set, one claim against one passage, and an explicit instruction to ignore outside knowledge — without which a judge marks a true statement “supported” even when the passage never mentions it.
Decomposing the answer
Judge one claim at a time. A verdict on a whole answer hides which sentence was the problem:
Two of three claims are supported and one contradicts its own citation — a specific, fixable finding. “This answer scored 0.67” would not have been.
Calibrate against humans
A judge’s number means nothing until you know how often it agrees with a person:
The biases
Running both orders doubles the cost and converts a confident wrong answer into an honest tie. That is a good trade.
Pairwise beats absolute
When absolute scores drift between runs, ask which of two is better instead:
Try it yourself
Write agreement(pairs) returning raw agreement and Cohen's kappa for a list of (human, judge) label pairs.
Write resolve(forward, backward, a, b): given the judge's pick shown in each order, return the winner when the two agree and None when position bias made them contradict.
Check yourself
Your judge agrees with human labels on 62% of a three-label task. What should you do?
Check yourself
Why run a pairwise comparison in both orders?
Recap
- Use mechanical checks for everything they can reach. A judge is for reading comprehension and preference, and it costs money, latency and a second model to manage.
- Ask for a label from a small fixed set, one claim against one passage, with an explicit instruction to ignore outside knowledge.
- Calibrate against human labels and report kappa, not raw agreement — raw agreement flatters a judge on imbalanced data.
- Design around the known biases: length, self-preference, position, sycophancy, authority, anchoring.
- Prefer pairwise comparison to absolute scores, run both orders, and count contradictions as ties.