Skip to content

Production AI Systems·Lesson 63 of 67·expert

LLM-as-judge, and its failure modes

A model can grade what no assertion can reach — and it has biases towards length, towards itself, and towards agreeing with whatever it just read. Calibrate it against humans before you trust a number it produced.


In this lesson · 22 min

  • →Decide when a judge is warranted and when a mechanical check is enough
  • →Write a judge prompt that returns a stable label
  • →Measure agreement between a judge and human labels
  • →Name the known biases and design around them
  • →Use pairwise comparison when absolute scores will not hold still

Some questions have no assertion: is this answer actually supported by the passage, is it helpful, is it better than the previous version. A model can answer those. It can also be confidently wrong in patterned ways, and the patterns are known.

Only when mechanics cannot

PythonPython not started

A judge prompt

Ask for a label from a fixed set, not a score out of ten:

PythonPython not started

Three properties make that prompt work: a fixed label set, one claim against one passage, and an explicit instruction to ignore outside knowledge — without which a judge marks a true statement “supported” even when the passage never mentions it.

Decomposing the answer

Judge one claim at a time. A verdict on a whole answer hides which sentence was the problem:

PythonPython not started

Two of three claims are supported and one contradicts its own citation — a specific, fixable finding. “This answer scored 0.67” would not have been.

Calibrate against humans

A judge’s number means nothing until you know how often it agrees with a person:

PythonPython not started

The biases

PythonPython not started
PythonPython not started

Running both orders doubles the cost and converts a confident wrong answer into an honest tie. That is a good trade.

Pairwise beats absolute

When absolute scores drift between runs, ask which of two is better instead:

PythonPython not started

Try it yourself

Exercise 1

Write agreement(pairs) returning raw agreement and Cohen's kappa for a list of (human, judge) label pairs.

Your turnPython not started
Exercise 2

Write resolve(forward, backward, a, b): given the judge's pick shown in each order, return the winner when the two agree and None when position bias made them contradict.

Your turnPython not started

Check yourself

Your judge agrees with human labels on 62% of a three-label task. What should you do?

Check yourself

Why run a pairwise comparison in both orders?

Recap

  • Use mechanical checks for everything they can reach. A judge is for reading comprehension and preference, and it costs money, latency and a second model to manage.
  • Ask for a label from a small fixed set, one claim against one passage, with an explicit instruction to ignore outside knowledge.
  • Calibrate against human labels and report kappa, not raw agreement — raw agreement flatters a judge on imbalanced data.
  • Design around the known biases: length, self-preference, position, sycophancy, authority, anchoring.
  • Prefer pairwise comparison to absolute scores, run both orders, and count contradictions as ties.