LLM-as-Judge
Using a model to grade another model's output — cheap, scalable, correlates decently with humans, and it prefers its own writing.
When not to use it
- On subtle factual accuracy. The judge can only check against what it believes, and that's the thing under test.
- Without swapping order. Position bias can flip the winner. This is the cheapest fix in evaluation.
- Judging its own family's output. Self-preference is documented and tied to self-recognition.
- As ground truth. It's an instrument with known systematic bias. Use it for comparison, not for claims.
Reach for something else instead
- Executable verification — tests that pass. The only evaluation that doesn't need trusting.
- Human evaluation — expensive, noisy, and the thing the judge is approximating.
- Reference-guided judging — much more reliable, and needs the gold answers.
- Task-specific metrics — narrow, checkable, boring, and they work.
Sources & further reading
- Zheng et al. (2023), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — the method and its biases, documented together.
- Wang et al. (2023), Large Language Models are not Fair Evaluators — position bias, quantified; swapping is not optional.
- Panickssery, Bowman & Feng (2024), LLM Evaluators Recognize and Favor Their Own Generations — self-preference tied to self-recognition. Systematic, not noise.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Not swapping the order. Position bias is large and the fix is free.
- Using the same model family to judge itself, then reporting the score as neutral.
- Ignoring verbosity bias, then wondering why your product got wordier over six months of optimisation.
- Treating agreement-with-humans as validation. The model may have learned humans' shared biases, which produces the same number for a worse reason.
- Using a panel of judges and assuming errors average out. They're correlated.
At a glance
FieldLanguage & LLMs
Why it's usedhuman evaluation doesn't scale
Three documented biasesposition, verbosity, self-preference
The mandatory fixswap the order and average
The bindcircular; a model grading reasoning with the faculties under test
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
LLM-as-judge vs. executable evaluation — one asks a model whether the output is good; the other runs it. Only the second brings information from outside the model.