LLM-as-Judge
Using a model to grade another model's output — cheap, scalable, correlates decently with humans, and it prefers its own writing.
Reviewed July 12, 2026Stable
When not to use it
- On subtle factual accuracy. The judge can only check against what it believes, and that's the thing under test.
- Without swapping order. Position bias can flip the winner. This is the cheapest fix in evaluation.
- Judging its own family's output. Self-preference is documented and tied to self-recognition.
- As ground truth. It's an instrument with known systematic bias. Use it for comparison, not for claims.
Reach for something else instead
- Executable verification — tests that pass. The only evaluation that doesn't need trusting.
- Human evaluation — expensive, noisy, and the thing the judge is approximating.
- Reference-guided judging — much more reliable, and needs the gold answers.
- Task-specific metrics — narrow, checkable, boring, and they work.
Read more on the blog
- LLM-as-a-judge: when a model can grade another modelGPT-4 agreed with human evaluators over 80% of the time, which is the rate humans agree with each other. A 2026 study found frontier models exceeding 50% error on hard bias benchmarks. Both numbers are real.
- Testing a system that answers differently every timeForty percent of organisations hit significant quality regressions within ninety days of deploying an LLM application. Exact-match assertions are the cause: they reject valid answers and occasionally accept wrong ones.
- How to check an AI claim before you believe itA vendor advertised a hallucination rate below 0.001% with no evidence behind it, and a state attorney general investigated. Six questions separate a claim you can act on from a number someone typed.
- Your agent returned 200 OK and did the wrong thingOrdinary software tells you when it breaks. AI systems return well-formed, confident, wrong answers with a success status. That one property is why observability for AI is a different discipline rather than the old one pointed at new infrastructure.
Further reading
- Zheng et al. (2023), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — the method and its biases, documented together.
- Wang et al. (2023), Large Language Models are not Fair Evaluators — position bias, quantified; swapping is not optional.
- Panickssery, Bowman & Feng (2024), LLM Evaluators Recognize and Favor Their Own Generations — self-preference tied to self-recognition. Systematic, not noise.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Not swapping the order. Position bias is large and the fix is free.
- Using the same model family to judge itself, then reporting the score as neutral.
- Ignoring verbosity bias, then wondering why your product got wordier over six months of optimisation.
- Treating agreement-with-humans as validation. The model may have learned humans' shared biases, which produces the same number for a worse reason.
- Using a panel of judges and assuming errors average out. They're correlated.
At a glance
FieldLanguage & LLMs
Why it's usedhuman evaluation doesn't scale
Three documented biasesposition, verbosity, self-preference
The mandatory fixswap the order and average
The bindcircular; a model grading reasoning with the faculties under test
DifficultyIntermediate
Often compared with
LLM-as-judge vs. executable evaluation — one asks a model whether the output is good; the other runs it. Only the second brings information from outside the model.
Where this sits
A destination. 24 concepts lead here, and nothing in the corpus depends on it.
11Levelsteps in
24Needs firstconcepts
0Opens upnothing further
1Areastays here
Learn these firstLarge Language Model
Computed from the prerequisite graph, not assigned. How this works