Limits and biases of LLM-as-a-judge

An LLM judge is wrong in specific, documented ways: it can accept a wrong answer to a math question, prefer text because of its position, length or authorship, say “correct” when unsure, and assign scores far from a human’s even when its labels mostly agree. Treat its output as a measurement that needs validating, not as ground truth. The limits below each come with the research behind them. For what a judge is and how to run one, see LLM-as-a-judge.

Where a judge is wrong

Reasoning questions. The MT-Bench paper tested GPT-4 on 10 math questions, judged twice with answer positions swapped. The judge failed in 70% of the 20 judgments with the default prompt, 30% with chain-of-thought and 15% with a reference answer. GPT-4 could solve the problems when asked directly, but as a judge it was “misled by the provided answers.” In some cases it copied the flawed answer’s arithmetic. GPT-3.5 and Claude-v1 showed similar weaknesses.

Errors hidden behind a right answer. The CALM study of 12 biases includes fallacy-oversight bias, where the judge overlooks logical errors and focuses on the final answer. A judge with this bias can pass a response whose reasoning is broken but whose ending is right.

Leniency. In the Judging the Judges study, most judge models tended to label an answer correct when unsure. A judge that passes almost everything looks accurate on a set that is mostly good, and misses the failures you built the check to find.

Score distance. The same study found that even the best judges, with agreement scores in the high 80s against a human benchmark of about 96, assigned scores up to 5 points away from human scores. Judges above 90% agreement could still differ by more than 10 points. The authors conclude that percent agreement is not enough on its own.

Biases beyond position and length

The LLM-as-a-judge page gives the MT-Bench figures for position, verbosity and self-enhancement bias. Later work adds more:

Bias	What happens	Source
Position	The judge favors an answer because of where it appears. Across 15 judges and over 150,000 instances, it was not due to chance, varied by judge and task, and was strongly affected by the quality gap between answers	Shi et al.
Self-preference	Judges score their own outputs above others’ that humans rate as equal, and recognize their own text better than chance	Panickssery et al.
Authority	The judge credits claims backed by authority figures or citations even when the evidence is weak or fake	Ye et al.
Distraction	The judge is pulled toward irrelevant details	Ye et al.
Sentiment	Verdicts shift with the emotional tone of the text	Ye et al.
Chain-of-thought	Verdicts change depending on whether the judge reasons step by step first	Ye et al.
Brand naming	Behavior changes when models are named with well-known brands instead of anonymous labels (compassion-fade bias)	Ye et al.

The CALM list also includes bandwagon, diversity and refinement-aware bias, and its authors report that although advanced models perform well overall, “significant biases persist in certain specific tasks.”

The self-preference result has a practical consequence. Panickssery and colleagues found a linear relationship between how well a model recognizes its own output and how strongly it prefers it, so a judge from the same model family as your generator deserves extra suspicion.

What a judge can still do

Ranking is easier than scoring. In the Judging the Judges study, a simple string-matching baseline and a 7B model ranked nine models almost as well as humans did (Spearman correlation of 0.98 to 0.99, against 0.98 for GPT-4 Turbo), even though both agreed with humans less on individual answers. If you only need to know which of two prompts or models is better on a fixed set, a modest judge can be enough. If you need an absolute pass rate to compare with a threshold, the score distance above matters.

The validity question

Chehbouni and colleagues argue, using measurement theory, that adoption has moved faster than scrutiny. They assess four assumptions behind judges: that they stand in for human judgment, are capable evaluators, scale, and are cost-effective. They look at summarization, data annotation and safety alignment, and call for more responsible evaluation practice. A survey reaches a milder version of the same point: judges can be scalable and consistent, but reliability is hard to guarantee and takes careful design.

Two scope limits come from the MT-Bench authors themselves. Their method emphasizes helpfulness and largely neglects safety, and it merges accuracy, relevance and creativity into one score.

Limit, check and fix

Limit	Check	Source for the fix
Wrong on reasoning	Give the judge a reference answer, or test the condition in code	MT-Bench paper
Position bias	Judge twice with the order swapped and count a win only if it holds both times	MT-Bench paper
Leniency	Report the true negative rate on labeled failures, not only overall agreement	Judging the Judges
Scores far from humans	Compare scores, not only labels, against human-labeled examples, and report Scott’s pi alongside percent agreement	Judging the Judges
Self-preference	Use a judge from a different model family than the generator	Panickssery et al.
One merged score	Grade each dimension with its own judge	Anthropic’s agent evals guide

The self-preference row follows from the finding above rather than from a tested fix, so verify it on your own data.

The human labels in these checks need to be trustworthy too. Human evaluation of LLM outputs covers how to collect them, and inter-annotator agreement how to test that reviewers agree. To find which failures are worth a judge at all, start with error analysis.

Tracking a judge in HoneyHive

HoneyHive’s LLM evaluator docs describe running a judge over traced events with a sampling rate, and its client-side evaluators run scoring in your own code, which suits the code checks above.

FAQ

What are the main limitations of LLM-as-a-judge?

Documented limits include bias (the CALM study quantifies 12), self-preference, weak grading of math and reasoning, leniency when unsure, and scores that drift from human scores even when agreement looks high. The table near the end maps each limit to a check.

Does an LLM judge favor its own outputs?

Evidence says yes. Panickssery, Bowman and Feng found that judges score their own outputs higher than other models' outputs that humans rate as equal, and that GPT-4 and Llama 2 can tell their own text from other models' and humans' text better than chance.

Can a small LLM judge replace a large one?

For absolute scores, usually not. Thakur and colleagues found that only the largest judges reached reasonable alignment with humans. For ranking models against each other, even much weaker judges gave a near-human order in the same study.

Is high percent agreement with humans enough to trust a judge?

No. In the same study, judges above 90% agreement could still assign scores that differ by more than 10 points, so check the scores as well as the labels.