# What is LLM-as-a-judge?

> LLM-as-a-judge uses one language model to grade another model's output against a rubric. Formats, known biases, and how to check a judge.

Source: https://www.honeyhive.ai/learn/llm-as-a-judge (updated 2026-11-07)

LLM-as-a-judge is an evaluation method in which one language model grades the output of another against criteria you write. The [MT-Bench paper](https://arxiv.org/abs/2306.05685) (Zheng et al., 2023) named the approach and studied it systematically, arguing that strong models such as GPT-4 can stand in for human raters of open-ended chat answers. A [survey of the field](https://arxiv.org/abs/2411.15594) describes judges as a scalable, lower-cost and consistent alternative to expert-driven evaluation, with the caveat that their reliability needs careful design.

Judges fit qualities that code cannot check, such as faithfulness to a source, tone or relevance. For the wider picture of graders, see [LLM evaluation](/learn/llm-evaluation).

## Three ways to run a judge

The [MT-Bench](/learn/mt-bench) paper describes [three formats](https://arxiv.org/html/2306.05685):

| Format | What the judge does | Trade-off the paper notes |
|---|---|---|
| Pairwise comparison | Gets a question and two answers, picks the better one or declares a tie | Scales poorly, because the number of pairs grows quadratically |
| Single answer grading | Assigns a score to one answer | More scalable, but may miss subtle differences between answers, and scores can shift when the judge model changes |
| Reference-guided grading | Gets a reference solution to compare with | The paper says a reference may help where one exists |

[G-Eval](/learn/g-eval) ([Liu et al., 2023](https://arxiv.org/abs/2303.16634)) is a single-answer variant. It needs no human-written reference text, and has the judge reason step by step and then fill in a form. With GPT-4 it [reached](https://arxiv.org/abs/2303.16634) a Spearman correlation of 0.514 with human judgments on summarization.

## Known biases

The MT-Bench paper [reports](https://arxiv.org/html/2306.05685) three biases, with these figures (to build one step by step, see [how to build an LLM judge evaluator](/learn/build-llm-judge-evaluator); more limits are in [limits and biases of LLM-as-a-judge](/learn/llm-judge-limitations), and when to use a person instead is in [LLM judge vs human evaluation](/learn/llm-judge-vs-human-evaluation)):

- Position bias. Most judges favored the first position, and only GPT-4 gave consistent results in more than 60% of cases.
- Verbosity bias. A "repetitive list" attack that padded answers fooled Claude-v1 and GPT-3.5 in 91.3% of cases and GPT-4 in 8.7%.
- Self-enhancement bias. GPT-4 won about 10% more often on its own answers and Claude-v1 about 25% more, though the authors say their data could not confirm the bias.

A separate [study](https://arxiv.org/abs/2305.17926) found that changing the order of two responses can change the ranking. In one example, Vicuna-13B beat ChatGPT on 66 of 80 queries when ChatGPT was the judge. The [G-Eval](/learn/g-eval) authors also [flag](https://arxiv.org/abs/2303.16634) that LLM evaluators may favor text written by LLMs.

## Reducing the bias

- Swap positions. The MT-Bench authors [suggest](https://arxiv.org/html/2306.05685) judging twice with the answer order reversed and counting a win only if the same answer is preferred both times, otherwise a tie.
- Use a reference. Giving the judge a reference answer [cut](https://arxiv.org/html/2306.05685) the failure rate on math questions from 70% (14 of 20) to 15% (3 of 20).
- Add examples, at a cost. Few-shot examples [raised](https://arxiv.org/html/2306.05685) GPT-4's consistency from 65.0% to 77.5%, and made prompts about four times more expensive.
- Collect evidence and aggregate. The calibration method in the [fair evaluators study](https://arxiv.org/abs/2305.17926) has the judge write several pieces of evidence before rating, averages over response orders, and sends hard examples to a person.

## Writing a judge you can trust

Anthropic's [guide to agent evals](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) (January 2026) lists model graders as flexible and able to handle open-ended output, but non-deterministic, more expensive than code, and in need of calibration with human graders. It adds that vague rubrics produce inconsistent judgments. Its practical advice:

- Give the judge a way out, such as returning "Unknown" when it lacks enough information, [to avoid hallucinated grades](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents).
- [Grade each dimension](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) with its own judge instead of one judge for everything.

Hamel Husain's [evals FAQ](https://hamel.dev/blog/posts/evals-faq/) adds a validation routine:

1. Check first whether a code assertion can test the condition, since that needs no judge.
2. Write one judge per failure mode, and prefer pass or fail over a numeric scale.
3. Label examples by hand and split them into train, development and test sets. The FAQ [suggests](https://hamel.dev/blog/posts/evals-faq/) 100 to 200 labeled examples per judge failure mode, with 30 to 50 passes and 30 to 50 fails in each of the development and test sets where possible. Tune on the development set, and use the test set once at the end.
4. Report the judge's true positive rate (share of real failures it catches) and true negative rate (share of good outputs it passes). A test score much worse than the development score suggests overfitting.
5. Start with a strong model while writing the prompt, then try cheaper models and measure how much accuracy you lose.

Human labels are the reference, so check that your labelers agree with each other first ([inter-annotator agreement](/learn/inter-annotator-agreement)). Finding which failures deserve a judge is the job of [error analysis](/learn/error-analysis-llm).

## How to set up a judge in HoneyHive

HoneyHive's [LLM evaluator docs](https://docs.honeyhive.ai/v2/evaluators/llm) describe the steps:

1. Add your model provider under Settings, Workspace, AI Providers.
2. In the Evaluators tab, choose Add Evaluator, then LLM Evaluator.
3. Write the prompt, referencing event data with template variables such as `{{ inputs.question }}` and `{{ outputs.content }}`, then pick the provider, model and return type (boolean, numeric or string).
4. Set event filters and a sampling rate, which the docs say [defaults to 10%](https://docs.honeyhive.ai/v2/evaluators/llm), and create the evaluator.

## FAQ

### How accurate is LLM-as-a-judge?

In the MT-Bench study, strong judges such as GPT-4 reached [over 80% agreement](https://arxiv.org/abs/2306.05685) with human preferences, the same level at which humans agree with each other. That result is for chat answers, so measure your own judge against human labels before trusting it on your task.

### What biases does an LLM judge have?

The [MT-Bench paper](https://arxiv.org/abs/2306.05685) studies position bias (favoring an answer because of where it appears), verbosity bias (favoring longer answers) and self-enhancement bias (favoring its own outputs). Separate work found that [changing answer order](https://arxiv.org/abs/2305.17926) can flip a ranking.

### Should an LLM judge give a score from 1 to 5 or pass or fail?

Hamel Husain's [evals FAQ](https://hamel.dev/blog/posts/evals-faq/) recommends pass or fail, because adjacent scale points are subjective and annotators drift to middle values. Binary labels also make it simple to compute true positive and true negative rates against human labels.