LLM-as-a-judge vs human evaluation: when to use each

Use code checks where a rule can catch the failure, an LLM judge where the failure needs judgment and you need to cover many outputs, and people for the labels that define quality, the cases that matter most, and checking the judge. A judge is only as trustworthy as the human labels it was checked against.

This page is about choosing between them. For how a judge works, see LLM-as-a-judge. For how to run review, see human evaluation of LLM outputs.

Which one for which job

Hamel Husain’s evals FAQ puts the order plainly: “Use a code-based eval when a deterministic rule can identify the failure” and “Use an LLM judge when the failure requires human judgment.” Anthropic’s evaluation guide takes the same direction with “Automate when possible” and says more questions with automated grading beat fewer, high-quality hand-graded ones.

How closely judges agree with people

The strongest number comes from the MT-Bench paper (Zheng et al., 2023). With ties removed, GPT-4 agreed with human votes 85% of the time (859 comparisons on first-turn questions), while two humans agreed 81% of the time (479 comparisons). When ties and inconsistent votes counted as ties, the figures were 66% for GPT-4 and humans, and 63% for two humans.

Two things limit how far that result travels. It covers chat answers judged in pairs, and the same paper reports weak grading on math and reasoning questions: a failure rate of 14 of 20 that fell to 3 of 20 once the judge got a reference answer.

A different setup gives a less flattering picture. Thakur and colleagues tested 13 judge models in a setting where humans agree closely with each other. Only the best and largest judges reached reasonable alignment, and even they fell well short of inter-human agreement. Judge scores could differ from human scores by up to 5 points, and judges leaned lenient. The authors also found that percent agreement alone can hide this, since a judge with high percent agreement can still assign very different scores. Smaller judges, and even a simple lexical metric, still gave a usable signal for ranking models.

The sources disagree because they measure different things. A judge that ranks systems correctly may still score individual outputs badly, so pick the check that matches your use. Biases behind these gaps are covered in LLM-as-a-judge.

What each costs

Per output, a judge is cheaper and faster than a reviewer, and Anthropic’s guide says more questions with automated grading beat fewer hand-graded ones. The setup cost sits elsewhere. The evals FAQ says judges “require 100+ labeled examples, ongoing weekly maintenance”, and advises building expensive evaluators only “for problems you’ll iterate on repeatedly”. Its default is to start with cheap code-based checks.

Humans cost the most per item, so spend them where they change decisions: defining criteria, labeling the examples a judge is tuned and tested on, and reviewing what the judge is unsure about.

Use both: the usual loop

1. Review real outputs and write down the failures (error analysis).
2. Turn failures a rule can catch into code checks.
3. For the rest, label examples by hand and check that your labelers agree (inter-annotator agreement).
4. Build a judge and test it against those labels. The evals FAQ says to split labeled data into training, development and test sets, keep the test set untouched until the end, and track the true positive rate and true negative rate. It warns that when failures are rare, “even a small false-alarm rate can create a lot of unnecessary reviews”.
5. Run the judge on the volume, and send low-confidence cases and a sample of its verdicts back to people.

Anthropic’s guide, in its code examples, comments that it is generally best practice to grade with a different model from the one that produced the output. For broader context on graders, see LLM evaluation.

Setting up both in HoneyHive

HoneyHive’s evaluator types include Python code, LLM-assisted and human evaluators. Human evaluators are server-side only, and server-side evaluators have higher latency because results are not immediately available. Annotation queues hold the events you want reviewed, and the docs say they can turn judgments into ground truth for evaluators. The LLM evaluator docs cover creating the judge itself.

FAQ

Is LLM-as-a-judge as good as human evaluation?

On MT-Bench chat answers, GPT-4 agreed with humans 85% of the time on non-tie votes, against 81% between humans. In a different setup with high human agreement, Thakur and colleagues found that only the best judges reached reasonable alignment, and even they fell well short of inter-human agreement. Test your own judge on your own labels.

Should I replace human evaluation with an LLM judge?

Not entirely. Humans label the examples that define what good means and validate the judge, then the judge handles the volume. Hamel Husain's evals FAQ says to test a judge against human-labeled examples, and HoneyHive's docs say annotation queues turn judgments into ground truth for evaluators.

Is an LLM judge cheaper than human review?

Per output, usually yes, but a judge is not free to set up. The evals FAQ says judges require 100+ labeled examples and ongoing weekly maintenance, and advises building them only for problems you will iterate on repeatedly.

When should I use code instead of either?

When a deterministic rule can find the failure. The evals FAQ says to use a code-based eval in that case, and an LLM judge only when the failure requires human judgment.