# How to build an LLM judge evaluator

> Build an LLM judge in order: pick one failure, label examples, write a narrow pass or fail prompt, then iterate against human labels before scaling.

Source: https://www.honeyhive.ai/learn/build-llm-judge-evaluator (updated 2026-11-12)

To build an LLM judge evaluator, pick one failure you care about, have an expert label real outputs pass or fail with written critiques, write a narrow prompt that returns a constrained verdict, and tune it against those labels on held-out data before you run it at scale. The order matters as much as the prompt. For what a judge is and its formats, see [LLM-as-a-judge](/learn/llm-as-a-judge).

## 1. Start from a failure, not a metric

Hamel Husain [says](https://hamel.dev/blog/posts/llm-judge/) to fix obvious system errors before building a judge: "Instead of plowing ahead and building an LLM judge, you want to fix any obvious errors." He also [advises](https://hamel.dev/blog/posts/evals-faq/) against ready-made metrics, because generic evaluations "waste time and create false confidence" when used as quality measures. [Error analysis](/learn/error-analysis-llm) is how you find the failures worth a judge, and the same FAQ [reserves](https://hamel.dev/blog/posts/evals-faq/) an LLM or other classifier for checks that require judgment.

## 2. Get expert labels with critiques

Husain [recommends](https://hamel.dev/blog/posts/llm-judge/) identifying early the principal domain expert, whose judgment is crucial to the product's success, and building a dataset that represents the interactions the product will have in production. The expert gives a binary verdict and a written critique, which should be "detailed enough so that you can use it in a few-shot prompt for a LLM judge."

How many examples:

- Discovery: Husain [starts](https://hamel.dev/blog/posts/llm-judge/) with around 30 examples and continues until no new failure modes appear.
- Validation: about 100 examples per failure mode, with 30 to 50 of each label in each set where possible.
- Floor: below 60 examples, he [says](https://hamel.dev/blog/posts/llm-judge/), the confidence intervals are often too wide for a useful conclusion.

If your labelers disagree, the judge has nothing stable to match; check [inter-annotator agreement](/learn/inter-annotator-agreement) first. A [golden dataset](/learn/golden-dataset) is where these labels end up.

## 3. Write the judge prompt

Provider and practitioner guidance lines up on these choices.

| Choice | What the sources say |
|---|---|
| Scope | Husain [calls](https://hamel.dev/blog/posts/evals-faq/) one judge for too many error types the most common mistake. Yan [finds](https://eugeneyan.com/writing/llm-evaluators/) that scoring one dimension per prompt usually performs better. Anthropic [recommends](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) an isolated judge per dimension. |
| Output | Binary where possible. Yan has evaluators [return binary outputs](https://eugeneyan.com/writing/llm-evaluators/) because it improves performance and makes recall and precision easy to apply. OpenAI [suggests](https://developers.openai.com/api/docs/guides/evaluation-best-practices) pairwise comparison or pass/fail for reliability. |
| Rubric | OpenAI says to [keep rubrics](https://developers.openai.com/api/docs/guides/evaluation-best-practices) clear and detailed. Anthropic wants [structured rubrics](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) for each dimension. |
| Reasoning | OpenAI says [reasoning before scoring](https://developers.openai.com/api/docs/guides/evaluation-best-practices) improves eval performance, and Yan [reports](https://eugeneyan.com/writing/llm-evaluators/) that chain of thought improved evaluator accuracy in Constitutional AI. |
| Escape hatch | Anthropic suggests giving the judge a way out, such as returning ["Unknown"](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) when it lacks enough information, to reduce hallucinated grades. |
| Context | Husain [advises](https://hamel.dev/blog/posts/evals-faq/) giving each judge only the parts of the trace it needs, because extra context can hurt. |
| Examples | Use the expert critiques as few-shot examples, but avoid [terse critiques](https://hamel.dev/blog/posts/llm-judge/) and non-diverse examples. Yan [warns](https://eugeneyan.com/writing/llm-evaluators/) that results can vary with label choice, example order and count. |

A constrained output also makes parsing simple. Anthropic's [grading examples](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests) end the prompt with "Output only 'yes' or 'no'." and its Likert example with "Output only the number." Those examples also note that it is generally best practice to [grade with a different model](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests) from the one that generated the output.

Pairwise or direct scoring depends on the quality. In an August 2024 review, Yan [describes](https://eugeneyan.com/writing/llm-evaluators/) direct scoring as better suited to objective checks such as faithfulness to a source text, and pairwise comparison as the more reliable choice for subjective qualities such as tone or coherence. He also [cites](https://eugeneyan.com/writing/llm-evaluators/) the Prometheus ablation, where excluding the reference answer caused the greatest drop in performance, so a reference-free judge needs checking against labels rather than trusting.

## 4. Iterate against the labels

Split the labeled examples so that a small training set supplies few-shot examples and the development and test sets stay out of the prompt, as Husain [describes](https://hamel.dev/blog/posts/llm-judge/). Run the judge on the development set, read the disagreements, and change the prompt. In his worked example, it took [three iterations](https://hamel.dev/blog/posts/llm-judge/) to reach over 90% agreement with the expert.

Measure more than raw agreement:

- Husain [warns](https://hamel.dev/blog/posts/llm-judge/) that raw agreement misleads when classes are imbalanced, and reports true positive and true negative rates separately.
- Yan [notes](https://eugeneyan.com/writing/llm-evaluators/) that Cohen's kappa adjusts for chance agreement and, in the Judging the Judges study, separated evaluators better than percent agreement.
- OpenAI [says](https://developers.openai.com/api/docs/guides/evaluation-best-practices) to use the most capable model to grade if you can, so start there while you write the prompt.

OpenAI [advises](https://developers.openai.com/api/docs/guides/evaluation-best-practices) scaling up once the judge is faster, cheaper and consistently agrees with human annotations. Anthropic [adds](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) that judges should be closely calibrated with human experts. Check the judge for the [known biases](/learn/llm-judge-limitations) before you rely on it, and see [LLM judge vs human evaluation](/learn/llm-judge-vs-human-evaluation) for what to keep with people.

## 5. Run it, then keep checking

For low-latency production guardrails, Yan [suggests](https://eugeneyan.com/writing/llm-evaluators/) finetuning a classifier or reward model rather than calling an LLM. A judge can serve [offline and online evaluation](/learn/online-vs-offline-evaluation) alike. Re-check it against fresh human labels when your product or judge model changes.

## Set up the evaluator in HoneyHive

HoneyHive's [LLM evaluator docs](https://docs.honeyhive.ai/v2/evaluators/llm) cover the setup. After adding a provider under Settings, Workspace, AI Providers, open the Evaluators tab, choose Add Evaluator, then LLM Evaluator. Write the prompt with template variables such as `{{ inputs.question }}` and `{{ outputs.content }}`, set the return type to [boolean, numeric or string](https://docs.honeyhive.ai/v2/evaluators/llm), and set event filters and a sampling rate, which [defaults to 10%](https://docs.honeyhive.ai/v2/evaluators/llm) for new evaluators. The [evaluators overview](https://docs.honeyhive.ai/v2/evaluators/introduction) explains how the same evaluator runs on offline experiments and on ingested traces.

## FAQ

### What goes in an LLM judge prompt?

Practitioner and provider guidance point to a [clear, detailed rubric](https://developers.openai.com/api/docs/guides/evaluation-best-practices), [one dimension per prompt](https://eugeneyan.com/writing/llm-evaluators/), a constrained output such as pass or fail, and [reasoning before the verdict](https://developers.openai.com/api/docs/guides/evaluation-best-practices). Examples written from expert critiques can be added, but Eugene Yan [notes](https://eugeneyan.com/writing/llm-evaluators/) that calibrating few-shot examples can be tricky.

### Should I write one judge or several?

Several. Hamel Husain [calls](https://hamel.dev/blog/posts/evals-faq/) directing one judge to catch too many kinds of errors at once the most common mistake, and Anthropic [recommends](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) grading each dimension with an isolated judge.

### Do I need a reference answer to build a judge?

Not always. Direct scoring [suits objective checks](https://eugeneyan.com/writing/llm-evaluators/) such as faithfulness to a source text with no comparison response. In the Prometheus ablation Yan cites, removing the reference answer caused the largest drop in performance, so test a reference-free judge against human labels.

### How do I know the judge is good enough to scale?

OpenAI [advises](https://developers.openai.com/api/docs/guides/evaluation-best-practices) scaling up once the judge is faster, cheaper and consistently agrees with human annotations. Measure that on a held-out set with true positive and true negative rates, as Husain [describes](https://hamel.dev/blog/posts/llm-judge/).