How to build an LLM judge evaluator

To build an LLM judge evaluator, pick one failure you care about, have an expert label real outputs pass or fail with written critiques, write a narrow prompt that returns a constrained verdict, and tune it against those labels on held-out data before you run it at scale. The order matters as much as the prompt. For what a judge is and its formats, see LLM-as-a-judge.

1. Start from a failure, not a metric

Hamel Husain says to fix obvious system errors before building a judge: “Instead of plowing ahead and building an LLM judge, you want to fix any obvious errors.” He also advises against ready-made metrics, because generic evaluations “waste time and create false confidence” when used as quality measures. Error analysis is how you find the failures worth a judge, and the same FAQ reserves an LLM or other classifier for checks that require judgment.

2. Get expert labels with critiques

Husain recommends identifying early the principal domain expert, whose judgment is crucial to the product’s success, and building a dataset that represents the interactions the product will have in production. The expert gives a binary verdict and a written critique, which should be “detailed enough so that you can use it in a few-shot prompt for a LLM judge.”

How many examples:

• Discovery: Husain starts with around 30 examples and continues until no new failure modes appear.
• Validation: about 100 examples per failure mode, with 30 to 50 of each label in each set where possible.
• Floor: below 60 examples, he says, the confidence intervals are often too wide for a useful conclusion.

If your labelers disagree, the judge has nothing stable to match; check inter-annotator agreement first. A golden dataset is where these labels end up.

3. Write the judge prompt

Provider and practitioner guidance lines up on these choices.

Choice	What the sources say
Scope	Husain calls one judge for too many error types the most common mistake. Yan finds that scoring one dimension per prompt usually performs better. Anthropic recommends an isolated judge per dimension.
Output	Binary where possible. Yan has evaluators return binary outputs because it improves performance and makes recall and precision easy to apply. OpenAI suggests pairwise comparison or pass/fail for reliability.
Rubric	OpenAI says to keep rubrics clear and detailed. Anthropic wants structured rubrics for each dimension.
Reasoning	OpenAI says reasoning before scoring improves eval performance, and Yan reports that chain of thought improved evaluator accuracy in Constitutional AI.
Escape hatch	Anthropic suggests giving the judge a way out, such as returning “Unknown” when it lacks enough information, to reduce hallucinated grades.
Context	Husain advises giving each judge only the parts of the trace it needs, because extra context can hurt.
Examples	Use the expert critiques as few-shot examples, but avoid terse critiques and non-diverse examples. Yan warns that results can vary with label choice, example order and count.

A constrained output also makes parsing simple. Anthropic’s grading examples end the prompt with “Output only ‘yes’ or ‘no’.” and its Likert example with “Output only the number.” Those examples also note that it is generally best practice to grade with a different model from the one that generated the output.

Pairwise or direct scoring depends on the quality. In an August 2024 review, Yan describes direct scoring as better suited to objective checks such as faithfulness to a source text, and pairwise comparison as the more reliable choice for subjective qualities such as tone or coherence. He also cites the Prometheus ablation, where excluding the reference answer caused the greatest drop in performance, so a reference-free judge needs checking against labels rather than trusting.

4. Iterate against the labels

Split the labeled examples so that a small training set supplies few-shot examples and the development and test sets stay out of the prompt, as Husain describes. Run the judge on the development set, read the disagreements, and change the prompt. In his worked example, it took three iterations to reach over 90% agreement with the expert.

Measure more than raw agreement:

• Husain warns that raw agreement misleads when classes are imbalanced, and reports true positive and true negative rates separately.
• Yan notes that Cohen’s kappa adjusts for chance agreement and, in the Judging the Judges study, separated evaluators better than percent agreement.
• OpenAI says to use the most capable model to grade if you can, so start there while you write the prompt.

OpenAI advises scaling up once the judge is faster, cheaper and consistently agrees with human annotations. Anthropic adds that judges should be closely calibrated with human experts. Check the judge for the known biases before you rely on it, and see LLM judge vs human evaluation for what to keep with people.

5. Run it, then keep checking

For low-latency production guardrails, Yan suggests finetuning a classifier or reward model rather than calling an LLM. A judge can serve offline and online evaluation alike. Re-check it against fresh human labels when your product or judge model changes.

Set up the evaluator in HoneyHive

HoneyHive’s LLM evaluator docs cover the setup. After adding a provider under Settings, Workspace, AI Providers, open the Evaluators tab, choose Add Evaluator, then LLM Evaluator. Write the prompt with template variables such as {{ inputs.question }} and {{ outputs.content }}, set the return type to boolean, numeric or string, and set event filters and a sampling rate, which defaults to 10% for new evaluators. The evaluators overview explains how the same evaluator runs on offline experiments and on ingested traces.

FAQ

What goes in an LLM judge prompt?

Practitioner and provider guidance point to a clear, detailed rubric, one dimension per prompt, a constrained output such as pass or fail, and reasoning before the verdict. Examples written from expert critiques can be added, but Eugene Yan notes that calibrating few-shot examples can be tricky.

Should I write one judge or several?

Several. Hamel Husain calls directing one judge to catch too many kinds of errors at once the most common mistake, and Anthropic recommends grading each dimension with an isolated judge.

Do I need a reference answer to build a judge?

Not always. Direct scoring suits objective checks such as faithfulness to a source text with no comparison response. In the Prometheus ablation Yan cites, removing the reference answer caused the largest drop in performance, so test a reference-free judge against human labels.

How do I know the judge is good enough to scale?

OpenAI advises scaling up once the judge is faster, cheaper and consistently agrees with human annotations. Measure that on a held-out set with true positive and true negative rates, as Husain describes.