How to Write an LLM Evaluation Rubric

Evaluate agents
HoneyHive Engineering

An LLM evaluation rubric is a written criterion, a scoring format and a way to check it. How to choose pass/fail or a scale, word criteria, and validate.

An LLM evaluation rubric is the written standard a judge, human or model, uses to score one output: the criterion, the format of the verdict, and the instructions that make two graders reach the same answer. A good one tests a single quality, returns a verdict that is easy to aggregate, and is checked against human labels before anyone trusts its numbers.

For the judging technique itself, see LLM-as-a-judge, its limitations and G-Eval, which generates evaluation steps from criteria automatically.

What a rubric contains

A survey of rubrics across LLM evaluation defines them as explicit sets of criteria and describes their evaluation role as decomposing an overall judgment into checkable dimensions. In practice, a rubric has three parts:

Part What it answers Source for the wording
Criterion What exactly counts as good or bad? Written by someone who knows the task
Verdict format Pass/fail, a number, or a label? Anthropic says to be specific, for example output only “correct” or “incorrect”, or judge from 1 to 5
Validation Does it agree with people? Labeled examples, covered below

Evaluator models can also take a rubric as an input. The Prometheus authors trained an open evaluator that receives a reference answer and a score rubric, and report a 0.897 Pearson correlation with human evaluators on 45 custom rubrics, against 0.882 for GPT-4.

Pass/fail or a scale

Practitioners and Anthropic differ on emphasis, not on the need to be specific. Husain and Shankar argue that “the difference between adjacent points (like 3 vs 4) is subjective and inconsistent across annotators,” and that annotators “often default to middle values to avoid making hard decisions.” Their advice is to start with binary labels to learn what “bad” looks like, and to call numeric labels an advanced option that is “usually not necessary.”

Anthropic’s grading guidance lists both formats as acceptable and warns that purely qualitative evaluations are hard to assess quickly and at scale. Its own examples include a yes/no grader and a 1 to 5 grader that defines only the two ends of the scale.

Choose When Watch for
Binary The standard can be stated as a rule a reader can apply A criterion that hides several qualities in one verdict
Several binary checks You need to see gradual progress More judges to validate
Numeric scale Degrees matter and you can describe each level Raters, human or model, clustering in the middle

For gradual progress without a scale, Husain and Shankar suggest counting how many expected facts a response includes instead of rating factual accuracy from 1 to 5.

Split quality into dimensions

Anthropic’s engineers recommend clear, structured rubrics that grade each dimension of a task, with an isolated LLM judge per dimension rather than one judge for all of them. Anthropic’s docs add that one success criterion may itself need several rubrics for holistic evaluation.

Large rubric sets are workable when every item is a single check. HealthBench uses about 48,500 criteria written by physicians to grade responses to 5,000 conversations, covering behaviors such as accuracy and communication.

Husain also advises building specialized judges for specific failures only after an expert has reviewed real outputs. That review is error analysis, and it tells you which criteria are worth writing.

Word the criterion

  • Be detailed and checkable. Anthropic gives this model rubric: “The answer should always mention ‘Acme Inc.’ in the first sentence. If it does not, the answer is automatically graded as ‘incorrect.’” It states the rule and the consequence.
  • Let the judge say it does not know. Anthropic advises giving the judge a way out, such as returning “Unknown” when it lacks the information.
  • Ask for reasoning before the verdict. Anthropic says a grader that reasons before scoring performs better, particularly on complex judgment. Husain wants a written critique with every human verdict, and warns that being too terse “is a common mistake.”
  • Show varied examples. For a judge prompt, Husain recommends few-shot examples that each contain an input, an output, a critique and an outcome, with enough variety to cover the inputs you expect.
  • Grade with a different model. A code comment in Anthropic’s docs calls it generally best practice to evaluate with a different model from the one that produced the output.

Check the rubric against people

A rubric is a hypothesis until you compare it with human labels. Anthropic says to test reliability first and then scale, and that judges should be calibrated against human experts, after which occasional human review is enough.

Husain’s guide gives a concrete procedure:

  1. Split labeled data into train, dev and test sets, and keep dev examples out of the prompt.
  2. Revise the prompt against the dev set.
  3. Freeze the prompt and run it on the test set once.
  4. Report the true positive rate and true negative rate separately, with human labels as ground truth, since raw agreement misleads when failures are rare.
  5. Aim for 30 to 50 examples of each label in each set when possible.

If two people cannot agree on your criterion, a model will not fix that. Measure inter-annotator agreement first, and keep a golden dataset of labeled cases to test the rubric on. See LLM judge vs human evaluation for when each is worth the cost.

Putting a rubric into HoneyHive

HoneyHive’s LLM evaluators take a custom prompt with your rubric and score traces or experiment outputs through a model provider you configure. The prompt references event fields such as inputs and outputs with {{ }} syntax, and the docs show an example that asks for a brief explanation and a 1 to 5 rating in a [[X]] pattern the evaluator extracts. Replace that example with your own criterion and format.

Reviewer labels for the validation step can come from human evaluators. To run the rubric on a dataset, see the evaluation docs, and for live traffic, online evaluations. The broader picture is in LLM evaluation.

FAQ

Should an LLM evaluation rubric use pass/fail or a 1-5 scale?

Practitioners Hamel Husain and Shreya Shankar recommend starting with binary labels and tracking gradual progress with separate yes/no checks. Anthropic's grading guidance accepts either, as long as the output is specific.

How many criteria should one judge prompt grade?

One dimension each. Anthropic's engineering team suggests grading each dimension with an isolated LLM judge instead of one judge for all of them.

How do I know a rubric works?

Compare the judge's verdicts with human labels on held-out examples. Husain recommends reporting true positive and true negative rates separately and running the frozen prompt on the test set once.

Can an LLM write the rubric for me?

Published rubric work puts people at the center. In HealthBench, 262 physicians wrote about 48,500 criteria, and Anthropic says judges should be calibrated against human experts.

Share