What is LLM evaluation?

LLM evaluation is the practice of testing whether a language model, or an app built on one, produces outputs that are good enough for its job. Each test case is an input plus grading logic, and Anthropic defines an eval as “a test for an AI system: give an AI an input, then apply grading logic to its output to measure success”. The grading logic is code, another model, or a person.

Model evaluation and application evaluation

Model evaluation compares base models on shared benchmarks. MMLU, for example, scores multiple-choice questions across 57 subjects (Hendrycks et al., 2020). It helps shortlist models and says little about your product.

Application evaluation, which IBM calls LLM system evaluation, scores the whole pipeline (prompt, retrieved context, tool calls, final answer) on inputs that look like your traffic. A model can lead a leaderboard and still answer your refund-policy questions wrongly.

Choosing a grader

Grader	Good at	Weak at	Use it for
Code check	Fast, cheap, objective, reproducible, easy to debug	Brittle to valid variations; little nuance	Format, schema, required fields, exact labels, tool-call arguments
LLM judge	Flexible and scalable; handles open-ended output	Non-deterministic, costs more, needs calibration with human graders	Faithfulness, tone, relevance, whether a policy was followed
Human review	Gold-standard quality that matches expert judgment	Expensive and slow	Labeling a sample, calibrating judges, edge cases

The strengths and weaknesses are from Anthropic’s guide to agent evals (January 2026). A typical suite uses code checks for structure, judges for meaning, and people to label a sample.

Making an LLM judge trustworthy

In the MT-Bench study (Zheng et al., 2023), strong judges such as GPT-4 reached over 80% agreement with human preferences, the same level at which humans agree with each other. The same paper lists position bias, verbosity bias and self-enhancement bias.

Hamel Husain’s evals FAQ gives a practical rule: test the judge against human-labeled examples of the failure you want to detect, and track both its true positive rate (how many real failures it catches) and its true negative rate (how often good outputs pass). The same FAQ prefers binary pass or fail labels over numeric scales because they force clearer thinking and more consistent labeling, and warns that generic scores such as helpfulness create false confidence as quality measures.

What to evaluate first

Begin by reading real outputs. Hamel Husain’s FAQ calls error analysis the most important activity in evals. Failures you find become test cases, and each recurring failure gets its own grader.

Capability and regression evals

Anthropic splits evals into two kinds. Capability evals ask what the system does well and should start at a low pass rate, so there is room to improve. Regression evals ask whether it still handles everything it used to, and should sit near 100%.

For agents, Anthropic’s guide separates pass@k (at least one success in k tries) from pass^k (all k tries succeed). At a 75% per-trial success rate over 3 trials, pass^k is roughly 42%, which is why pass^k fits customer-facing agents that must be consistent.

When evals run

Offline evals run before release on a fixed dataset, to compare a change against the current version and catch regressions in CI. Online evals score live production traces, where inputs your dataset never covered show up. Failures found online become new offline test cases.

Tooling status (as of 2026-10-30)

OpenAI’s Evals guide says the Evals platform is being deprecated: read-only for existing users on October 31, 2026, with shutdown scheduled for November 30, 2026; the guide suggests Datasets for a more iterative workflow.

Running evals in HoneyHive

In HoneyHive, an experiment runs a function on a dataset through evaluate(), scores each output with evaluators and aggregates the metrics in the dashboard. Evaluators can be Python code, LLM-assisted or human. Python and LLM evaluators run client-side (for CI and quick checks) or server-side, where they can also score ingested production traces automatically; human evaluators are server-side only. A coding agent can do the setup with the honeyhive-evaluate skill (setup steps):

Use the honeyhive-evaluate skill to set up a HoneyHive experiment that runs my app's main
function on my test cases, scored by one Python evaluator and one LLM-judge evaluator.

FAQ

What is LLM-as-a-judge evaluation?

It is an eval where one language model grades another model's output against a rubric you write, usually as pass or fail. Judges handle qualities code cannot check, such as faithfulness to a source. They need calibration against human labels, and the MT-Bench paper names position, verbosity and self-enhancement bias as known failure modes.

What is LLM testing?

LLM testing is LLM evaluation run like software tests. A fixed set of inputs with pass or fail criteria runs on every prompt, model or code change, often in CI. Anthropic calls this a regression eval, and expects a pass rate near 100%.