# LLM evaluation metrics explained: which to use and when

> Pick an LLM evaluation metric by what you have: a reference answer, retrieved context, tests or a rubric. Table of metric families, with the papers behind each.

Source: https://www.honeyhive.ai/learn/llm-evaluation-metrics (updated 2026-11-06)

Choose an LLM evaluation metric by what you can compare the output with. With a reference answer, use an overlap or embedding score. With unit tests, use functional correctness. With retrieved context, use RAG metrics. With only a rubric, use an LLM judge. Most real systems need two or three of these, because no single number covers a model or an agent.

The decision table below maps each situation to a metric family, and the sections after it link to the paper behind each. For how metrics fit into a test set and a workflow, see [LLM evaluation](/learn/llm-evaluation).

## Which metric for which situation

| You have | Question | Metric family | Examples | Main limit |
|---|---|---|---|---|
| A reference text and a task where wording matters | Does the output match the reference? | Reference-based overlap | [BLEU](/learn/bleu-score), [ROUGE](/learn/rouge-score) | Counts words, not meaning |
| A reference text and varied wording | Does it mean the same thing? | Embedding-based | [BERTScore](/learn/bertscore), [COMET](https://aclanthology.org/2020.emnlp-main.213/) | Misses factual errors |
| Token probabilities from your own model | How well does the model predict this text? | Language model score | [Perplexity](/learn/perplexity-metric) | Says nothing about correctness |
| A rubric, no reference | Is the answer good by criteria I can write down? | [LLM-as-a-judge](/learn/llm-as-a-judge) | [G-Eval](https://arxiv.org/abs/2303.16634), MT-Bench-style grading | Judge biases; needs human calibration |
| Retrieved context | Is the answer supported by the context? | RAG metrics | Faithfulness, answer relevance, context relevance | Depends on an LLM judge inside the metric |
| Unit tests or a checkable end state | Did the task actually get done? | Task and agent metrics | pass@k, pass^k, resolved rate | Needs tests or a goal state |

## Reference-based metrics

These compare generated text with one or more human references. [BLEU](/learn/bleu-score) came from machine translation and measures n-gram precision with a brevity penalty, per the [2002 paper](https://aclanthology.org/P02-1040/). [ROUGE](/learn/rouge-score) came from summarization and counts overlapping units with reference summaries, per [Lin's 2004 paper](https://aclanthology.org/W04-1013/).

They are cheap, deterministic and need no model call. The cost is that they reward matching words. The Codex paper's authors [reported](https://ar5iv.labs.arxiv.org/html/2107.03374) that functionally incorrect programs often scored higher on BLEU than correct ones. Kryscinski and colleagues [wrote](https://arxiv.org/abs/1908.08960) that the standard summarization evaluation protocol "is weakly correlated with human judgment and does not account for important characteristics such as factual correctness".

## Embedding-based metrics

These compare meaning instead of exact words; [semantic similarity](/learn/semantic-similarity) explains how the scores are computed and why they have no universal scale. [BERTScore](/learn/bertscore) matches contextual token embeddings between candidate and reference and returns precision, recall and F1, per the [paper](https://arxiv.org/abs/1904.09675). [COMET](https://aclanthology.org/2020.emnlp-main.213/) is a neural framework for translation that scores using both the source text and a reference, and is trained on human judgments.

They handle paraphrase better than overlap. They still need a reference, and the BERTScore paper says factual errors are hard for it to detect. [Perplexity](/learn/perplexity-metric) belongs nearby as the language model metric: the exponentiated average negative log-likelihood of a text, per the [Hugging Face docs](https://huggingface.co/docs/transformers/perplexity). It tests a model's fit to a text, not whether an answer is right.

## LLM-as-judge metrics

A judge model scores an output against a rubric, which handles qualities that code cannot check, such as tone or helpfulness. The [MT-Bench paper](https://arxiv.org/abs/2306.05685) describes three setups: pairwise comparison, single answer grading and reference-guided grading. It found that strong judges reach over 80% agreement with human preferences, the same level as agreement between humans.

It also names the failure modes: position bias, verbosity bias, self-enhancement bias, and limited ability to grade math and reasoning. [G-Eval](https://arxiv.org/abs/2303.16634) adds chain-of-thought reasoning and a form-filling format, and reached a Spearman correlation of 0.514 with human judgments on summarization with GPT-4. Its authors also note that LLM-based evaluators may favor LLM-written text.

Treat a judge as a metric you have to validate. Label a sample by hand, run the judge on the same sample and measure agreement with [Cohen's kappa](/learn/inter-annotator-agreement). Then read failures to find new criteria, as in [error analysis](/learn/error-analysis-llm).

## RAG metrics

A RAG system can fail in retrieval or in generation, so score them separately. Es and colleagues [name](https://arxiv.org/abs/2309.15217) three dimensions: whether the retriever finds relevant and focused passages, whether the LLM uses them faithfully, and the quality of the generation. Its [definitions](https://arxiv.org/html/2309.15217) are short:

- **Faithfulness:** the answer is split into statements, each is checked against the context, and the score is the supported share.
- **Answer relevance:** an LLM generates questions from the answer, and the score is the mean embedding similarity with the original question.
- **Context relevance:** the share of context sentences an LLM extracts as needed to answer the question.

For retrieval alone, [Hamel Husain's evals FAQ](https://hamel.dev/blog/posts/evals-faq/) points to recall@k, precision@k and MRR. More on the pipeline is in [RAG pipeline](/learn/rag-pipeline) and [agentic RAG](/learn/agentic-rag).

## Task and agent metrics

When the output can be run or checked, score the result. The [Codex paper](https://arxiv.org/abs/2107.03374) introduced HumanEval for functional correctness and defined pass@k: generate n samples per problem, count the c that pass unit tests, and apply an unbiased estimator, 1 - C(n-c, k) / C(n, k), averaged over problems. [SWE-bench](/learn/swe-bench) uses the same idea at repository scale: a task is [resolved](https://arxiv.org/abs/2310.06770) when the patch applies and the tests pass. [LiveCodeBench](/learn/livecodebench) [applies](https://arxiv.org/html/2403.07974v2) pass@1 to new problems from LeetCode, AtCoder and Codeforces.

Agents add the question of consistency. [tau-bench](/learn/tau-bench) scores the final database state against an annotated goal state and introduced pass^k, which asks for all k trials of a task to succeed. The authors [reported](https://arxiv.org/abs/2406.12045) that even a GPT-4o agent solved under half of the tasks, with pass^8 below 25% in the retail domain. Anthropic's [guide to agent evals](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) separates pass@k from pass^k the same way and grades both outcomes and transcripts. See [how to evaluate AI agents](/learn/ai-agent-evaluation) and [how to test AI agents](/learn/testing-ai-agents).

## Running metrics in HoneyHive

HoneyHive's [evaluator docs](https://docs.honeyhive.ai/v2/evaluators/introduction.md) describe three types: Python code evaluators, LLM-assisted evaluators and human evaluators. Code and LLM evaluators can run client-side in offline experiments or in CI, and server-side evaluators can run on production traces. Any metric above that you can write as a function or a rubric fits one of the three.

## FAQ

### What are the main LLM evaluation metrics?

They fall into five families: reference-based overlap scores such as [BLEU](/learn/bleu-score) and [ROUGE](/learn/rouge-score), embedding-based scores such as [BERTScore](/learn/bertscore), LLM-as-judge scores, RAG metrics such as faithfulness and context relevance, and task or agent metrics such as pass@k and pass^k.

### Which metric should I use for a chatbot or RAG app with no reference answers?

Use reference-free metrics. A 2023 [paper](https://arxiv.org/abs/2309.15217) by Es and colleagues proposes metrics for RAG that work without ground truth annotations, and [MT-Bench](https://arxiv.org/abs/2306.05685) shows LLM judges can grade single answers against a rubric. Check the judge against human labels first.

### Is BLEU or ROUGE still useful for LLM outputs?

Only when a reference text exists and wording overlap is a fair proxy for quality, such as translation or extractive summaries. The Codex authors [found](https://ar5iv.labs.arxiv.org/html/2107.03374) that incorrect programs often scored higher on BLEU than correct ones, so for open-ended output use a judge or tests.

### What is the difference between pass@k and pass^k?

pass@k counts a task as solved if any of k attempts succeeds, as defined in the [Codex paper](https://arxiv.org/abs/2107.03374). pass^k, from the [tau-bench paper](https://arxiv.org/abs/2406.12045), asks for all k trials to succeed, so it measures reliability.

### How do I know an LLM-as-judge metric can be trusted?

Compare its scores with human labels on a sample. The MT-Bench authors [report](https://arxiv.org/abs/2306.05685) over 80% agreement with humans for strong judges, and also position, verbosity and self-enhancement bias. [Cohen's kappa](/learn/inter-annotator-agreement) is the usual agreement score.