# BERTScore explained

> BERTScore rates generated text by matching contextual token embeddings with a reference, giving precision, recall and F1. It has known limits.

Source: https://www.honeyhive.ai/learn/bertscore (updated 2026-11-01)

BERTScore is an automatic metric that scores a piece of generated text against a reference by comparing the meaning of their tokens. It matches contextual embeddings from a pretrained language model instead of exact words, and returns precision, recall and F1. It was introduced in [a 2019 paper by Zhang and colleagues](https://arxiv.org/abs/1904.09675) and presented at ICLR 2020.

It is one way to grade outputs when you have reference answers. For how it fits next to code checks, LLM judges and human review, see [LLM evaluation](/learn/llm-evaluation).

## How it works

The paper's [method section](https://ar5iv.labs.arxiv.org/html/1904.09675) works in four steps:

1. Embed every token of the reference and the candidate with a pretrained model, so each vector depends on the surrounding words.
2. Compare each reference token with each candidate token by cosine similarity. The vectors are pre-normalized, so this is an inner product.
3. Match greedily. Recall takes the best candidate match for each reference token and averages them. Precision does the reverse.
4. Combine precision and recall into F1. The authors [recommend F1](https://ar5iv.labs.arxiv.org/html/1904.09675) as the default for machine translation.

Because matching is greedy, the same candidate token can be matched by several reference tokens. The authors tested swapping it for Word Mover's Distance and [found no consistent gain](https://ar5iv.labs.arxiv.org/html/1904.09675).

## Optional settings

| Setting | What it does | Source note |
|---|---|---|
| idf weighting | Weights rare tokens more, using idf computed from the reference sentences of the test set. | The [paper](https://ar5iv.labs.arxiv.org/html/1904.09675) says it helps in some cases and not others. The README [warns](https://github.com/Tiiiger/bert_score) it can be invalid when the reference set is very small. |
| Baseline rescaling | Rescales scores linearly against a lower bound from random sentence pairs. | The README [says](https://github.com/Tiiiger/bert_score) it makes scores "more human-readable". The [paper](https://ar5iv.labs.arxiv.org/html/1904.09675) says it does not change rankings. |
| Model and layer | Picks which pretrained model and layer supply the embeddings. | The README maps English to roberta-large by default and [recommends deberta-xlarge-mnli](https://github.com/Tiiiger/bert_score) for better correlation with [human evaluation](/learn/human-evaluation-llm). The default layer was tuned on WMT16 data. |

## Reading the numbers

The theoretical range is -1 to 1, but scores are [usually found in a small range](https://github.com/Tiiiger/bert_score/blob/master/journal/rescale_baseline.md). For RoBERTa-large the authors give 0.85 to 0.95 as typical. A raw 0.88 therefore says little by itself.

Rescaling computes (X - Base) / (1 - Base), where Base is the average score over about half a million random sentence pairs for that language, model, layer and metric ([rescaling note](https://github.com/Tiiiger/bert_score/blob/master/journal/rescale_baseline.md)). The note says this leaves the correlation with human judgment unchanged. It only makes the scale easier to read.

Scores are only comparable when the setup is. The README [says](https://github.com/Tiiiger/bert_score) to report the hash code so readers know your settings, and notes that the transformers version and the tokenizer implementation can change results. The library also truncates sentences longer than 510 tokens, where BERTScore is undefined.

## What the paper found

The authors evaluated on [machine translation and image captioning systems](https://arxiv.org/abs/1904.09675), 363 in total, and report better correlation with human judgments than existing metrics. In the [paper](https://ar5iv.labs.arxiv.org/html/1904.09675):

- On WMT18 translation data it was a consistently strong metric, particularly at the segment level.
- On the 2015 COCO captioning entries, BERTScore beat all task-agnostic baselines and idf weighting helped significantly, though the trained metric Leic scored higher.
- On the PAWS-QQP adversarial paraphrase set, most metrics fell near chance and BERTScore dropped only slightly.

## Limits

- **Facts.** The authors say it [finds it difficult to detect factual errors](https://ar5iv.labs.arxiv.org/html/1904.09675), such as Tuesday in place of Monday in a translation. A fluent answer with a wrong figure can still score well, so pair it with checks on the facts.
- **No best configuration.** The paper says no single setup wins everywhere and that when idf helps is an open question.
- **Model dependence.** Multilingual BERT was ["less stable"](https://ar5iv.labs.arxiv.org/html/1904.09675) on low-resource languages, and the paper reports that precision-only scores correlate weakly with human judgments for captioning.
- **Needs a reference.** It compares against a reference text, so it fits tasks with known good answers such as translation, summarization or captioning. Open-ended answers often need an [LLM judge or human review](/learn/llm-evaluation) instead.

These limits come from the paper (v3, February 2020).

## Using it in an eval pipeline

The reference implementation installs with [`pip install bert-score`](https://github.com/Tiiiger/bert_score). Pin the library and model versions and keep the printed hash code with each result so runs stay comparable.

HoneyHive's [client-side evaluators](https://docs.honeyhive.ai/v2/evaluators/client_side.md) let you compute a score in your own code, such as one from a library, and either attach it to a trace with `enrich_span` or return it from an evaluator function passed to `evaluate()`. For what to trace around those calls, see [LLM observability](/learn/llm-observability).

## FAQ

### What does BERTScore measure?

How similar a candidate text is to a reference in meaning. It compares [contextual embeddings](https://arxiv.org/abs/1904.09675) of the tokens rather than counting exact word matches, and reports precision, recall and F1.

### Is a higher BERTScore always better?

Within one setup, higher means closer to the reference. Across setups, no. The [reference implementation](https://github.com/Tiiiger/bert_score) says the model, layer and library versions change the score, and it asks you to report the hash code so others know the settings.

### Why are my BERTScores all around 0.85 to 0.95?

Raw scores sit in a narrow band. The authors' [rescaling note](https://github.com/Tiiiger/bert_score/blob/master/journal/rescale_baseline.md) says RoBERTa-large scores are often between 0.85 and 0.95. Baseline rescaling spreads them out without changing rankings.

### Can BERTScore catch factual errors?

Poorly. The [paper](https://ar5iv.labs.arxiv.org/html/1904.09675) says it finds it difficult to detect factual errors. Its examples include a translation that says Tuesday where the reference says Monday, and one that swaps German for English.

### What is the difference between BERTScore and BLEU?

BLEU is an n-gram overlap metric, and BERTScore uses embedding similarity instead of exact matches. The [paper](https://arxiv.org/abs/1904.09675) reports that this correlates better with human judgments on machine translation and captioning systems.