BLEU score explained
BLEU scores generated text by its n-gram overlap with reference texts, with a penalty for short output.
BLEU (Bilingual Evaluation Understudy) is an automatic metric that scores machine-generated text by how many of its word sequences also appear in one or more human reference texts, with a penalty for outputs that are too short. It was introduced for machine translation in a 2002 ACL paper, whose authors proposed an evaluation method that is “quick, inexpensive, and language-independent” and correlates highly with human judgment (abstract).
BLEU is one reference-based method among many. For the wider picture, see LLM evaluation.
How BLEU is computed
The NLTK implementation, which cites the 2002 paper, has three parts:
- Modified precision. The count of each n-gram in the output is clipped to the largest count in any single reference. This stops a repeated common word from inflating the score.
- Uniform n-gram weights. The default weights are 0.25 for each of 1-grams to 4-grams, a setup called BLEU-4. The scores for each order combine as a geometric mean.
- Brevity penalty. Output shorter than the closest reference length is penalized. The Hugging Face card explains that short outputs can otherwise get inflated scores.
NLTK’s corpus_bleu pools match and total counts across all pairs of output and reference before dividing, while sentence_bleu scores one output (NLTK docs).
Reading the number
The Hugging Face implementation returns a value between 0 and 1, where higher means closer to the references. The same card warns that scores “are not comparable across different datasets, nor are they comparable across different languages.”
A zero is easy to hit on short text. Without smoothing, zero overlap at any n-gram order drives the whole score to 0, because the geometric mean multiplies by that zero precision. NLTK offers smoothing methods 0 to 7, based on experiments from Chen and Cherry (2014), for sentence-level use (NLTK docs).
Why two BLEU numbers often disagree
Post (2018) points out that “BLEU is in fact a parameterized metric whose values can vary wildly with changes to these parameters.” He found differences of up to 1.8 between common configurations, mainly from tokenization and normalization of the reference, and recommended the scheme used by the WMT translation conference.
The tokenizer is the usual culprit. The Hugging Face default is based on whitespace and regexes, designed to match the mteval-v13a standard used by WMT, and can be swapped for another tokenizer (card).
SacreBLEU, the tool from that paper, exists because “Comparing BLEU scores is harder than it should be.” It prints a signature string recording the settings, such as this example from its README, BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0, and its README strongly recommends sharing signatures in papers (README). It also supports chrF, chrF++ and TER.
Limits
| Limit | Source |
|---|---|
| Counts token overlap, not meaning, so human ratings can diverge from BLEU | Hugging Face BLEU card |
| Scores an output against one or more reference translations, so it needs references | NLTK docs |
| Not comparable across datasets or languages | Hugging Face BLEU card |
| Settings such as tokenization change the value | Post (2018) |
| Relying on BLEU alone can lead to poor ship decisions | Kocmi et al. (2021) |
The last row comes from a study that took human judgement as the gold standard and tested which automatic metrics best predict how two translation systems rank, using 2.3 million sentence-level human judgements (Kocmi et al., 2021).
When BLEU still fits
BLEU suits tasks with a close reference answer, such as translation, where wording overlap is a fair proxy. For open-ended LLM output, there are many correct phrasings, so a low overlap does not mean a bad answer. Pair any overlap metric with human review. Inter-annotator agreement covers how to check that reviewers agree with each other.
Running BLEU in an evaluation workflow
If you track BLEU across prompt or model changes, keep the test set, references and tokenizer fixed, and record the settings next to each score. HoneyHive supports client-side evaluators that run scoring in your own code during experiments, and Python evaluators for custom deterministic metrics.
FAQ
What does BLEU stand for?
Bilingual Evaluation Understudy. It comes from the 2002 ACL paper by Papineni, Roukos, Ward and Zhu, titled "Bleu: a Method for Automatic Evaluation of Machine Translation".
What is a good BLEU score?
There is no universal threshold. The Hugging Face BLEU card says scores are not comparable across datasets or languages, and Post (2018) shows that tokenization and normalization settings change the value. Compare only runs that share the test set and the settings.
Does BLEU measure meaning?
No. The Hugging Face BLEU card says it "compares overlap in tokens from the predictions and references, instead of comparing meaning."
Is BLEU a 0 to 1 or a 0 to 100 score?
The Hugging Face implementation returns a number between 0 and 1, while SacreBLEU reports on a 0 to 100 scale (its README example prints BLEU ... = 20.8). Check which scale a tool reports before comparing numbers across papers or libraries.
Can I use BLEU to evaluate LLM outputs?
Only where a reference text exists and wording overlap is a fair proxy for quality. The NLTK docs describe it as a machine translation metric, and a 2021 WMT study argues that relying only on BLEU led to poor deployment decisions.