BLEU vs ROUGE: differences and when to use each
BLEU is precision-based and built for translation; ROUGE is recall-oriented and built for summaries. Both count word overlap, not meaning.
BLEU and ROUGE both score generated text by how many words or word sequences it shares with human reference text. The difference is the side they measure: BLEU is precision-based and was built for machine translation, while ROUGE is recall-oriented and was built for summaries. Lin’s ROUGE paper says as much, calling BLEU “a closely related measure” that is “precision-based”.
Each metric has its own glossary entry: BLEU score and ROUGE score. For how overlap metrics sit next to judges and human review, see LLM evaluation.
The difference in one table
| BLEU | ROUGE | |
|---|---|---|
| Built for | Machine translation, by Papineni et al. in 2002 | Summaries, by Lin in 2004 |
| Question it answers | How much of the candidate appears in the references (precision) | How much of the references appears in the candidate (recall) |
| Length handling | A brevity penalty for output shorter than the references | Recall rewards covering the reference; ROUGE-L also has a precision side, combined in an F-measure |
| N-gram orders | Several orders combined; the Hugging Face default is up to 4 | One order per variant (ROUGE-1, ROUGE-2), plus LCS and skip-bigram variants |
| Multiple references | A good candidate uses only one of the references’ word choices, so recall over all of them misleads | More references enlarge the denominator, since, as Lin puts it, “there might exist multiple good summaries” |
| Score range | 0 to 1 in Hugging Face | 0 to 1 in Hugging Face |
Why BLEU is precision and ROUGE is recall
BLEU’s authors explain why they avoided recall. A translation task has several valid references, each possibly using a different word choice, and a good candidate uses only one of them. Their example compares “I always invariably perpetually do.” with “I always do.” against three references. The first candidate recalls more reference words but is the poorer translation, so naive recall “is not a good measure”. Precision alone rewards very short output, so they added a multiplicative brevity penalty, computed over the whole corpus so that short sentences are not punished harshly.
ROUGE took the opposite side for summaries. Lin notes that ROUGE-N is recall-related because its denominator counts the n-grams on the reference side, and that adding references expands the space of acceptable summaries. The paper follows the success of BLEU in translation and the finding by Lin and Hovy that BLEU-like n-gram statistics can also evaluate summaries.
What the two share
Both are reference-based, so they need human-written text to compare against. Both count tokens, not meaning: the Hugging Face BLEU card says BLEU “compares overlap in tokens from the predictions and references, instead of comparing meaning”, and Kryscinski et al. report that the summarization evaluation protocol is “weakly correlated with human judgment” and “does not account for important characteristics such as factual correctness”.
Both also depend on settings. Post showed that “BLEU is in fact a parameterized metric whose values can vary wildly with changes to these parameters”. The Hugging Face ROUGE card lists stemming (off by default) and a custom tokenizer as inputs. Log the settings next to every score.
Which one to use
| Your task | Start with | Why |
|---|---|---|
| Translation, or other output with one close reference answer | BLEU | It was designed for it, and the brevity penalty guards against short output |
| Summaries checked against reference summaries | ROUGE | Recall measures how much of the reference content the summary covers |
| Summaries where word order matters | ROUGE-L or ROUGE-S | The paper’s example shows LCS and skip-bigram scoring word-order differences that ROUGE-2 ties on |
| Free-form answers with many valid phrasings | Neither alone | Rewordings score low on both; add BERTScore, an LLM judge or human review |
Neither metric checks facts, so a fluent summary with a wrong figure can still score well. When you add human ratings to cover that, measure whether raters agree with inter-annotator agreement. For a wider list of options, see LLM evaluation metrics.
Using either in an evaluation workflow
Fix the test set, references and tokenizer across runs, and compare only scores computed the same way. HoneyHive’s client-side evaluators run scoring code in your own application during experiments, so a BLEU or ROUGE function can sit beside other metrics in an experiment on a dataset.
FAQ
Is BLEU precision and ROUGE recall?
Mostly. Lin's ROUGE paper calls ROUGE-N a recall-related measure and BLEU a precision-based one. ROUGE-L is an F-measure of LCS recall and precision, and Hugging Face's ROUGE card lets you pick the variant.
Which is better for evaluating LLM output?
Neither is a general answer. Pick by task: BLEU for translation-like output with a close reference, ROUGE for summaries. For open-ended answers, both undercount valid rewordings, so add semantic similarity, an LLM judge or human review.
Why does BLEU not use recall?
The BLEU paper argues that with several references, a good translation uses only one of the possible word choices, so recalling all of them "leads to a bad translation". It adds a brevity penalty to cover short outputs instead.
Can I use BLEU for summarization or ROUGE for translation?
You can compute them, but each was designed for one task. The ROUGE paper was written for summaries and the BLEU paper for machine translation. Check how well the score tracks human ratings on your own data before relying on it.
Do BLEU and ROUGE measure meaning?
No. The Hugging Face BLEU card says BLEU compares overlap in tokens instead of meaning, and Kryscinski et al. report that the summarization protocol "does not account for important characteristics such as factual correctness".