ROUGE score explained

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a set of automatic metrics that score a generated summary by how much it overlaps with one or more human-written reference summaries. Chin-Yew Lin’s 2004 paper says the measures “count the number of overlapping units such as n-gram, word sequences, and word pairs” between the computer-generated summary and the ideal summaries made by humans.

ROUGE needs a reference text, and it counts words rather than meaning. It sits beside BLEU, which came first and is precision-based (see BLEU vs ROUGE), and BERTScore, which compares embeddings. For how all of these fit with judges and human review, see LLM evaluation.

Why it was built

Lin’s paper argues that even simple manual evaluation of the DUC summaries “would require over 3,000 hours of human efforts”, which is too expensive to repeat often. The paper follows the success of BLEU in machine translation and introduces four measures: ROUGE-N, ROUGE-L, ROUGE-W and ROUGE-S.

The main variants

Variant	What it counts	Source note
ROUGE-N	Overlapping n-grams. ROUGE-1 uses single words and ROUGE-2 uses pairs.	The paper calls it “an n-gram recall between a candidate summary and a set of reference summaries”.
ROUGE-L	The longest common subsequence (LCS) of the two texts.	Matches must be in order but not consecutive, so no n-gram length is chosen in advance (paper).
ROUGE-Lsum	LCS computed per sentence, then combined as a union across the summary.	The Google package treats newlines as sentence boundaries.
ROUGE-S	Skip-bigrams: any word pair in sentence order, with gaps allowed.	Defined in the paper; ROUGE-W is a weighted LCS variant.

How the scores are computed

For ROUGE-N, the denominator is the total number of n-grams on the reference side, which is why it is a recall measure. The paper notes that adding references grows that denominator, and that because the numerator sums over all references, n-grams shared by several references count for more.

For ROUGE-L, the paper defines recall as the LCS length divided by the reference length (m) and precision as the LCS length divided by the candidate length (n). It combines them in an F-measure with a weight beta. In DUC, beta was set to a very big number, so only recall was considered.

With several references, the paper scores the candidate against each reference and takes the maximum. Its package also uses a jackknifing procedure, averaging the best score over each set of M-1 references.

What the paper’s own example shows

Lin uses four short sentences to show how the variants differ. Taking “police killed the gunman” as the reference, the paper compares “police kill the gunman” and “the gunman kill police”. Both share one bigram (“the gunman”) with the reference, so they tie on ROUGE-2 even though their meanings differ. ROUGE-L scores them 3/4 = 0.75 and 2/4 = 0.5 with beta = 1, so it prefers the first.

The paper also names a weakness of LCS. Because it counts only the main in-sequence words, “the gunman police killed” gets the same ROUGE-L score as “the gunman kill police”, while ROUGE-2 prefers the former. ROUGE-S with beta = 1 scores the three candidates 0.5, 0.167 and 0.333 respectively, and the paper calls this more intuitive than BLEU-2 or ROUGE-L.

Settings that change the number

The Google rouge-score package is a native Python implementation designed to replicate the original Perl results, but “not all options provided by the original perl ROUGE script are supported”. Its README says stopword removal is left out because the roughly 600-word list cannot be reproduced exactly and has no clear license for redistribution. Lin’s paper found that excluding stopwords usually improved correlation with human judgment, so scores from the two tools can differ for that reason.

Other settings, from the Google README and the Hugging Face card:

• Stemming. Porter stemming is optional and off by default in Hugging Face’s use_stemmer.
• Case. ROUGE is case insensitive.
• Tokenizer. The Hugging Face card says a custom tokenizer is “especially useful for non-latin languages”.
• Sentence splitting. rougeLsum needs newlines between sentences. The Google README says to use its split option when the text has none.
• Range. Values run from 0 to 1.

The Google README shows the package’s basic call:

from rouge_score import rouge_scorer

scorer = rouge_scorer.RougeScorer(['rouge1', 'rougeL'], use_stemmer=True)
scores = scorer.score('The quick brown fox jumps over the lazy dog',
                      'The quick brown dog jumps on the log.')

Report the library, its version and these options with every score, as with BLEU.

Limits

• Weak link to human judgment. Kryscinski et al. wrote that the current evaluation protocol “is weakly correlated with human judgment and does not account for important characteristics such as factual correctness”.
• Old conclusions may not hold. Bhandari et al. note that ROUGE has been the default summarization metric for almost two decades and that conclusions about evaluation metrics drawn on older datasets do not necessarily hold on modern datasets and systems.
• Little headroom information. Schluter shows that summarisation is NP-hard under ROUGE, yet greedy algorithms seem to score optimally on three benchmark datasets, and argues there is no natural upper bound on summary quality.
• Needs references. Lin’s original evaluation used human summaries and, in the paper’s conclusions, correlation improved with multiple references.
• Words, not facts. Overlap does not check that a figure or name is right. The Hugging Face card points readers to Schluter’s paper for a fuller discussion of ROUGE’s limits.

When ROUGE still fits

ROUGE suits tasks with a close reference text, such as single-document summaries or very short, headline-like summaries, where Lin’s conclusions report that several variants correlated well with human judgment on DUC data. For abstractive or open-ended LLM output, many different wordings are correct, so a low score does not mean a bad answer. Pair it with human review; inter-annotator agreement covers how to check that reviewers agree with each other.

Running ROUGE in an evaluation workflow

Keep the reference set, tokenizer and stemming choice fixed across runs, and log the settings next to each score. HoneyHive’s client-side evaluators let you compute a score such as ROUGE in your own code and send it with the experiment results.

FAQ

What does ROUGE stand for?

Recall-Oriented Understudy for Gisting Evaluation. Chin-Yew Lin introduced it in a 2004 paper titled "ROUGE: A Package for Automatic Evaluation of Summaries".

What is the difference between ROUGE-1, ROUGE-2 and ROUGE-L?

The Hugging Face card describes rouge1 as unigram scoring, rouge2 as bigram scoring and rougeL as scoring by the longest common subsequence. ROUGE-L does not need consecutive matches, only words in the same order.

Is ROUGE recall or precision?

Lin's paper defines ROUGE-N as an n-gram recall. ROUGE-L is an F-measure of LCS recall and precision, but the paper says DUC set the weighting so that only recall counted. Google's rouge-score package returns precision, recall and F-measure for each score.

What is a good ROUGE score?

There is no universal threshold. The Hugging Face card says values range from 0 to 1. The score also depends on your references, tokenization and stemming settings. Compare only runs that share all three.

Does ROUGE agree with human judgment?

Often weakly on modern systems. Kryscinski et al. report that the current evaluation protocol "is weakly correlated with human judgment and does not account for important characteristics such as factual correctness", and Bhandari et al. find that conclusions about evaluation metrics drawn on older datasets do not necessarily hold on modern datasets and systems.