Metrics that score generated text by its n-gram overlap with reference texts. BLEU, built for machine translation, is precision-oriented: how many of the output's n-grams appear in the references. ROUGE, built for summarization, is recall-oriented: how many of the reference's n-grams the output recovers.
Card 1 of 292. Answer: Metrics that score generated text by its n-gram overlap with reference texts. BLEU, built for machine translation, is precision-oriented: how many of the output's n-grams appear in the references. ROUGE, built for summarization, is recall-oriented: how many of the reference's n-grams the output recovers.
About · Quizzes · History of AI · To suggest new questions or report an error send me a dm.
BLEU (Papineni and colleagues at IBM, 2002) takes the geometric mean of the clipped n-gram precisions for n = 1 to 4 and multiplies it by a brevity penalty, so very short outputs can't score well by being precise. ROUGE (Chin-Yew Lin, 2004) comes in several forms: ROUGE-N counts recalled n-grams, and ROUGE-L uses the longest common subsequence.
Example: the reference is "the cat sat on the mat" and the output is "the cat sat on mat". Every output word appears in the reference, so unigram precision is 5/5, and 3 of its 4 bigrams match, 0.75. Recall is lower: ROUGE-1 is 5/6 ≈ 0.83 and ROUGE-2 is 3/5 = 0.6. The output is one word short, so BLEU's brevity penalty is e^(1 − 6/5) ≈ 0.82.
The limits come up in interviews: both reward surface overlap, so a correct paraphrase can score low and a fluent wrong answer with the right words can score high, and they correlate poorly with human judgment on open-ended LLM output. Embedding-based scores such as BERTScore, LLM-as-a-judge and task-specific checks are the usual alternatives. Report BLEU with a standard tool such as SacreBLEU, since tokenization changes the number.
