# Better Summarization Evaluation with Word Embeddings for ROUGE

arXiv: https://arxiv.org/abs/1508.06034 · HTML: https://mlsanity.com/paper/1508.06034/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Aug 2015.

## Built on by (most cited first, 30 of 66)

- [Generative Judge for Evaluating Alignment](https://mlsanity.com/paper/2310.05470/) (2023)
- [Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Method](https://mlsanity.com/paper/2305.13412/) (2023)
- [Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation](https://mlsanity.com/paper/2208.11646/) (2022)
- [An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics](https://mlsanity.com/paper/2207.00939/) (2022)
- [Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory](https://mlsanity.com/paper/2305.14889/) (2023)
- [DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence](https://mlsanity.com/paper/2201.11176/) (2022)
- [InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation](https://mlsanity.com/paper/2112.01589/) (2021)
- [Better than Average: Paired Evaluation of NLP Systems](https://mlsanity.com/paper/2110.10746/) (2021)
- [Controllable Abstractive Dialogue Summarization with Sketch Supervision](https://mlsanity.com/paper/2105.14064/) (2021)
- [A Survey on Multi-modal Summarization](https://mlsanity.com/paper/2109.05199/) (2021)
- [EditLens: Quantifying the Extent of AI Editing in Text](https://mlsanity.com/paper/2510.03154/) (2025)
- [How Far are We from Robust Long Abstractive Summarization?](https://mlsanity.com/paper/2210.16732/) (2022)
- [FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metricsfor Automatic Text Generation](https://mlsanity.com/paper/2110.08559/) (2021)
- [The statistical advantage of automatic NLG metrics at the system level](https://mlsanity.com/paper/2105.12437/) (2021)
- [SMART: Sentences as Basic Units for Text Evaluation](https://mlsanity.com/paper/2208.01030/) (2022)
- [Controllable Text Summarization: Unraveling Challenges, Approaches, and Prospects -- A Survey](https://mlsanity.com/paper/2311.09212/) (2023)
- [Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challenges](https://mlsanity.com/paper/2405.15604/) (2024)
- [A global analysis of metrics used for measuring performance in natural language processing](https://mlsanity.com/paper/2204.11574/) (2022)
- [Towards Explainable Evaluation Metrics for Natural Language Generation](https://mlsanity.com/paper/2203.11131/) (2022)
- [Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment](https://mlsanity.com/paper/2405.03594/) (2024)
- [MetaMetrics: Calibrating Metrics For Generation Tasks Using Human Preferences](https://mlsanity.com/paper/2410.02381/) (2024)
- [Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarks](https://mlsanity.com/paper/2305.10284/) (2023)
- [Play the Shannon Game With Language Models: A Human-Free Approach to Summary Evaluation](https://mlsanity.com/paper/2103.10918/) (2021)
- [MentSum: A Resource for Exploring Summarization of Mental Health Online Posts](https://mlsanity.com/paper/2206.00856/) (2022)
- [The Glass Ceiling of Automatic Evaluation in Natural Language Generation](https://mlsanity.com/paper/2208.14585/) (2022)
- [Unlocking Structure Measuring: Introducing PDD, an Automatic Metric for Positional Discourse Coherence](https://mlsanity.com/paper/2402.10175/) (2024)
- [Model-based Preference Optimization in Abstractive Summarization without Human Feedback](https://mlsanity.com/paper/2409.18618/) (2024)
- [SummerTime: Text Summarization Toolkit for Non-experts](https://mlsanity.com/paper/2108.12738/) (2021)
- [WikiDes: A Wikipedia-Based Dataset for Generating Short Descriptions from Paragraphs](https://mlsanity.com/paper/2209.13101/) (2022)
- [Is Summary Useful or Not? An Extrinsic Human Evaluation of Text Summaries on Downstream Tasks](https://mlsanity.com/paper/2305.15044/) (2023)

## Abstract (arXiv)

ROUGE is a widely adopted, automatic evaluation measure for text summarization. While it has been shown to correlate well with human judgements, it is biased towards surface lexical similarities. This makes it unsuitable for the evaluation of abstractive summarization, or summaries with substantial paraphrasing. We study the effectiveness of word embeddings to overcome this disadvantage of ROUGE. Specifically, instead of measuring lexical overlaps, word embeddings are used to compute the semantic similarity of the words used in summaries instead. Our experimental results show that our proposal is able to achieve better correlations with human judgements when measured with the Spearman and Kendall rank coefficients.