mlsanityData to 7 Oct 2026

Better Summarization Evaluation with Word Embeddings for ROUGE

arXiv 1508.06034 ↗First posted 25 Aug 2015cs.CL cs.IRcited by 66

Not on the map. Why?

Abstract (arXiv)

ROUGE is a widely adopted, automatic evaluation measure for text summarization. While it has been shown to correlate well with human judgements, it is biased towards surface lexical similarities. This makes it unsuitable for the evaluation of abstractive summarization, or summaries with substantial paraphrasing. We study the effectiveness of word embeddings to overcome this disadvantage of ROUGE. Specifically, instead of measuring lexical overlaps, word embeddings are used to compute the semantic similarity of the words used in summaries instead. Our experimental results show that our proposal is able to achieve better correlations with human judgements when measured with the Spearman and Kendall rank coefficients.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Aug 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 18; 2023: 16; 2024: 9; 2025: 8; 2026: 3.

66 papers.

Sort:

By area:

  1. 2023Generative Judge for Evaluating Alignmentcited by 163
  2. 2023Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Methodcited by 76
  3. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  4. 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 73
  5. 2023Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theorycited by 41
  6. 2022DiscoScore: Evaluating Text Generation with BERT and Discourse Coherencecited by 35
  7. 2021InfoLM: A New Metric to Evaluate Summarization & Data2Text Generationcited by 33
  8. 2021Better than Average: Paired Evaluation of NLP Systemscited by 30
  9. 2021Controllable Abstractive Dialogue Summarization with Sketch Supervisioncited by 27
  10. 2021A Survey on Multi-modal Summarizationcited by 26
  11. 2025EditLens: Quantifying the Extent of AI Editing in Textcited by 26
  12. 2022How Far are We from Robust Long Abstractive Summarization?cited by 25
38 more
  1. 2021FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metricsfor Automatic Text Generationcited by 19
  2. 2021The statistical advantage of automatic NLG metrics at the system levelcited by 18
  3. 2023Controllable Text Summarization: Unraveling Challenges, Approaches, and Prospects -- A Surveycited by 18
  4. 2024Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challengescited by 18
  5. 2022A global analysis of metrics used for measuring performance in natural language processingcited by 17
  6. 2022SMART: Sentences as Basic Units for Text Evaluationcited by 17
  7. 2022Towards Explainable Evaluation Metrics for Natural Language Generationcited by 15
  8. 2024MetaMetrics: Calibrating Metrics For Generation Tasks Using Human Preferencescited by 15
  9. 2023Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarkscited by 14
  10. 2021Play the Shannon Game With Language Models: A Human-Free Approach to Summary Evaluationcited by 12
  11. 2022MentSum: A Resource for Exploring Summarization of Mental Health Online Postscited by 11
  12. 2022The Glass Ceiling of Automatic Evaluation in Natural Language Generationcited by 10
  13. 2024Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deploymentcited by 10
  14. 2024Unlocking Structure Measuring: Introducing PDD, an Automatic Metric for Positional Discourse Coherencecited by 7
  15. 2024Model-based Preference Optimization in Abstractive Summarization without Human Feedbackcited by 7
  16. 2021SummerTime: Text Summarization Toolkit for Non-expertscited by 6
  17. 2022WikiDes: A Wikipedia-Based Dataset for Generating Short Descriptions from Paragraphscited by 6
  18. 2023Is Summary Useful or Not? An Extrinsic Human Evaluation of Text Summaries on Downstream Taskscited by 6
  19. 2021Summary Explorer: Visualizing the State of the Art in Text Summarizationcited by 4
  20. 2023A comprehensive review of automatic text summarization techniques: method, data, evaluation and codingcited by 4
  21. 2023MISMATCH: Fine-grained Evaluation of Machine-generated Text with Mismatch Error Typescited by 4
  22. 2023UMSE: Unified Multi-scenario Summarization Evaluationcited by 3
  23. 2024Speech vs. Transcript: Does It Matter for Human Annotators in Speech Summarization?cited by 3
  24. 2026Pangram 4 Technical Reportcited by 3
  25. 2022WIDAR -- Weighted Input Document Augmented ROUGEcited by 2
  26. 2022DATScore: Evaluating Translation with Data Augmented Translationscited by 2
  27. 2023Automatic Personalized Impression Generation for PET Reports Using Large Language Modelscited by 2
  28. 2025Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond Englishcited by 2
  29. 2025References Matter: Investigating the Impact of Reference Set Variation on Summarization Evaluationcited by 2
  30. 2025Navigating Through Paper Flood: Advancing LLM-based Paper Evaluation through Domain-Aware Retrieval and Latent Reasoningcited by 2
  31. 2026MMViR: A Multi-Modal and Multi-Granularity Representation for Long-range Video Understandingcited by 2
  32. 2022Text Summarization with Oracle Expectationcited by 1
  33. 2023Lay Text Summarisation Using Natural Language Processing: A Narrative Literature Reviewcited by 1
  34. 2023On Bias and Fairness in NLP: Investigating the Impact of Bias and Debiasing in Language Models on the Fairness of Toxicity Detectioncited by 1
  35. 2023Redundancy Aware Multi-Reference Based Gainwise Evaluation of Extractive Summarizationcited by 1
  36. 2023Comparative Experimentation of Accuracy Metrics in Automated Medical Reporting: The Case of Otitis Consultationscited by 1
  37. 2024LUNA: A Framework for Language Understanding and Naturalness Assessmentcited by 1
  38. 2024ROUGE-K: Do Your Summaries Have Keywords?cited by 1

50 most cited of 66 All 66 (JSON)