mlsanityData to 7 Oct 2026

Re-evaluating Evaluation in Text Summarization

arXiv 2010.07100 ↗First posted 14 Oct 2020cs.CL cs.IR cs.LGcited by 144

Not on the map. Why?

Abstract (arXiv)

Automated evaluation metrics as a stand-in for manual evaluation are an essential part of the development of text-generation tasks such as text summarization. However, while the field has progressed, our standard metrics have not -- for nearly 20 years ROUGE has been the standard evaluation in most summarization papers. In this paper, we make an attempt to re-evaluate the evaluation method for text summarization: assessing the reliability of automatic metrics using top-scoring system outputs, both abstractive and extractive, on recently popular datasets for both system-level and summary-level evaluation settings. We find that conclusions about evaluation metrics on older datasets do not necessarily hold on modern datasets and systems.

Key facts. Cited by 144 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 22; 2022: 34; 2023: 30; 2024: 30; 2025: 17; 2026: 11.

144 papers.

Sort:

By area:

  1. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  2. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  3. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  4. 2023GPTScore: Evaluate as You Desirecited by 357
  5. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  6. 2024Are We Done with MMLU?cited by 178
  7. 2023Generative Judge for Evaluating Alignmentcited by 163
  8. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  9. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  10. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  11. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  12. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
38 more
  1. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
  2. 2024Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systemscited by 87
  3. 2024FineSurE: Fine-grained Summarization Evaluation using LLMscited by 83
  4. 2021Evaluating Attribution in Dialogue Systems: The BEGIN Benchmarkcited by 75
  5. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  6. 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 73
  7. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  8. 2021A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methodscited by 71
  9. 2023Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarizationcited by 69
  10. 2022On the Limitations of Reference-Free Evaluations of Generated Textcited by 52
  11. 2021ExplainaBoard: An Explainable Leaderboard for NLPcited by 50
  12. 2022BERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generationcited by 48
  13. 2021Finding a Balanced Degree of Automation for Summary Evaluationcited by 43
  14. 2023SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluationcited by 42
  15. 2023BioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biologycited by 42
  16. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debatecited by 40
  17. 2021Automatic Text Evaluation through the Lens of Wasserstein Barycenterscited by 38
  18. 2022Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metricscited by 37
  19. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  20. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 37
  21. 2021What's in a Summary? Laying the Groundwork for Advances in Hospital-Course Summarizationcited by 35
  22. 2022DiscoScore: Evaluating Text Generation with BERT and Discourse Coherencecited by 35
  23. 2021InfoLM: A New Metric to Evaluate Summarization & Data2Text Generationcited by 33
  24. 2022Extractive is not Faithful: An Investigation of Broad Unfaithfulness Problems in Extractive Summarizationcited by 32
  25. 2024Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluationcited by 32
  26. 2023The Trickle-down Impact of Reward (In-)consistency on RLHFcited by 29
  27. 2022Generative Language Models for Paragraph-Level Question Generationcited by 28
  28. 2023Less is More for Long Document Summary Evaluation by LLMscited by 28
  29. 2022Spurious Correlations in Reference-Free Evaluation of Text Generationcited by 26
  30. 2022How Far are We from Robust Long Abstractive Summarization?cited by 25
  31. 2021EmailSum: Abstractive Email Thread Summarizationcited by 24
  32. 2023Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluationcited by 24
  33. 2024Learning to Summarize from LLM-generated Feedbackcited by 24
  34. 2021Translation Quality Assessment: A Brief Survey on Manual and Automatic Methodscited by 23
  35. 2022SNaC: Coherence Error Detection for Narrative Summarizationcited by 19
  36. 2022Readability Controllable Biomedical Document Summarizationcited by 19
  37. 2022MENLI: Robust Evaluation Metrics from Natural Language Inferencecited by 18
  38. 2022Toward Human-Like Evaluation for Natural Language Generation with Error Analysiscited by 18

50 most cited of 144 All 144 (JSON)