mlsanityData to 7 Oct 2026

SummEval: Re-evaluating Summarization Evaluation

arXiv 2007.12626 ↗First posted 24 Jul 2020cs.CLcited by 666

Not on the map. Why?

Cited by 20 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The scarcity of comprehensive up-to-date studies on evaluation metrics for text summarization and the lack of consensus regarding evaluation protocols continue to inhibit progress. We address the existing shortcomings of summarization evaluation methods along five dimensions: 1) we re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion using neural summarization model outputs along with expert and crowd-sourced human annotations, 2) we consistently benchmark 23 recent summarization models using the aforementioned automatic evaluation metrics, 3) we assemble the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset and share it in a unified format, 4) we implement and share a toolkit that provides an extensible and unified API for evaluating summarization models across a broad range of automatic metrics, 5) we assemble and share the largest and most diverse, in terms of model types, collection of human judgments of model-generated summaries on the CNN/Daily Mail dataset annotated by both expert judges and crowd-source workers. We hope that this work will help promote a more complete evaluation protocol for text summarization as well as advance research in developing evaluation metrics that better correlate with human judgments.

Key facts. Cited by 666 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 50; 2022: 96; 2023: 137; 2024: 145; 2025: 131; 2026: 107.

666 papers.

Sort:

By area:

  1. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  3. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  4. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  5. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  6. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  7. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  8. 2023GPTScore: Evaluate as You Desirecited by 357
  9. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  10. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  11. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  12. 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AIcited by 299
38 more
  1. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 286
  2. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  3. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  4. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  5. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 251
  6. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  7. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  8. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  9. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  10. 2024Generative Representational Instruction Tuningcited by 227
  11. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  12. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  13. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  14. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  15. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  16. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  17. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  18. 2021Annotating and Modeling Fine-grained Factuality in Summarizationcited by 131
  19. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  20. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  21. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  22. 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 119
  23. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  24. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  25. 2021Robustness Gym: Unifying the NLP Evaluation Landscapecited by 107
  26. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 103
  27. 2023Enhancing Uncertainty-Based Hallucination Detection with Stronger Focuscited by 98
  28. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  29. 2022Evaluating Human-Language Model Interactioncited by 90
  30. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
  31. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 87
  32. 2023Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Studycited by 85
  33. 2024FineSurE: Fine-grained Summarization Evaluation using LLMscited by 83
  34. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 82
  35. 2023MeetingBank: A Benchmark Dataset for Meeting Summarizationcited by 81
  36. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  37. 2023Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Methodcited by 76
  38. 2021Evaluating Attribution in Dialogue Systems: The BEGIN Benchmarkcited by 75

50 most cited of 666 All 666 (JSON)