mlsanityData to 7 Oct 2026

Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics

arXiv 2204.10216 ↗First posted 21 Apr 2022cs.CLcited by 37

Not on the map. Why?

Abstract (arXiv)

How reliably an automatic summarization evaluation metric replicates human judgments of summary quality is quantified by system-level correlations. We identify two ways in which the definition of the system-level correlation is inconsistent with how metrics are used to evaluate systems in practice and propose changes to rectify this disconnect. First, we calculate the system score for an automatic metric using the full test set instead of the subset of summaries judged by humans, which is currently standard practice. We demonstrate how this small change leads to more precise estimates of system-level correlations. Second, we propose to calculate correlations only on pairs of systems that are separated by small differences in automatic scores which are commonly observed in practice. This allows us to demonstrate that our best estimate of the correlation of ROUGE to human judgments is near 0 in realistic scenarios. The results from the analyses point to the need to collect more high-quality human judgments and to improve automatic metrics when differences in system scores are small.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 26 of its 46 references that are in the map. First posted on arXiv 21 Apr 2022.

Builds on

26 of 46 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  4. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  5. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  6. 2020Extractive Summarization as Text Matchingcited by 189
  7. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 155
  8. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
  9. 2021SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarizationcited by 131
  10. 2020GSum: A General Framework for Guided Neural Abstractive Summarizationcited by 129
  11. 2019Discourse-Aware Neural Extractive Text Summarizationcited by 104
  12. 2020Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summarycited by 87
14 more
  1. 2021A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methodscited by 71
  2. 2020Knowledge Graph-Augmented Abstractive Summarization with Semantic-Driven Cloze Rewardcited by 66
  3. 2021Improving Factual Consistency of Abstractive Summarization via Question Answeringcited by 55
  4. 2019Crowdsourcing Lightweight Pyramids for Manual Summary Evaluationcited by 39
  5. 2021Structure-Aware Abstractive Conversation Summarization via Discourse and Action Graphscited by 36
  6. 2020Noisy Self-Knowledge Distillation for Text Summarizationcited by 31
  7. 2020TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoisingcited by 30
  8. 2021RefSum: Refactoring Neural Summarizationcited by 25
  9. 2021Unsupervised Extractive Summarization using Pointwise Mutual Informationcited by 23
  10. 2021The statistical advantage of automatic NLG metrics at the system levelcited by 18
  11. 2020Stepwise Extractive Summarization and Planning with Structured Transformerscited by 16
  12. 2021Demoting the Lead Bias in News Summarization via Alternating Adversarial Learningcited by 10
  13. 2020Compressive Summarization with Plausibility and Salience Modelingcited by 9
  14. 2020Understanding the Extent to which Summarization Evaluation Metrics Measure the Information Quality of Summariescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 10; 2024: 13; 2025: 6; 2026: 2.

37 papers.

Sort:

By area:

  1. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  2. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 175
  3. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  4. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  5. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
  6. 2022LENS: A Learnable Evaluation Metric for Text Simplificationcited by 54
  7. 2024Attribute First, then Generate: Locally-attributable Grounded Text Generationcited by 54
  8. 2023From Sparse to Dense: GPT-4 Summarization with Chain of Density Promptingcited by 52
  9. 2024On the Challenges and Opportunities in Generative AIcited by 31
  10. 2024Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judgecited by 29
  11. 2024JuStRank: Benchmarking LLM Judges for System Rankingcited by 27
  12. 2023It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performancecited by 25
25 more
  1. 2025How to Select Datapoints for Efficient Human Evaluation of NLG Models?cited by 23
  2. 2022Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrievalcited by 20
  3. 2023Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluationscited by 18
  4. 2024Re-evaluating Automatic LLM System Ranking for Alignment with Human Preferencecited by 16
  5. 2023An Investigation of Evaluation Metrics for Automated Medical Note Generationcited by 10
  6. 2023PersonaPKT: Building Personalized Dialogue Agents via Parameter-efficient Knowledge Transfercited by 8
  7. 2023WangLab at MEDIQA-Chat 2023: Clinical Note Generation from Doctor-Patient Conversations using Large Language Modelscited by 7
  8. 2024CLINICSUM: Utilizing Language Models for Generating Clinical Summaries from Patient-Doctor Conversationscited by 7
  9. 2025LiteraryQA: Towards Effective Evaluation of Long-document Narrative QAcited by 6
  10. 2024Mitigating the Impact of Reference Quality on Evaluation of Summarization Systems with Reference-Free Metricscited by 5
  11. 2025DRAGOn: Designing RAG On Periodically Updated Corpuscited by 5
  12. 2025LLMs Do Not See Age: Assessing Demographic Bias in Automated Systematic Review Synthesiscited by 5
  13. 2026Replicating Human Motivated Reasoning Studies with LLMscited by 5
  14. 2026Slipstream: Trajectory-Grounded Compaction Validation for Long-Horizon Agentscited by 5
  15. 2022How to Find Strong Summary Coherence Measures? A Toolbox and a Comparative Study for Summary Coherence Measure Evaluationcited by 4
  16. 2023Evaluating Factual Consistency of Texts with Semantic Role Labelingcited by 4
  17. 2024Analyzing and Evaluating Correlation Measures in NLG Meta-Evaluationcited by 4
  18. 2024Contextualizing Generated Citation Textscited by 3
  19. 2023Leveraging GPT-4 for Food Effect Summarization to Enhance Product-Specific Guidance Development via Iterative Promptingcited by 2
  20. 2024A Critical Look at Meta-evaluating Summarisation Evaluation Metricscited by 2
  21. 2025Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond Englishcited by 2
  22. 2023Revisiting Automatic Question Summarization Evaluation in the Biomedical Domaincited by 1
  23. 2024Multi-Review Fusion-in-Contextcited by 1
  24. 2023Evaluating Robustness of Dialogue Summarization Models in the Presence of Naturally Occurring Variationscited by 0
  25. 2025Progress Ratio Embeddings: An Impatience Signal for Robust Length Control in Neural Text Generationcited by 0