mlsanityData to 7 Oct 2026

A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods

arXiv 2104.00054 ↗First posted 31 Mar 2021cs.CLcited by 71

Not on the map. Why?

Abstract (arXiv)

The quality of a summarization evaluation metric is quantified by calculating the correlation between its scores and human annotations across a large number of summaries. Currently, it is unclear how precise these correlation estimates are, nor whether differences between two metrics' correlations reflect a true difference or if it is due to mere chance. In this work, we address these two problems by proposing methods for calculating confidence intervals and running hypothesis tests for correlations using two resampling methods, bootstrapping and permutation. After evaluating which of the proposed methods is most appropriate for summarization through two simulation experiments, we analyze the results of applying these methods to several different automatic evaluation metrics across three sets of human annotations. We find that the confidence intervals are rather wide, demonstrating high uncertainty in the reliability of automatic metrics. Further, although many metrics fail to show statistical improvements over ROUGE, two recent works, QAEval and BERTScore, do in some evaluation settings.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Builds on 10 of its 49 references that are in the map. First posted on arXiv 31 Mar 2021.

Builds on

10 of 49 references.

References · 10

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  2. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  3. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  4. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  5. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
  6. 2020Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summarycited by 87
  7. 2019Crowdsourcing Lightweight Pyramids for Manual Summary Evaluationcited by 39
  8. 2017Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasetscited by 32
  9. 2020SacreROUGE: An Open-Source Library for Using and Developing Summarization Evaluation Metricscited by 18
  10. 2019Not All Claims are Created Equal: Choosing the Right Statistical Approach to Assess Hypothesescited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 14; 2023: 15; 2024: 17; 2025: 14; 2026: 7.

71 papers.

Sort:

By area:

  1. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
  2. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  3. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 144
  4. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  5. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  6. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  7. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  8. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Modelscited by 87
  9. 2022Mutual Information Alleviates Hallucinations in Abstractive Summarizationcited by 67
  10. 2023Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translationcited by 57
  11. 2022Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric Into a Document-Level Metriccited by 56
  12. 2022LENS: A Learnable Evaluation Metric for Text Simplificationcited by 54
38 more
  1. 2022On the Limitations of Reference-Free Evaluations of Generated Textcited by 52
  2. 2023Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theorycited by 41
  3. 2021Automatic Text Evaluation through the Lens of Wasserstein Barycenterscited by 38
  4. 2022Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metricscited by 37
  5. 2024Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judgecited by 29
  6. 2024Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generationcited by 25
  7. 2024Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracycited by 25
  8. 2023AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languagescited by 24
  9. 2023The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metricscited by 23
  10. 2025How to Select Datapoints for Efficient Human Evaluation of NLG Models?cited by 23
  11. 2024MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluatorscited by 22
  12. 2024Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!cited by 21
  13. 2024URIEL+: Enhancing Linguistic Inclusion and Usability in a Typological and Multilingual Knowledge Basecited by 21
  14. 2025SedarEval: Automated Evaluation using Self-Adaptive Rubricscited by 19
  15. 2024Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translationcited by 16
  16. 2023The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metricscited by 15
  17. 2024PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluationcited by 15
  18. 2025Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI's Latest Open Source Modelscited by 11
  19. 2025Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progresscited by 10
  20. 2024Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?cited by 9
  21. 2025Remedy: Learning Machine Translation Evaluation from Human Preferences with Reward Modelingcited by 9
  22. 2024SportsMetrics: Blending Text and Numerical Data to Understand Information Fusion in LLMscited by 8
  23. 2024QAPyramid: Fine-grained Evaluation of Content Selection for Text Summarizationcited by 8
  24. 2022NMTScore: A Multilingual Analysis of Translation-based Text Similarity Measurescited by 6
  25. 2022Exact Paired-Permutation Testing for Structured Test Statisticscited by 6
  26. 2022HaRiM^+: Evaluating Summary Quality with Hallucination Riskcited by 6
  27. 2023Interpretable Automatic Fine-grained Inconsistency Detection in Text Summarizationcited by 6
  28. 2024How Did We Get Here? Summarizing Conversation Dynamicscited by 6
  29. 2025Estimating Machine Translation Difficultycited by 6
  30. 2025LiteraryQA: Towards Effective Evaluation of Long-document Narrative QAcited by 6
  31. 2022How to Find Strong Summary Coherence Measures? A Toolbox and a Comparative Study for Summary Coherence Measure Evaluationcited by 4
  32. 2023KPEval: Towards Fine-Grained Semantic-Based Keyphrase Evaluationcited by 4
  33. 2023Evaluating Factual Consistency of Texts with Semantic Role Labelingcited by 4
  34. 2024Analyzing and Evaluating Correlation Measures in NLG Meta-Evaluationcited by 4
  35. 2023Grounded Intuition of GPT-Vision's Abilities with Scientific Imagescited by 3
  36. 2024Revisiting Meta-evaluation for Grammatical Error Correctioncited by 3
  37. 2025CAPO: Confidence Aware Preference Optimization Learning for Multilingual Preferencescited by 3
  38. 2022Benchmarking Answer Verification Methods for Question Answering-Based Summarization Evaluation Metricscited by 2

50 most cited of 71 All 71 (JSON)