mlsanityData to 7 Oct 2026

A Critical Look at Meta-evaluating Summarisation Evaluation Metrics

arXiv 2409.19507 ↗First posted 29 Sep 2024cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Effective summarisation evaluation metrics enable researchers and practitioners to compare different summarisation systems efficiently. Estimating the effectiveness of an automatic evaluation metric, termed meta-evaluation, is a critically important research question. In this position paper, we review recent meta-evaluation practices for summarisation evaluation metrics and find that (1) evaluation metrics are primarily meta-evaluated on datasets consisting of examples from news summarisation datasets, and (2) there has been a noticeable shift in research focus towards evaluating the faithfulness of generated summaries. We argue that the time is ripe to build more diverse benchmarks that enable the development of more robust evaluation metrics and analyze the generalization ability of existing evaluation metrics. In addition, we call for research focusing on user-centric quality dimensions that consider the generated summary's communicative goal and the role of summarisation in the workflow.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 55 of its 80 references that are in the map. First posted on arXiv 29 Sep 2024.

Builds on

55 of 80 references.

References · 55

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2018Datasheets for Datasetscited by 1,371
  7. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  8. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  9. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  10. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  11. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  12. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
38 more
  1. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  2. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  3. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  4. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  5. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  6. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  7. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  8. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  9. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  10. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  11. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  12. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  13. 2024Fine-grained Hallucination Detection and Editing for Language Modelscited by 146
  14. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
  15. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  16. 2020With Little Power Comes Great Responsibilitycited by 138
  17. 2020MLSUM: The Multilingual Summarization Corpuscited by 121
  18. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 103
  19. 2019Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reportscited by 96
  20. 2022Evaluating the Factual Consistency of Large Language Models Through News Summarizationcited by 90
  21. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
  22. 2020Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summarycited by 87
  23. 2020GO FIGURE: A Meta Evaluation of Factuality in Summarizationcited by 75
  24. 2021A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methodscited by 71
  25. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarizationcited by 70
  26. 2018RankME: Reliable Human Ratings for Natural Language Generationcited by 67
  27. 2020What Have We Achieved on Text Summarization?cited by 65
  28. 2018The price of debiasing automatic metrics in natural language evaluationcited by 60
  29. 2022Making Science Simple: Corpora for the Lay Summarisation of Scientific Literaturecited by 54
  30. 2022On the Limitations of Reference-Free Evaluations of Generated Textcited by 52
  31. 2022Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularitiescited by 47
  32. 2022mFACE: Multilingual Summarization with Factual Consistency Evaluationcited by 42
  33. 2023SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluationcited by 42
  34. 2022Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metricscited by 37
  35. 2022How Far are We from Robust Long Abstractive Summarization?cited by 25
  36. 2020FFCI: A Framework for Interpretable Automatic Evaluation of Summarizationcited by 24
  37. 2021GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generationcited by 20
  38. 2024Overview of the BioLaySumm 2024 Shared Task on the Lay Summarization of Biomedical Research Articlescited by 18

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2026MATCHA: Matching Text via Contrastive Semantic Alignmentcited by 1
  2. 2025Making Knowledge Accessible: Divergent Readability-Accuracy Strategies of Mistral and QWen in Biomedical Text Simplificationcited by 0