mlsanityData to 7 Oct 2026

A Comparative Study of Quality Evaluation Methods for Text Summarization

arXiv 2407.00747 ↗First posted 30 Jun 2024cs.CL cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

Evaluating text summarization has been a challenging task in natural language processing (NLP). Automatic metrics which heavily rely on reference summaries are not suitable in many situations, while human evaluation is time-consuming and labor-intensive. To bridge this gap, this paper proposes a novel method based on large language models (LLMs) for evaluating text summarization. We also conducts a comparative study on eight automatic metrics, human evaluation, and our proposed LLM-based method. Seven different types of state-of-the-art (SOTA) summarization models were evaluated. We perform extensive experiments and analysis on datasets with patent documents. Our results show that LLMs evaluation aligns closely with human evaluation, while widely-used automatic metrics such as ROUGE-2, BERTScore, and SummaC do not and also lack consistency. Based on the empirical comparison, we propose a LLM-powered framework for automatically evaluating and improving text summarization, which is beneficial and could attract wide attention among the community.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 31 of its 42 references that are in the map. First posted on arXiv 30 Jun 2024.

Builds on

31 of 42 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  12. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
19 more
  1. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  2. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  3. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  4. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  5. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  6. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  7. 2019BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarizationcited by 155
  8. 2017Faithful to the Original: Fact Aware Neural Abstractive Summarizationcited by 141
  9. 2020CTRLsum: Towards Generic Controllable Text Summarizationcited by 98
  10. 2020Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summarycited by 87
  11. 2018BLEU is Not Suitable for the Evaluation of Text Simplificationcited by 78
  12. 2018ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Taskscited by 66
  13. 2023Large Language Models are Diverse Role-Players for Summarization Evaluationcited by 62
  14. 2017Text Summarization Techniques: A Brief Surveycited by 59
  15. 2023SummIt: Iterative Text Summarization via ChatGPTcited by 53
  16. 2022BERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generationcited by 48
  17. 2021Bringing Structure into Summaries: a Faceted Summarization Dataset for Long Scientific Documentscited by 33
  18. 2022The Harvard USPTO Patent Dataset: A Large-Scale, Well-Structured, and Multi-Purpose Corpus of Patent Applicationscited by 32
  19. 2020Automatic Text Summarization of COVID-19 Medical Research Articles using BERT and GPT-2cited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 6.

13 papers.

Sort:

By area:

  1. 2025FutureGen: A RAG-based Approach to Generate the Future Work of Scientific Articlecited by 2
  2. 2025TF1-EN-3M: Three Million Synthetic Moral Fables for Training Small, Open Language Modelscited by 2
  3. 2025Helping Large Language Models Protect Themselves: An Enhanced Filtering and Summarization Systemcited by 2
  4. 2025Taming LLMs with Negative Samples: A Reference-Free Framework to Evaluate Presentation Content with Actionable Feedbackcited by 2
  5. 2026Agentic AI-Empowered Dynamic Survey Frameworkcited by 2
  6. 2026LongSumEval: Question-Answering Based Evaluation and Feedback-Driven Refinement for Long Document Summarizationcited by 1
  7. 2025An Empirical Comparison of Text Summarization: A Multi-Dimensional Evaluation of Large Language Modelscited by 0
  8. 2025Medical Interpretability and Knowledge Maps of Large Language Modelscited by 0
  9. 2025DiscoVerse: Multi-Agent Pharmaceutical Co-Scientist for Traceable Drug Discovery and Reverse Translationcited by 0
  10. 2026LLM-ReSum: A Framework for LLM Reflective Summarization through Self-Evaluationcited by 0
  11. 2026GigaChat Audio: Time-aware Large Audio Language Modelcited by 0
  12. 2026GRADRAG: Cross-Component Prompt Adaptation for Coordinated Multi-Agent RAGcited by 0
1 more
  1. 2026MIDAS: Multi-LLM Iterative Data-Adaptive Summarizationcited by 0