mlsanityData to 7 Oct 2026

Neural Text Summarization: A Critical Evaluation

arXiv 1908.08960 ↗First posted 23 Aug 2019cs.CLcited by 189

Not on the map. Why?

Abstract (arXiv)

Text summarization aims at compressing long documents into a shorter form that conveys the most important parts of the original document. Despite increased interest in the community and notable research effort, progress on benchmark datasets has stagnated. We critically evaluate key ingredients of the current research setup: datasets, evaluation metrics, and models, and highlight three primary shortcomings: 1) automatically collected datasets leave the task underconstrained and may contain noise detrimental to training and evaluation, 2) current evaluation protocol is weakly correlated with human judgment and does not account for important characteristics such as factual correctness, 3) models overfit to layout biases of current datasets and offer limited diversity in their outputs.

Key facts. Cited by 189 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 1; 2021: 49; 2022: 47; 2023: 38; 2024: 26; 2025: 18; 2026: 9.

189 papers.

Sort:

By area:

  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  3. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  4. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  5. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  6. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 266
  7. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  8. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  9. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  10. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  11. 2021DialogSum: A Real-Life Scenario Dialogue Summarization Datasetcited by 181
  12. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
38 more
  1. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  2. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  3. 2021Entity-level Factual Consistency of Abstractive Text Summarizationcited by 118
  4. 2021Robustness Gym: Unifying the NLP Evaluation Landscapecited by 107
  5. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  6. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 103
  7. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  8. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 82
  9. 2023Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Methodcited by 76
  10. 2021Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selectioncited by 75
  11. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  12. 2023Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarizationcited by 69
  13. 2023Do Androids Know They're Only Dreaming of Electric Sheep?cited by 67
  14. 2024TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarizationcited by 66
  15. 2025Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decodingcited by 62
  16. 2022SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarizationcited by 61
  17. 2021Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generationcited by 60
  18. 2021IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generationcited by 57
  19. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 56
  20. 2021Improving Factual Consistency of Abstractive Summarization via Question Answeringcited by 55
  21. 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languagescited by 55
  22. 2021Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarizationcited by 50
  23. 2022SQuALITY: Building a Long-Document Summarization Dataset the Hard Waycited by 50
  24. 2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 49
  25. 2022FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarizationcited by 49
  26. 2021D2S: Document-to-Slide Generation Via Query-Based Text Summarizationcited by 48
  27. 2021"Let Your Characters Tell Their Story": A Dataset for Character-Centric Narrative Understandingcited by 48
  28. 2022Beyond Text Generation: Supporting Writers with Continuous Automatic Text Summariescited by 47
  29. 2021Exploring Neural Models for Query-Focused Summarizationcited by 44
  30. 2022Prompted Opinion Summarization with GPT-3.5cited by 44
  31. 2021Finding a Balanced Degree of Automation for Summary Evaluationcited by 43
  32. 2021Transparent Human Evaluation for Image Captioningcited by 40
  33. 2022Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infillingcited by 38
  34. 2021ConvoSumm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Miningcited by 37
  35. 2021Controllable Neural Dialogue Summarization with Personal Named Entity Planningcited by 36
  36. 2021What's in a Summary? Laying the Groundwork for Advances in Hospital-Course Summarizationcited by 35
  37. 2022DiscoScore: Evaluating Text Generation with BERT and Discourse Coherencecited by 35
  38. 2023Beyond the Chat: Executable and Verifiable Text-Editing with LLMscited by 35

50 most cited of 189 All 189 (JSON)