mlsanityData to 7 Oct 2026

How Far are We from Robust Long Abstractive Summarization?

arXiv 2210.16732 ↗First posted 30 Oct 2022cs.CLcited by 25

Not on the map. Why?

Abstract (arXiv)

Abstractive summarization has made tremendous progress in recent years. In this work, we perform fine-grained human annotations to evaluate long document abstractive summarization systems (i.e., models and metrics) with the aim of implementing them to generate reliable summaries. For long document abstractive models, we show that the constant strive for state-of-the-art ROUGE results can lead us to generate more relevant summaries but not factual ones. For long document evaluation metrics, human evaluation results show that ROUGE remains the best at evaluating the relevancy of a summary. It also reveals important limitations of factuality metrics in detecting different types of factual errors and the reasons behind the effectiveness of BARTScore. We then suggest promising directions in the endeavor of developing factual consistency metrics. Finally, we release our annotated long document dataset with the hope that it can contribute to the development of metrics across a broader range of summarization settings.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 67 of its 82 references that are in the map. First posted on arXiv 30 Oct 2022.

Builds on

67 of 82 references.

References · 67

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  11. 2020Reformer: The Efficient Transformercited by 1,658
  12. 2020Rethinking Attention with Performerscited by 1,584
38 more
  1. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  2. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  3. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  4. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  5. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  6. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  7. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  8. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  9. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  10. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  11. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  12. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  13. 2018QuAC : Question Answering in Contextcited by 379
  14. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  15. 2021Efficient Attentions for Long Document Summarizationcited by 342
  16. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 292
  17. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  18. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  19. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  20. 2018WikiHow: A Large Scale Text Summarization Datasetcited by 203
  21. 2019Neural Text Summarization: A Critical Evaluationcited by 189
  22. 2019BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarizationcited by 155
  23. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
  24. 2017Faithful to the Original: Fact Aware Neural Abstractive Summarizationcited by 141
  25. 2021Annotating and Modeling Fine-grained Factuality in Summarizationcited by 131
  26. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  27. 2019BillSum: A Corpus for Automatic Summarization of US Legislationcited by 117
  28. 2019Assessing The Factual Accuracy of Generated Textcited by 103
  29. 2018Abstractive Summarization of Reddit Posts with Multi-level Memory Networkscited by 102
  30. 2019Do Massively Pretrained Language Models Make Better Storytellers?cited by 90
  31. 2019On Extractive and Abstractive Neural Document Summarization with Transformer Language Modelscited by 83
  32. 2020GO FIGURE: A Meta Evaluation of Factuality in Summarizationcited by 75
  33. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  34. 2015Better Summarization Evaluation with Word Embeddings for ROUGEcited by 66
  35. 2018ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Taskscited by 66
  36. 2021Summ^N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documentscited by 64
  37. 2018The price of debiasing automatic metrics in natural language evaluationcited by 60
  38. 2019Extractive Summarization of Long Documents by Combining Global and Local Contextcited by 59

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 8; 2025: 6; 2026: 3.

25 papers.

Sort:

By area:

  1. 2023A Survey on Graph Neural Networks for Time Series: Forecasting, Classification, Imputation, and Anomaly Detectioncited by 145
  2. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
  3. 2025Measuring What Makes You Unique: Difference-Aware User Modeling for Enhancing LLM Personalizationcited by 44
  4. 2023Less is More for Long Document Summary Evaluation by LLMscited by 28
  5. 2025ExPerT: Effective and Explainable Evaluation of Personalized Long-Form Text Generationcited by 26
  6. 2024Large Language Models in Drug Discovery and Development: From Disease Mechanisms to Clinical Trialscited by 24
  7. 2024Fine-Grained Natural Language Inference Based Faithfulness Evaluation for Diverse Summarisation Taskscited by 23
  8. 2024On Positional Bias of Faithfulness for Long-form Summarizationcited by 22
  9. 2024Graph Spatiotemporal Process for Multivariate Time Series Anomaly Detection with Missing Valuescited by 10
  10. 2024A dataset and benchmark for hospital course summarization with adapted large language modelscited by 10
  11. 2023Enhancing Biomedical Lay Summarisation with External Knowledge Graphscited by 8
  12. 2025Discourse-Driven Evaluation: Unveiling Factual Inconsistency in Long Document Summarizationcited by 8
13 more
  1. 2023Overview of the BioLaySumm 2023 Shared Task on Lay Summarization of Biomedical Research Articlescited by 6
  2. 2023Hybrid Long Document Summarization using C2F-FAR and ChatGPT: A Practical Studycited by 5
  3. 2024Identifying Factual Inconsistencies in Summaries: Grounding LLM Inference via Task Taxonomycited by 5
  4. 2024Mitigating the Impact of Reference Quality on Evaluation of Summarization Systems with Reference-Free Metricscited by 5
  5. 2025Unstructured Evidence Attribution for Long Context Query Focused Summarizationcited by 5
  6. 2025Extract, Match, and Score: An Evaluation Paradigm for Long Question-context-answer Triplets in Financial Analysiscited by 3
  7. 2024A Critical Look at Meta-evaluating Summarisation Evaluation Metricscited by 2
  8. 2022RISE: Leveraging Retrieval Techniques for Summarization Evaluationcited by 1
  9. 2023RTSUM: Relation Triple-based Interpretable Summarization with Multi-level Salience Visualizationcited by 1
  10. 2026LongSumEval: Question-Answering Based Evaluation and Feedback-Driven Refinement for Long Document Summarizationcited by 1
  11. 2025Summarization of Multimodal Presentations with Vision-Language Models: Study of the Effect of Modalities and Structurecited by 0
  12. 2026LLM-ReSum: A Framework for LLM Reflective Summarization through Self-Evaluationcited by 0
  13. 2026Averaging Bias: Human Faithfulness Annotations are not Locally Faithfulcited by 0