mlsanityData to 7 Oct 2026

Efficient Attentions for Long Document Summarization

arXiv 2104.02112 ↗First posted 5 Apr 2021cs.CLcited by 342

Not on the map. Why?

Cited by 59 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The quadratic computational and memory complexities of large Transformers have limited their scalability for long document summarization. In this paper, we propose Hepos, a novel efficient encoder-decoder attention with head-wise positional strides to effectively pinpoint salient information from the source. We further conduct a systematic study of existing efficient self-attentions. Combined with Hepos, we are able to process ten times more tokens than existing models that use full attentions. For evaluation, we present a new dataset, GovReport, with significantly longer documents and summaries. Results show that our models produce significantly higher ROUGE scores than competitive comparisons, including new state-of-the-art results on PubMed. Human evaluation also shows that our models generate more informative summaries with fewer unfaithful errors.

Key facts. Cited by 342 papers in the map (data through 7 Oct 2026). Builds on 38 of its 62 references that are in the map. First posted on arXiv 5 Apr 2021.

Builds on

38 of 62 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  9. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  10. 2020Reformer: The Efficient Transformercited by 1,658
  11. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  12. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
26 more
  1. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  2. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  3. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  4. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  5. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  6. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  7. 2020Efficient Transformers: A Surveycited by 784
  8. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  9. 2019Text Summarization with Pretrained Encoderscited by 553
  10. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  11. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  12. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  13. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 292
  14. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  15. 2020Sparse Sinkhorn Attentioncited by 250
  16. 2018Bottom-Up Abstractive Summarizationcited by 177
  17. 2019BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarizationcited by 155
  18. 2019Adaptive Attention Span in Transformerscited by 148
  19. 2018Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networkscited by 129
  20. 2019BillSum: A Corpus for Automatic Summarization of US Legislationcited by 117
  21. 2019On Extractive and Abstractive Neural Document Summarization with Transformer Language Modelscited by 83
  22. 2019Question Answering as an Automatic Evaluation Metric for News Article Summarizationcited by 63
  23. 2019Extractive Summarization of Long Documents by Combining Global and Local Contextcited by 59
  24. 2018Deep Communicating Agents for Abstractive Summarizationcited by 50
  25. 2020SEAL: Segment-wise Extractive-Abstractive Long-form Text Summarizationcited by 15
  26. 2020Discourse-Aware Unsupervised Summarization of Long Scientific Documentscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 37; 2023: 45; 2024: 86; 2025: 83; 2026: 77.

342 papers.

Sort:

By area:

  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  3. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  4. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  5. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  7. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  8. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  9. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  10. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  11. 2022UL2: Unifying Language Learning Paradigmscited by 292
  12. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
38 more
  1. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  2. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  3. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  4. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  5. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  6. 2023Universal Self-Consistency for Large Language Model Generationcited by 198
  7. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  8. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  9. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  10. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 142
  11. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 129
  12. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  13. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  14. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  15. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  16. 2024Make Your LLM Fully Utilize the Contextcited by 118
  17. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
  18. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  19. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  20. 2024Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuningcited by 85
  21. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  22. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  23. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  24. 2023MeetingBank: A Benchmark Dataset for Meeting Summarizationcited by 81
  25. 2021SummScreen: A Dataset for Abstractive Screenplay Summarizationcited by 80
  26. 2024LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QAcited by 78
  27. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  28. 2021DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarizationcited by 72
  29. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  30. 2022Efficient Long-Text Understanding with Short-Text Modelscited by 71
  31. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  32. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 68
  33. 2023RewriteLM: An Instruction-Tuned Large Language Model for Text Rewritingcited by 67
  34. 2021Summ^N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documentscited by 64
  35. 2022ECTSum: A New Benchmark Dataset For Bullet Point Summarization of Long Earnings Call Transcriptscited by 59
  36. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 57
  37. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
  38. 2023InstructRetro: Instruction Tuning post Retrieval-Augmented Pretrainingcited by 53

50 most cited of 342 All 342 (JSON)