# Leveraging Locality in Abstractive Text Summarization

arXiv: https://arxiv.org/abs/2205.12476 · HTML: https://mlsanity.com/paper/2205.12476/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 72 references that are in the map.
- First posted on arXiv 25 May 2022.

## Builds on (most cited first, 30 of 48)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://mlsanity.com/paper/1906.01749/) (2019)
- [A Neural Attention Model for Abstractive Sentence Summarization](https://mlsanity.com/paper/1509.00685/) (2015)
- [Random Feature Attention](https://mlsanity.com/paper/2103.02143/) (2021)
- [Efficient Attentions for Long Document Summarization](https://mlsanity.com/paper/2104.02112/) (2021)
- [Generating Wikipedia by Summarizing Long Sequences](https://mlsanity.com/paper/1801.10198/) (2018)
- [Sparse Sinkhorn Attention](https://mlsanity.com/paper/2002.11296/) (2020)
- [LongT5: Efficient Text-To-Text Transformer for Long Sequences](https://mlsanity.com/paper/2112.07916/) (2021)
- [Blockwise Self-Attention for Long Document Understanding](https://mlsanity.com/paper/1911.02972/) (2019)
- [Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting](https://mlsanity.com/paper/1805.11080/) (2018)
- [Hierarchical Transformers for Multi-Document Summarization](https://mlsanity.com/paper/1905.13164/) (2019)

## Built on by (most cited first, 9 of 9)

- [A Survey on Long Text Modeling with Transformers](https://mlsanity.com/paper/2302.14502/) (2023)
- [Adapting Pretrained Text-to-Text Models for Long Text Sequences](https://mlsanity.com/paper/2209.10052/) (2022)
- [Chunk, Align, Select: A Simple Long-sequence Processing Method for Transformers](https://mlsanity.com/paper/2308.13191/) (2023)
- [End-to-End Long Document Summarization using Gradient Caching](https://mlsanity.com/paper/2501.01805/) (2025)
- [A Hierarchical Encoding-Decoding Scheme for Abstractive Multi-document Summarization](https://mlsanity.com/paper/2305.08503/) (2023)
- [AWESOME: GPU Memory-constrained Long Document Summarization using Memory Mechanism and Global Salient Content](https://mlsanity.com/paper/2305.14806/) (2023)
- [Abstractive Text Summarization Using the BRIO Training Paradigm](https://mlsanity.com/paper/2305.13696/) (2023)
- [Equipping Transformer with Random-Access Reading for Long-Context Understanding](https://mlsanity.com/paper/2405.13216/) (2024)
- [JupOtter: Cell-Level Bug Detection in Jupyter Notebooks](https://mlsanity.com/paper/2606.23877/) (2026)

## Abstract (arXiv)

Neural attention models have achieved significant improvements on many natural language processing tasks. However, the quadratic memory complexity of the self-attention module with respect to the input length hinders their applications in long text summarization. Instead of designing more efficient attention modules, we approach this problem by investigating if models with a restricted context can have competitive performance compared with the memory-efficient attention models that maintain a global context by treating the input as a single sequence. Our model is applied to individual pages which contain parts of inputs grouped by the principle of locality during both encoding and decoding. We empirically investigated three kinds of locality in text summarization at different levels of granularity, ranging from sentences to documents. Our experimental results show that our model has a better performance compared with strong baselines with efficient attention modules, and our analysis provides further insights into our locality-aware modeling strategy.