# LSG Attention: Extrapolation of pretrained Transformers to long sequences

arXiv: https://arxiv.org/abs/2210.15497 · HTML: https://mlsanity.com/paper/2210.15497/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 42 references that are in the map.
- First posted on arXiv 13 Oct 2022.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://mlsanity.com/paper/1906.01749/) (2019)
- [Efficient Attentions for Long Document Summarization](https://mlsanity.com/paper/2104.02112/) (2021)
- [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021)
- [LongT5: Efficient Text-To-Text Transformer for Long Sequences](https://mlsanity.com/paper/2112.07916/) (2021)
- [Pay Less Attention with Lightweight and Dynamic Convolutions](https://mlsanity.com/paper/1901.10430/) (2019)
- [BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization](https://mlsanity.com/paper/1906.03741/) (2019)
- [HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization](https://mlsanity.com/paper/1905.06566/) (2019)
- [Practical and Optimal LSH for Angular Distance](https://mlsanity.com/paper/1509.02897/) (2015)
- [MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization](https://mlsanity.com/paper/2103.06410/) (2021)
- [Star-Transformer](https://mlsanity.com/paper/1902.09113/) (2019)

## Built on by (most cited first, 16 of 16)

- [Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrieval](https://mlsanity.com/paper/2212.10526/) (2022)
- [SinkLoRA: Enhanced Efficiency and Chat Capabilities for Long-Context Large Language Models](https://mlsanity.com/paper/2406.05678/) (2024)
- [Overview of the BioLaySumm 2023 Shared Task on Lay Summarization of Biomedical Research Articles](https://mlsanity.com/paper/2309.17332/) (2023)
- [EUROPA: A Legal Multilingual Keyphrase Generation Dataset](https://mlsanity.com/paper/2403.00252/) (2024)
- [Narrative Theory-Driven LLM Methods for Automatic Story Generation and Understanding: A Survey](https://mlsanity.com/paper/2602.15851/) (2026)
- [Exploring Large Language Models and Hierarchical Frameworks for Classification of Large Unstructured Legal Documents](https://mlsanity.com/paper/2403.06872/) (2024)
- [Reducing Reasoning Costs: The Path of Optimization for Chain of Thought via Sparse Attention Mechanism](https://mlsanity.com/paper/2411.09111/) (2024)
- [CodeSSM: Towards State Space Models for Code Understanding](https://mlsanity.com/paper/2505.01475/) (2025)
- [Fovea Transformer: Efficient Long-Context Modeling with Structured Fine-to-Coarse Attention](https://mlsanity.com/paper/2311.07102/) (2023)
- [Efficient Transformer Knowledge Distillation: A Performance Review](https://mlsanity.com/paper/2311.13657/) (2023)
- [Adaptation of Biomedical and Clinical Pretrained Models to French Long Documents: A Comparative Study](https://mlsanity.com/paper/2402.16689/) (2024)
- [Multi-Scenario Reasoning: Unlocking Cognitive Autonomy in Humanoid Robots for Multimodal Understanding](https://mlsanity.com/paper/2412.20429/) (2024)
- [Rethinking Graph-Based Document Classification: Learning Data-Driven Structures Beyond Heuristic Approaches](https://mlsanity.com/paper/2508.00864/) (2025)
- [IUTEAM1 at MEDIQA-Chat 2023: Is simple fine tuning effective for multilayer summarization of clinical conversations?](https://mlsanity.com/paper/2306.04328/) (2023)
- [Wormhole Memory: A Rubik's Cube for Cross-Dialogue Retrieval](https://mlsanity.com/paper/2501.14846/) (2025)
- [Skim-Aware Contrastive Learning for Efficient Document Representation](https://mlsanity.com/paper/2512.24373/) (2025)

## Abstract (arXiv)

Transformer models achieve state-of-the-art performance on a wide range of NLP tasks. They however suffer from a prohibitive limitation due to the self-attention mechanism, inducing O(n2) complexity with regard to sequence length. To answer this limitation we introduce the LSG architecture which relies on Local, Sparse and Global attention. We show that LSG attention is fast, efficient and competitive in classification and summarization tasks on long documents. Interestingly, it can also be used to adapt existing pretrained models to efficiently extrapolate to longer sequences with no additional training. Along with the introduction of the LSG attention mechanism, we propose tools to train new models and adapt existing ones based on this mechanism.