mlsanityData to 14 Oct 2026

LSG Attention: Extrapolation of pretrained Transformers to long sequences

arXiv 2210.15497 ↗First posted 13 Oct 2022cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Transformer models achieve state-of-the-art performance on a wide range of NLP tasks. They however suffer from a prohibitive limitation due to the self-attention mechanism, inducing O(n2) complexity with regard to sequence length. To answer this limitation we introduce the LSG architecture which relies on Local, Sparse and Global attention. We show that LSG attention is fast, efficient and competitive in classification and summarization tasks on long documents. Interestingly, it can also be used to adapt existing pretrained models to efficiently extrapolate to longer sequences with no additional training. Along with the introduction of the LSG attention mechanism, we propose tools to train new models and adapt existing ones based on this mechanism.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 36 of its 42 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

36 of 42 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2020Longformer: The Long-Document Transformercited by 3,413
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  11. 2020Reformer: The Efficient Transformercited by 1,668
  12. 2020Rethinking Attention with Performerscited by 1,601
24 more
  1. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  2. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  3. 2018Self-Attention with Relative Position Representationscited by 1,120
  4. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  5. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  6. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  7. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  8. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  9. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  10. 2021Efficient Attentions for Long Document Summarizationcited by 345
  11. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 310
  12. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  13. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  14. 2019BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarizationcited by 156
  15. 2019HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarizationcited by 121
  16. 2015Practical and Optimal LSH for Angular Distancecited by 106
  17. 2021MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarizationcited by 104
  18. 2019Star-Transformercited by 94
  19. 2017Monotonic Chunkwise Attentioncited by 83
  20. 2021PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarizationcited by 62
  21. 2020Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translationcited by 43
  22. 2021Hierarchical Learning for Generation with Long Source Sequencescited by 32
  23. 2018Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modelingcited by 12
  24. 2017Efficient Attention using a Fixed-Size Memory Representationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 6; 2025: 4; 2026: 1.

16 papers.

Sort:

By area:

  1. 2022Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrievalcited by 20
  2. 2024SinkLoRA: Enhanced Efficiency and Chat Capabilities for Long-Context Large Language Modelscited by 9
  3. 2023Overview of the BioLaySumm 2023 Shared Task on Lay Summarization of Biomedical Research Articlescited by 6
  4. 2024EUROPA: A Legal Multilingual Keyphrase Generation Datasetcited by 5
  5. 2026Narrative Theory-Driven LLM Methods for Automatic Story Generation and Understanding: A Surveycited by 5
  6. 2024Exploring Large Language Models and Hierarchical Frameworks for Classification of Large Unstructured Legal Documentscited by 4
  7. 2024Reducing Reasoning Costs: The Path of Optimization for Chain of Thought via Sparse Attention Mechanismcited by 4
  8. 2025CodeSSM: Towards State Space Models for Code Understandingcited by 3
  9. 2023Fovea Transformer: Efficient Long-Context Modeling with Structured Fine-to-Coarse Attentioncited by 2
  10. 2023Efficient Transformer Knowledge Distillation: A Performance Reviewcited by 2
  11. 2024Adaptation of Biomedical and Clinical Pretrained Models to French Long Documents: A Comparative Studycited by 2
  12. 2024Multi-Scenario Reasoning: Unlocking Cognitive Autonomy in Humanoid Robots for Multimodal Understandingcited by 1
4 more
  1. 2025Rethinking Graph-Based Document Classification: Learning Data-Driven Structures Beyond Heuristic Approachescited by 1
  2. 2023IUTEAM1 at MEDIQA-Chat 2023: Is simple fine tuning effective for multilayer summarization of clinical conversations?cited by 0
  3. 2025Wormhole Memory: A Rubik's Cube for Cross-Dialogue Retrievalcited by 0
  4. 2025Skim-Aware Contrastive Learning for Efficient Document Representationcited by 0