mlsanityData to 7 Oct 2026

Adaptive Attention Span in Transformers

arXiv 1905.07799 ↗First posted 19 May 2019cs.LG stat.MLcited by 148

Not on the map. Why?

Abstract (arXiv)

We propose a novel self-attention mechanism that can learn its optimal attention span. This allows us to extend significantly the maximum context size used in Transformer, while maintaining control over their memory footprint and computational time. We show the effectiveness of our approach on the task of character level language modeling, where we achieve state-of-the-art performances on text8 and enwiki8 by using a maximum context of 8k characters.

Key facts. Cited by 148 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 42; 2022: 30; 2023: 27; 2024: 18; 2025: 20; 2026: 11.

148 papers.

Sort:

By area:

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  4. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  5. 2022Efficiently Scaling Transformer Inferencecited by 529
  6. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  7. 2021A Survey of Transformerscited by 399
  8. 2021Random Feature Attentioncited by 350
  9. 2021Efficient Attentions for Long Document Summarizationcited by 342
  10. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  11. 2021Efficient Training of Audio Transformers with Patchoutcited by 222
  12. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
38 more
  1. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  2. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  3. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Modelscited by 177
  4. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 128
  5. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  6. 2021Lawformer: A Pre-trained Language Model for Chinese Legal Long Documentscited by 115
  7. 2022Training Language Models with Memory Augmentationcited by 112
  8. 2021An Attention Free Transformercited by 111
  9. 2021Long-Short Transformer: Efficient Transformers for Language and Visioncited by 102
  10. 2021Hierarchical Transformers Are More Efficient Language Modelscited by 99
  11. 2021Finetuning Pretrained Transformers into RNNscited by 94
  12. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  13. 2021Sparse is Enough in Scaling Transformerscited by 86
  14. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  15. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 77
  16. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  17. 2021Making Transformers Solve Compositional Taskscited by 72
  18. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  19. 2023Bayesian Flow Networkscited by 62
  20. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  21. 2024FouriScale: A Frequency Perspective on Training-Free High-Resolution Image Synthesiscited by 58
  22. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 51
  23. 2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 50
  24. 2024Text Prompt with Normality Guidance for Weakly Supervised Video Anomaly Detectioncited by 49
  25. 2022Efficient Long Sequence Modeling via State Space Augmented Transformercited by 37
  26. 2021When Attention Meets Fast Recurrence: Training Language Models with Reduced Computecited by 34
  27. 2021Not All Memories are Created Equal: Learning to Forget by Expiringcited by 32
  28. 2021Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systemscited by 29
  29. 2021Mask Attention Networks: Rethinking and Strengthen Transformercited by 28
  30. 2021GNN-LM: Language Modeling based on Global Contexts via GNNcited by 27
  31. 2022Efficient Self-supervised Vision Pretraining with Local Masked Reconstructioncited by 27
  32. 2023Transformer-VQ: Linear-Time Transformers via Vector Quantizationcited by 26
  33. 2022Learning strides in convolutional neural networkscited by 25
  34. 2022Towards Lightweight Transformer via Group-wise Transformation for Vision-and-Language Taskscited by 21
  35. 2023HRVQA: A Visual Question Answering Benchmark for High-Resolution Aerial Imagescited by 21
  36. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budgetcited by 21
  37. 2024Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Model Enhancementcited by 21
  38. 2021Evolving Attention with Residual Convolutionscited by 20

50 most cited of 148 All 148 (JSON)