mlsanityData to 7 Oct 2026

Transformer Quality in Linear Time

arXiv 2202.10447 ↗First posted 21 Feb 2022cs.LG cs.AI cs.CLcited by 189

Not on the map. Why?

Abstract (arXiv)

We revisit the design choices in Transformers, and propose methods to address their weaknesses in handling long sequences. First, we propose a simple layer named gated attention unit, which allows the use of a weaker single-head attention with minimal quality loss. We then propose a linear approximation method complementary to this new layer, which is accelerator-friendly and highly competitive in quality. The resulting model, named FLASH, matches the perplexity of improved Transformers over both short (512) and long (8K) context lengths, achieving training speedups of up to 4.9× on Wiki-40B and 12.1× on PG-19 for auto-regressive language modeling, and 4.8× on C4 for masked language modeling.

Key facts. Cited by 189 papers in the map (data through 7 Oct 2026). Builds on 35 of its 39 references that are in the map. First posted on arXiv 21 Feb 2022.

Builds on

35 of 39 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2016Layer Normalizationcited by 5,579
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
23 more
  1. 2020GLU Variants Improve Transformercited by 1,724
  2. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  3. 2020Reformer: The Efficient Transformercited by 1,658
  4. 2020Rethinking Attention with Performerscited by 1,584
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  6. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  7. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  8. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  9. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  10. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  11. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 717
  12. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  13. 2016Language Modeling with Gated Convolutional Networkscited by 688
  14. 2019Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecastingcited by 606
  15. 2017Convolutional Sequence to Sequence Learningcited by 550
  16. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  17. 2021Pay Attention to MLPscited by 367
  18. 2019Axial Attention in Multidimensional Transformerscited by 350
  19. 2021Random Feature Attentioncited by 350
  20. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  21. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  22. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  23. 2021Combiner: Full Attention Transformer with Sparse Computation Costcited by 64

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 25; 2023: 35; 2024: 48; 2025: 41; 2026: 40.

189 papers.

Sort:

By area:

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2022Simple Baselines for Image Restorationcited by 701
  3. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  4. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  5. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  6. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  7. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  8. 2022Long Range Language Modeling via Gated State Spacescited by 265
  9. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  10. 2024Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendationscited by 184
  11. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  12. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
38 more
  1. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
  2. 2023RTMPose: Real-Time Multi-Person Pose Estimation based on MMPosecited by 170
  3. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  4. 2022Decoupling Features in Hierarchical Propagation for Video Object Segmentationcited by 146
  5. 2024PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognitioncited by 141
  6. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  7. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  8. 2025Test-Time Training Done Rightcited by 121
  9. 2022Block-Recurrent Transformerscited by 107
  10. 2024Demystify Mamba in Vision: A Linear Attention Perspectivecited by 101
  11. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  12. 2022The Devil in Linear Transformercited by 98
  13. 2023Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Predictioncited by 92
  14. 2024The Hidden Attention of Mamba Modelscited by 88
  15. 2024In-Context Language Learning: Architectures and Algorithmscited by 86
  16. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  17. 2023Diffusion Models Without Attentioncited by 75
  18. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 73
  19. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  20. 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Productscited by 69
  21. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  22. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 63
  23. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  24. 2022MogaNet: Multi-order Gated Aggregation Networkcited by 59
  25. 2024Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Modelscited by 57
  26. 2024Mechanistic Design and Scaling of Hybrid Architecturescited by 57
  27. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 57
  28. 2022Pretraining Without Attentioncited by 51
  29. 2022Efficient Movie Scene Detection using State-Space Transformerscited by 51
  30. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 51
  31. 2024Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Lengthcited by 46
  32. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  33. 2023Toeplitz Neural Network for Sequence Modelingcited by 43
  34. 2023Replacing softmax with ReLU in Vision Transformerscited by 43
  35. 2024Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Modelscited by 41
  36. 2025Log-Linear Attentioncited by 39
  37. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
  38. 2022Efficient Long Sequence Modeling via State Space Augmented Transformercited by 37

50 most cited of 189 All 189 (JSON)