mlsanityData to 14 Oct 2026

Gated Linear Attention Transformers with Hardware-Efficient Training

arXiv 2312.06635 ↗First posted 11 Dec 2023cs.LG cs.CLcited by 514

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Transformers with linear attention allow for efficient parallel training but can simultaneously be formulated as an RNN with 2D (matrix-valued) hidden states, thus enjoying linear-time inference complexity. However, linear attention generally underperforms ordinary softmax attention. Moreover, current implementations of linear attention lack I/O-awareness and are thus slower than highly optimized implementations of softmax attention. This work describes a hardware-efficient algorithm for linear attention that trades off memory movement against parallelizability. The resulting implementation, dubbed FLASHLINEARATTENTION, is faster than FLASHATTENTION-2 (Dao, 2023) as a standalone layer even on short sequence lengths (e.g., 1K). We then generalize this algorithm to a more expressive variant of linear attention with data-dependent gates. When used as a replacement for the standard attention layer in Transformers, the resulting gated linear attention (GLA) Transformer is found to perform competitively against the LLaMA-architecture Transformer (Touvron et al., 2023) as well recent linear-time-inference baselines such as RetNet (Sun et al., 2023a) and Mamba (Gu & Dao, 2023) on moderate-scale language modeling experiments. GLA Transformer is especially effective at length generalization, enabling a model trained on 2K to generalize to sequences longer than 20K without significant perplexity degradations. For training speed, the GLA Transformer has higher throughput than a similarly-sized Mamba model.

Key facts. Cited by 514 papers in the map (data through 14 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 79 of its 118 references that are in the map. First posted on arXiv 11 Dec 2023.

Builds on

79 of 118 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  2. 2022Simplified State Space Layers for Sequence Modeling9 Aug
  3. 2022Diagonal State Spaces are as Effective as Structured State Spaces27 Mar
  4. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 75

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2023Mistral 7Bcited by 3,164
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
38 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  2. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  3. 2017Decoupled Weight Decay Regularizationcited by 1,831
  4. 2020GLU Variants Improve Transformercited by 1,804
  5. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  6. 2020Reformer: The Efficient Transformercited by 1,668
  7. 2020Rethinking Attention with Performerscited by 1,601
  8. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  10. 2019Root Mean Square Layer Normalizationcited by 1,480
  11. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  12. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  13. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  14. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  15. 2018CoQA: A Conversational Question Answering Challengecited by 683
  16. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  17. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 608
  18. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  19. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  20. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
  21. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  22. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  23. 2021Random Feature Attentioncited by 354
  24. 2022Long Range Language Modeling via Gated State Spacescited by 265
  25. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  26. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 204
  27. 2022Transformer Quality in Linear Timecited by 191
  28. 2022Mega: Moving Average Equipped Gated Attentioncited by 179
  29. 2016Using Fast Weights to Attend to the Recent Pastcited by 170
  30. 2017Searching for Activation Functionscited by 169
  31. 2020The Hardware Lotterycited by 160
  32. 2022Liquid Structural State-Space Modelscited by 136
  33. 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 127
  34. 2022A Length-Extrapolatable Transformercited by 127
  35. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 126
  36. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 120
  37. 2023Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakescited by 111
  38. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 106

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 100; 2025: 175; 2026: 239.

514 papers.

Sort:

By area:

  1. 2024VMamba: Visual State Space Modelcited by 969
  2. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  3. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  4. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 355
  5. 2024Titans: Learning to Memorize at Test Timecited by 328
  6. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  7. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  8. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  9. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  10. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 174
  11. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  12. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
38 more
  1. 2026Kimi K3: Open Frontier Intelligencecited by 138
  2. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 132
  3. 2025Test-Time Training Done Rightcited by 130
  4. 2025TTT3R: 3D Reconstruction as Test-Time Trainingcited by 129
  5. 2025Fast Video Generation with Sliding Tile Attentioncited by 128
  6. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 127
  7. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 122
  8. 2024Transformers are Multi-State RNNscited by 120
  9. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 105
  10. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 101
  11. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 99
  12. 2024In-Context Language Learning: Architectures and Algorithmscited by 89
  13. 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 89
  14. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 86
  15. 2024ZigMa: A DiT-style Zigzag Mamba Diffusion Modelcited by 84
  16. 2025A Survey on Latent Reasoningcited by 81
  17. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 76
  18. 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Productscited by 75
  19. 2024Theoretical Foundations of Deep Selective State-Space Modelscited by 74
  20. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 74
  21. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 72
  22. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 69
  23. 2024Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Modelscited by 67
  24. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 66
  25. 2026Helios: Real Real-Time Long Video Generation Modelcited by 65
  26. 2026Attention Residualscited by 65
  27. 2025It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimizationcited by 64
  28. 2025Long-Context State-Space Video World Modelscited by 64
  29. 2025End-to-End Test-Time Training for Long Contextcited by 62
  30. 2024MambaByte: Token-free Selective State Space Modelcited by 59
  31. 2024Mechanistic Design and Scaling of Hybrid Architecturescited by 59
  32. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  33. 2024Longhorn: State Space Models are Amortized Online Learnerscited by 59
  34. 2024DiG: Scalable and Efficient Diffusion Models with Gated Linear Attentioncited by 56
  35. 2026SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformercited by 55
  36. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 53
  37. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 52
  38. 2024Vision-LSTM: xLSTM as Generic Vision Backbonecited by 51

50 most cited of 514 All 514 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024FlashMask: Efficient and Rich Mask Extension of FlashAttention
  2. 2025Rethinking Transformer Connectivity: TLinFormer, A Path to Exact, Full Context-Aware Linear Attention
  3. 2023AttMEMO : Accelerating Transformers with Memoization on Big Memory Systems
  4. 2026HLA: Hadamard Linear Attention
  5. 2023Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers
  6. 2026Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers
  7. 2026Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beat
  8. 2024Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences