mlsanityData to 14 Oct 2026

Gating is Weighting: Understanding Gated Linear Attention through In-context Learning

arXiv 2504.04308 ↗First posted 6 Apr 2025cs.LG cs.AI cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Linear attention methods offer a compelling alternative to softmax attention due to their efficiency in recurrent decoding. Recent research has focused on enhancing standard linear attention by incorporating gating while retaining its computational benefits. Such Gated Linear Attention (GLA) architectures include competitive models such as Mamba and RWKV. In this work, we investigate the in-context learning capabilities of the GLA model and make the following contributions. We show that a multilayer GLA can implement a general class of Weighted Preconditioned Gradient Descent (WPGD) algorithms with data-dependent weights. These weights are induced by the gating mechanism and the input, enabling the model to control the contribution of individual tokens to prediction. To further understand the mechanics of this weighting, we introduce a novel data model with multitask prompts and characterize the optimization landscape of learning a WPGD algorithm. Under mild conditions, we establish the existence and uniqueness (up to scaling) of a global minimum, corresponding to a unique WPGD solution. Finally, we translate these findings to explore the optimization landscape of GLA and shed light on how gating facilitates context-aware learning and when it is provably better than vanilla linear attention.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 66 of its 76 references that are in the map. First posted on arXiv 6 Apr 2025.

Builds on

66 of 76 references.

References · 66

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  6. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  7. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  8. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  9. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  10. 2022Transformers learn in-context by gradient descentcited by 732
  11. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  12. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
38 more
  1. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  2. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  3. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  4. 2021Random Feature Attentioncited by 354
  5. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  6. 2024xLSTM: Extended Long Short-Term Memorycited by 327
  7. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  8. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  9. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 216
  10. 2024Simple linear attention language models balance the recall-throughput tradeoffcited by 208
  11. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  12. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 159
  13. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  14. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 127
  15. 2023In-Context Convergence of Transformerscited by 115
  16. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 105
  17. 2023Uncovering mesa-optimization algorithms in Transformerscited by 100
  18. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 96
  19. 2022Vision Transformers provably learn spatial structurecited by 93
  20. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  21. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  22. 2023Transformers as Support Vector Machinescited by 74
  23. 2024Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimalitycited by 74
  24. 2024Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?cited by 65
  25. 2023Max-Margin Token Selection in Attention Mechanismcited by 58
  26. 2023JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attentioncited by 52
  27. 2023On the Role of Attention in Prompt-tuningcited by 50
  28. 2023GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modelingcited by 46
  29. 2024Is Mamba Capable of In-Context Learning?cited by 45
  30. 2024Dual Operating Modes of In-Context Learningcited by 43
  31. 2024An Information-Theoretic Analysis of In-Context Learningcited by 40
  32. 2024Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chainscited by 40
  33. 2023On the Optimization and Generalization of Multi-head Attentioncited by 37
  34. 2024From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformerscited by 34
  35. 2024Mechanics of Next Token Prediction with Self-Attentioncited by 34
  36. 2024Implicit Bias and Fast Convergence Rates for Self-attentioncited by 31
  37. 2022Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformerscited by 30
  38. 2023CausalLM is not optimal for in-context learningcited by 26

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 7.

12 papers.

  1. 2026MeMix: Writing Less, Remembering More for Streaming 3D Reconstructioncited by 8
  2. 2025When and How Unlabeled Data Provably Improve In-Context Learningcited by 6
  3. 2026ZeroS: Zero-Sum Linear Attention for Efficient Transformerscited by 5
  4. 2026A Theoretical Analysis of Mamba's Training Dynamics: Filtering Relevant Features for Generalization in State Space Modelscited by 3
  5. 2025Trained Mamba Emulates Online Gradient Descent in In-Context Linear Regressioncited by 2
  6. 2025How Can Mamba Learn In Context with Outliers and Generalize Provably?cited by 2
  7. 2026Learning to Adapt: In-Context Learning Beyond Stationaritycited by 2
  8. 2025Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learningcited by 1
  9. 2026In-Context Learning Under Regime Changecited by 1
  10. 2025Hecto: Modular Sparse Experts for Adaptive and Interpretable Reasoningcited by 0
  11. 2026On the Importance of Gating: Memorization vs. In-Context Learning in State Space Modelscited by 0
  12. 2026Generalization of Transformer-Based Neural Quantum States via In-Context Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
  2. 2026MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution
  3. 2025GatedFWA: Linear Flash Windowed Attention with Gated Associative Memory
  4. 2025Kimi Linear: An Expressive, Efficient Attention Architecture
  5. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs
  6. 2026A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts
  7. 2025In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention
  8. 2025On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation