mlsanityData to 7 Oct 2026

Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free

arXiv 2505.06708 ↗First posted 10 May 2025cs.CLcited by 266

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Gating mechanisms have been widely utilized, from early models like LSTMs and Highway Networks to recent state space models, linear attention, and also softmax attention. Yet, existing literature rarely examines the specific effects of gating. In this work, we conduct comprehensive experiments to systematically investigate gating-augmented softmax attention variants. Specifically, we perform a comprehensive comparison over 30 variants of 15B Mixture-of-Experts (MoE) models and 1.7B dense models trained on a 3.5 trillion token dataset. Our central finding is that a simple modification-applying a head-specific sigmoid gate after the Scaled Dot-Product Attention (SDPA)-consistently improves performance. This modification also enhances training stability, tolerates larger learning rates, and improves scaling properties. By comparing various gating positions and computational variants, we attribute this effectiveness to two key factors: (1) introducing non-linearity upon the low-rank mapping in the softmax attention, and (2) applying query-dependent sparse gating scores to modulate the SDPA output. Notably, we find this sparse gating mechanism mitigates 'attention sink' and enhances long-context extrapolation performance, and we also release related hrefhttps://github.com/qiuzh20/gatedattentioncodes and hrefhttps://huggingface.co/QwQZh/gatedattentionmodels to facilitate future research.

Key facts. Cited by 266 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 68 of its 77 references that are in the map. First posted on arXiv 10 May 2025.

Builds on

68 of 77 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025MiniMax-01: Scaling Foundation Models with Lightning Attention14 Jan
  2. 2024Qwen2.5 Technical Report19 Dec
  3. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  4. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb

Other references · 64

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  11. 2020GLU Variants Improve Transformercited by 1,724
  12. 2019Root Mean Square Layer Normalizationcited by 1,425
38 more
  1. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  2. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  4. 2022In-context Learning and Induction Headscited by 930
  5. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  6. 2023Vision Transformers Need Registerscited by 844
  7. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  8. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  9. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  10. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  11. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  12. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  13. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  14. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  15. 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
  16. 2019Attention on Attention for Image Captioningcited by 289
  17. 2024Massive Activations in Large Language Modelscited by 277
  18. 2018An Empirical Model of Large-Batch Trainingcited by 268
  19. 2015Highway Networkscited by 257
  20. 2017Gate-Variants of Gated Recurrent Unit (GRU) Neural Networkscited by 239
  21. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  22. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  23. 2022Transformer Quality in Linear Timecited by 189
  24. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  25. 2018GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphscited by 164
  26. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
  27. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  28. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 70
  29. 2024Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibrationcited by 70
  30. 2024Theory, Analysis, and Best Practices for Sigmoid Self-Attentioncited by 62
  31. 2019Explicit Sparse Transformer: Concentrated Attention Through Explicit Selectioncited by 61
  32. 2020Talking-Heads Attentioncited by 57
  33. 2024Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Modelscited by 57
  34. 2024MoEUT: Mixture-of-Experts Universal Transformerscited by 55
  35. 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Modelscited by 53
  36. 2023Spike No More: Stabilizing the Pre-training of Large Language Modelscited by 51
  37. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 50
  38. 2023SwitchHead: Accelerating Transformers with Mixture-of-Experts Attentioncited by 34

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 36; 2026: 230.

266 papers.

Sort:

By area:

  1. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  2. 2026MiMo-V2-Flash Technical Reportcited by 163
  3. 2025TTT3R: 3D Reconstruction as Test-Time Trainingcited by 122
  4. 2026Kimi K3: Open Frontier Intelligencecited by 99
  5. 2025Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scalingcited by 29
  6. 2026ZipMap: Linear-Time Stateful 3D Reconstruction via Test-Time Trainingcited by 23
  7. 2026The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinkscited by 23
  8. 2026InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editingcited by 23
  9. 2026Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Modelcited by 23
  10. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  11. 2026A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Trainingcited by 21
  12. 2025GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignmentcited by 20
38 more
  1. 2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitycited by 20
  2. 2026Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contextscited by 16
  3. 2025Unveiling Super Experts in Mixture-of-Experts Large Language Modelscited by 15
  4. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 13
  5. 2025TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Controlcited by 13
  6. 2026Gated Memory Policy: In-Context Memorization and Adaptationcited by 13
  7. 2025BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholdingcited by 12
  8. 2025AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agentscited by 12
  9. 2026SMELT: Scaling Laws for Compute-Matched MoE Looped Transformerscited by 12
  10. 2026Fast-weight Product Key Memorycited by 11
  11. 2025Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformerscited by 10
  12. 2025Which Heads Matter for Reasoning? RL-Guided KV Cache Compressioncited by 10
  13. 2026Baichuan-M3: Modeling Clinical Inquiry for Reliable Medical Decision-Makingcited by 10
  14. 2026Arcee Trinity Large Technical Reportcited by 10
  15. 2026Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Taskscited by 10
  16. 2025Mitigating Attention Sinks and Massive Activations in Audio-Visual Speech Recognition with LLMscited by 9
  17. 2026Train Short, Inference Long: Training-free Horizon Extension for Autoregressive Video Generationcited by 9
  18. 2025A scalable and real-time neural decoder for topological quantum codescited by 8
  19. 2025HiStream: Efficient High-Resolution Video Generation via Redundancy-Eliminated Streamingcited by 8
  20. 2026HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharingcited by 8
  21. 2026SiameseNorm: Breaking the Barrier to Reconciling Pre/Post-Normcited by 8
  22. 2026HoloMotion-1 Technical Reportcited by 8
  23. 2026HorizonStream: Long-Horizon Attention for Streaming 3D Reconstructioncited by 8
  24. 2025Value-State Gated Attention for Mitigating Extreme-Token Phenomena in Transformerscited by 7
  25. 2025TNT: Improving Chunkwise Training for Test-Time Memorizationcited by 7
  26. 2026MASCOT: Multi-Agent Socio-Collaborative Companion Systemscited by 7
  27. 2026Long-Horizon Streaming Video Generation via Hybrid Attention with Decoupled Distillationcited by 7
  28. 2026HRM-Text: Efficient Pretraining Beyond Scalingcited by 7
  29. 2026Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsitycited by 7
  30. 2025On the Expressiveness of Softmax Attention: A Recurrent Neural Network Perspectivecited by 6
  31. 2026What Matters For Safety Alignment?cited by 6
  32. 2026You Need an Encoder for Native Position-Independent Cachingcited by 6
  33. 2025Revisiting Long-context Modeling from Context Denoising Perspectivecited by 5
  34. 2025Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Designcited by 5
  35. 2025TWEO: Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummiescited by 5
  36. 2026A Brain-inspired Embodied Intelligence for Fluid and Fast Reflexive Robotics Controlcited by 5
  37. 2026Hardware Co-Design Scaling Laws via Roofline Modelling for On-Device LLMscited by 5
  38. 2026Beyond VLM-Based Rewards: Diffusion-Native Latent Reward Modelingcited by 5

50 most cited of 266 All 266 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs
  2. 2025Gating is Weighting: Understanding Gated Linear Attention through In-context Learning
  3. 2026On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models
  4. 2026Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMs
  5. 2023TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormer
  6. 2025Softplus Attention with Re-weighting Boosts Length Extrapolation in Large Language Models
  7. 2022Mega: Moving Average Equipped Gated Attention
  8. 2026AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient Long-Context Inference