mlsanityData to 7 Oct 2026

Gated Slot Attention for Efficient Linear-Time Sequence Modeling

arXiv 2409.07146 ↗First posted 11 Sep 2024cs.CLcited by 73

Not on the map. Why?

Abstract (arXiv)

Linear attention Transformers and their gated variants, celebrated for enabling parallel training and efficient recurrent inference, still fall short in recall-intensive tasks compared to traditional Transformers and demand significant resources for training from scratch. This paper introduces Gated Slot Attention (GSA), which enhances Attention with Bounded-memory-Control (ABC) by incorporating a gating mechanism inspired by Gated Linear Attention (GLA). Essentially, GSA comprises a two-layer GLA linked via operatornamesoftmax, utilizing context-aware memory reading and adaptive forgetting to improve memory capacity while maintaining compact recurrent state size. This design greatly enhances both training and inference efficiency through GLA's hardware-efficient training algorithm and reduced state size. Additionally, retaining the operatornamesoftmax operation is particularly beneficial in "finetuning pretrained Transformers to RNNs" (T2R) settings, reducing the need for extensive training from scratch. Extensive experiments confirm GSA's superior performance in scenarios requiring in-context recall and in T2R settings.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Builds on 69 of its 92 references that are in the map. First posted on arXiv 11 Sep 2024.

Builds on

69 of 92 references.

References · 69

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
38 more
  1. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  2. 2020GLU Variants Improve Transformercited by 1,724
  3. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  4. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  5. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  6. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  7. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  8. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  9. 2016Language Modeling with Gated Convolutional Networkscited by 688
  10. 2014Neural Turing Machinescited by 630
  11. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  12. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  13. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  14. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  15. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  16. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  17. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  18. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  19. 2021Random Feature Attentioncited by 350
  20. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  21. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  22. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  23. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  24. 2022Transformer Quality in Linear Timecited by 189
  25. 2024An Empirical Study of Mamba-based Language Modelscited by 177
  26. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  27. 2016Using Fast Weights to Attend to the Recent Pastcited by 161
  28. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  29. 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 126
  30. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  31. 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 119
  32. 2020Large Associative Memory Problem in Neurobiology and Machine Learningcited by 110
  33. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  34. 2022The Devil in Linear Transformercited by 98
  35. 2021Luna: Linear Unified Nested Attentioncited by 96
  36. 2021Finetuning Pretrained Transformers into RNNscited by 94
  37. 2024In-Context Language Learning: Architectures and Algorithmscited by 86
  38. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 84

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 37; 2026: 31.

73 papers.

Sort:

By area:

  1. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  2. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  3. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  4. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  5. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  6. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 50
  7. 2025MoM: Linear Sequence Modeling with Mixture-of-Memoriescited by 44
  8. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
  9. 2025MesaNet: Sequence Modeling by Locally Optimal Test-Time Trainingcited by 37
  10. 2025Liger: Linearizing Large Language Models to Gated Recurrent Structurescited by 32
  11. 2025Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layerscited by 32
  12. 2024CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Upcited by 26
38 more
  1. 2025RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scalecited by 24
  2. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 21
  3. 2025Lattice: Learning to Efficiently Compress the Memorycited by 18
  4. 2025Distilling to Hybrid Attention Models via KL-Guided Layer Selectioncited by 17
  5. 2025Structured Linear CDEs: Maximally Expressive and Parallel-in-Time Sequence Modelscited by 16
  6. 2026Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contextscited by 16
  7. 2026PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformerscited by 16
  8. 2025Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanismcited by 15
  9. 2025Mamba Modulation: On the Length Generalization of Mambacited by 13
  10. 2025Comba: Improving Bilinear RNNs with Closed-loop Controlcited by 12
  11. 2025Scaling Linear Attention with Sparse State Expansioncited by 12
  12. 2026Test-Time Training with KV Binding Is Secretly Linear Attentioncited by 10
  13. 2025Hadamard product in deep learning: Introduction, Advances and Challengescited by 9
  14. 2025AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understandingcited by 9
  15. 2025Artificial Hippocampus Networks for Efficient Long-Context Modelingcited by 9
  16. 2025BiECVC: Gated Diversification of Bidirectional Contexts for Learned Video Compressioncited by 8
  17. 2024Stuffed Mamba: Oversized States Lead to the Inability to Forgetcited by 7
  18. 2025ARFlow: Autoregressive Flow with Hybrid Linear Attentioncited by 7
  19. 2025Hardware-aligned Hierarchical Sparse Attention for Efficient Long-term Memory Accesscited by 7
  20. 2025Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Servingcited by 7
  21. 2025TNT: Improving Chunkwise Training for Test-Time Memorizationcited by 7
  22. 2025Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Expertscited by 6
  23. 2025Resona: Improving Context Copying in Linear Recurrence Models with Retrievalcited by 5
  24. 2025SpikingBrain: Spiking Brain-inspired Large Modelscited by 5
  25. 2026CoMeT: Collaborative Memory Transformer for Efficient Long Context Modelingcited by 5
  26. 2026MeMix: Writing Less, Remembering More for Streaming 3D Reconstructioncited by 5
  27. 2025Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worldscited by 4
  28. 2026A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgetscited by 4
  29. 2026Raven: High-Recall Sequence Modeling with Sparse Memory Routingcited by 4
  30. 2025RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modelingcited by 3
  31. 2025SAS: Simulated Attention Scorecited by 3
  32. 2025Context and Diversity Matter: The Emergence of In-Context Learning in World Modelscited by 3
  33. 2025Trellis: Learning to Compress Key-Value Memory in Attention Modelscited by 3
  34. 2026MDN: Parallelizing Stepwise Momentum for Delta Linear Attentioncited by 3
  35. 2024Learning in the Recurrent State: Gradient Descent with Linear Recurrent Networkscited by 2
  36. 2026Dissecting Outlier Dynamics in LLM NVFP4 Pretrainingcited by 2
  37. 2026RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable Statecited by 2
  38. 2026HyperMLP: An Integrated Perspective for Sequence Modelingcited by 2

50 most cited of 73 All 73 (JSON)