mlsanityData to 7 Oct 2026

GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling

arXiv 2311.01927 ↗First posted 3 Nov 2023cs.LG cs.AI cs.CLcited by 45

Not on the map. Why?

Abstract (arXiv)

Linear Recurrence has proven to be a powerful tool for modeling long sequences efficiently. In this work, we show that existing models fail to take full advantage of its potential. Motivated by this finding, we develop GateLoop, a foundational sequence model that generalizes linear recurrent models such as S4, S5, LRU and RetNet, by employing data-controlled state transitions. Utilizing this theoretical advance, GateLoop empirically outperforms existing models for auto-regressive language modeling. Our method comes with a low-cost O(l) recurrent mode and an efficient O(l log2 l) parallel mode making use of highly optimized associative scan implementations. Furthermore, we derive an O(l2) surrogate attention mode, revealing remarkable implications for Transformer and recently proposed architectures. Specifically, we prove that our approach can be interpreted as providing data-controlled relative-positional information to Attention. While many existing models solely rely on data-controlled cumulative sums for context aggregation, our findings suggest that incorporating data-controlled complex cumulative products may be a crucial step towards more powerful sequence models.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 5 of its 11 references that are in the map. First posted on arXiv 3 Nov 2023.

Builds on

5 of 11 references.

References · 5

  1. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  2. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  3. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  4. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  5. 2019Jointly Learning to Align and Translate with Transformer Modelscited by 75

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 21; 2025: 16; 2026: 7.

45 papers.

Sort:

By area:

  1. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  2. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  3. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  4. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  5. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  6. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  7. 2024Theoretical Foundations of Deep Selective State-Space Modelscited by 73
  8. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 73
  9. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 63
  10. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  11. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 50
  12. 2024LinFusion: 1 GPU, 1 Minute, 16K Imagecited by 42
33 more
  1. 2025Log-Linear Attentioncited by 39
  2. 2024Were RNNs All We Needed?cited by 36
  3. 2024Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networkscited by 26
  4. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 21
  5. 2024MetaLA: Unified Optimal Linear Approximation to Softmax Attention Mapcited by 20
  6. 2025Understanding and Improving Length Generalization in Recurrent Modelscited by 19
  7. 2025Distilling to Hybrid Attention Models via KL-Guided Layer Selectioncited by 17
  8. 2024Selective Attention: Enhancing Transformer through Principled Context Controlcited by 16
  9. 2024SymbolicAI: A framework for logic-based approaches combining generative models and solverscited by 15
  10. 2024Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusterscited by 13
  11. 2025Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernelscited by 12
  12. 2024Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulationcited by 11
  13. 2025Gating is Weighting: Understanding Gated Linear Attention through In-context Learningcited by 11
  14. 2024Linear Attention Sequence Parallelismcited by 8
  15. 2025Latent Mamba Operator for Partial Differential Equationscited by 8
  16. 2026Kairos: A Regret-Aware Native World-Action Model Stack for Physical AIcited by 8
  17. 2024Unlocking the Secrets of Linear Complexity Sequence Model from A Unified Perspectivecited by 6
  18. 2025Linear Attention for Efficient Bidirectional Sequence Modelingcited by 4
  19. 2025Transformers as Intrinsic Optimizers: Forward Inference through the Energy Principlecited by 3
  20. 2025Let SSMs be ConvNets: State-space Modeling with Optimal Tensor Contractionscited by 2
  21. 2025Technologies on Effectiveness and Efficiency: A Survey of State Spaces Modelscited by 2
  22. 2024On the low-shot transferability of [V]-Mambacited by 1
  23. 2024Softmax Attention with Constant Cost per Tokencited by 1
  24. 2024Lina-Speech: Gated Linear Attention and Initial-State Tuning for Multi-Sample Prompting Text-To-Speech Synthesiscited by 1
  25. 2026Kaczmarz Linear Attentioncited by 1
  26. 2026SMat-Attention: Structured Long-Context Sequence Modelingcited by 1
  27. 2025Conformal Transformations for Symmetric Power Transformerscited by 0
  28. 2025Attention as Conditioning: What Classical Learning Theory Predicts About Linear Transformerscited by 0
  29. 2025MossNet: Mixture of State-Space Experts is a Multi-Head Attentioncited by 0
  30. 2026Phase-Associative Memory: Sequence Modeling in Complex Hilbert Spacecited by 0
  31. 2026Learning to Adapt: In-Context Learning Beyond Stationaritycited by 0
  32. 2026Adaptive Memory Decay for Log-Linear Attentioncited by 0
  33. 2026Shared Phase and Retention Control for Efficient Adaptive Spectral Recurrencecited by 0