# GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling

arXiv: https://arxiv.org/abs/2311.01927 · HTML: https://mlsanity.com/paper/2311.01927/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 11 references that are in the map.
- First posted on arXiv 3 Nov 2023.

## Builds on (most cited first, 5 of 5)

- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [Retentive Network: A Successor to Transformer for Large Language Models](https://mlsanity.com/paper/2307.08621/) (2023)
- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [Jointly Learning to Align and Translate with Transformer Models](https://mlsanity.com/paper/1909.02074/) (2019)

## Built on by (most cited first, 30 of 46)

- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Gated Linear Attention Transformers with Hardware-Efficient Training](https://mlsanity.com/paper/2312.06635/) (2023)
- [Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models](https://mlsanity.com/paper/2402.19427/) (2024)
- [Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling](https://mlsanity.com/paper/2406.07522/) (2024)
- [You Only Cache Once: Decoder-Decoder Architectures for Language Models](https://mlsanity.com/paper/2405.05254/) (2024)
- [HGRN2: Gated Linear RNNs with State Expansion](https://mlsanity.com/paper/2404.07904/) (2024)
- [Gated Slot Attention for Efficient Linear-Time Sequence Modeling](https://mlsanity.com/paper/2409.07146/) (2024)
- [Theoretical Foundations of Deep Selective State-Space Models](https://mlsanity.com/paper/2402.19047/) (2024)
- [Test-time regression: a unifying framework for designing sequence models with associative memory](https://mlsanity.com/paper/2501.12352/) (2025)
- [State Space Model for New-Generation Network Alternative to Transformers: A Survey](https://mlsanity.com/paper/2404.09516/) (2024)
- [Forgetting Transformer: Softmax Attention with a Forget Gate](https://mlsanity.com/paper/2503.02130/) (2025)
- [LinFusion: 1 GPU, 1 Minute, 16K Image](https://mlsanity.com/paper/2409.02097/) (2024)
- [Log-Linear Attention](https://mlsanity.com/paper/2506.04761/) (2025)
- [Were RNNs All We Needed?](https://mlsanity.com/paper/2410.01201/) (2024)
- [Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks](https://mlsanity.com/paper/2405.15731/) (2024)
- [Understanding and Improving Length Generalization in Recurrent Models](https://mlsanity.com/paper/2507.02782/) (2025)
- [Efficient Attention Mechanisms for Large Language Models: A Survey](https://mlsanity.com/paper/2507.19595/) (2025)
- [MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map](https://mlsanity.com/paper/2411.10741/) (2024)
- [Distilling to Hybrid Attention Models via KL-Guided Layer Selection](https://mlsanity.com/paper/2512.20569/) (2025)
- [SymbolicAI: A framework for logic-based approaches combining generative models and solvers](https://mlsanity.com/paper/2402.00854/) (2024)
- [Selective Attention: Enhancing Transformer through Principled Context Control](https://mlsanity.com/paper/2411.12892/) (2024)
- [Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters](https://mlsanity.com/paper/2408.04093/) (2024)
- [Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels](https://mlsanity.com/paper/2503.14376/) (2025)
- [Gating is Weighting: Understanding Gated Linear Attention through In-context Learning](https://mlsanity.com/paper/2504.04308/) (2025)
- [Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulation](https://mlsanity.com/paper/2405.16504/) (2024)
- [Latent Mamba Operator for Partial Differential Equations](https://mlsanity.com/paper/2505.19105/) (2025)
- [Kairos: A Regret-Aware Native World-Action Model Stack for Physical AI](https://mlsanity.com/paper/2606.16533/) (2026)
- [Linear Attention Sequence Parallelism](https://mlsanity.com/paper/2404.02882/) (2024)
- [Unlocking the Secrets of Linear Complexity Sequence Model from A Unified Perspective](https://mlsanity.com/paper/2405.17383/) (2024)
- [Linear Attention for Efficient Bidirectional Sequence Modeling](https://mlsanity.com/paper/2502.16249/) (2025)

## Abstract (arXiv)

Linear Recurrence has proven to be a powerful tool for modeling long sequences efficiently. In this work, we show that existing models fail to take full advantage of its potential. Motivated by this finding, we develop GateLoop, a foundational sequence model that generalizes linear recurrent models such as S4, S5, LRU and RetNet, by employing data-controlled state transitions. Utilizing this theoretical advance, GateLoop empirically outperforms existing models for auto-regressive language modeling. Our method comes with a low-cost O(l) recurrent mode and an efficient O(l log2 l) parallel mode making use of highly optimized associative scan implementations. Furthermore, we derive an O(l2) surrogate attention mode, revealing remarkable implications for Transformer and recently proposed architectures. Specifically, we prove that our approach can be interpreted as providing data-controlled relative-positional information to Attention. While many existing models solely rely on data-controlled cumulative sums for context aggregation, our findings suggest that incorporating data-controlled complex cumulative products may be a crucial step towards more powerful sequence models.