# Were RNNs All We Needed?

arXiv: https://arxiv.org/abs/2410.01201 · HTML: https://mlsanity.com/paper/2410.01201/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 37 references that are in the map.
- First posted on arXiv 2 Oct 2024.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Retentive Network: A Successor to Transformer for Large Language Models](https://mlsanity.com/paper/2307.08621/) (2023)
- [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://mlsanity.com/paper/2008.07669/) (2020)
- [Gated Linear Attention Transformers with Hardware-Efficient Training](https://mlsanity.com/paper/2312.06635/) (2023)
- [Hungry Hungry Hippos: Towards Language Modeling with State Space Models](https://mlsanity.com/paper/2212.14052/) (2022)
- [On the Parameterization and Initialization of Diagonal State Space Models](https://mlsanity.com/paper/2206.11893/) (2022)
- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [Hyena Hierarchy: Towards Larger Convolutional Language Models](https://mlsanity.com/paper/2302.10866/) (2023)
- [Diagonal State Spaces are as Effective as Structured State Spaces](https://mlsanity.com/paper/2203.14343/) (2022)
- [xLSTM: Extended Long Short-Term Memory](https://mlsanity.com/paper/2405.04517/) (2024)
- [Unitary Evolution Recurrent Neural Networks](https://mlsanity.com/paper/1511.06464/) (2015)
- [Long Range Language Modeling via Gated State Spaces](https://mlsanity.com/paper/2206.13947/) (2022)
- [It's Raw! Audio Generation with State-Space Models](https://mlsanity.com/paper/2202.09729/) (2022)
- [Liquid Structural State-Space Models](https://mlsanity.com/paper/2209.12951/) (2022)
- [Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges](https://mlsanity.com/paper/2404.16112/) (2024)
- [State Space Model for New-Generation Network Alternative to Transformers: A Survey](https://mlsanity.com/paper/2404.09516/) (2024)
- [GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling](https://mlsanity.com/paper/2311.01927/) (2023)
- [A Survey of Mamba](https://mlsanity.com/paper/2408.01129/) (2024)
- [Decision S4: Efficient Sequence-Based RL via State Spaces Layers](https://mlsanity.com/paper/2306.05167/) (2023)
- [Decision Mamba: Reinforcement Learning via Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2403.19925/) (2024)

## Built on by (most cited first, 30 of 38)

- [Transformers without Normalization](https://mlsanity.com/paper/2503.10622/) (2025)
- [Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing](https://mlsanity.com/paper/2501.00658/) (2024)
- [A Comprehensive Survey of Mamba Architectures for Medical Image Analysis: Classification, Segmentation, Restoration and Beyond](https://mlsanity.com/paper/2410.02362/) (2024)
- [AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding](https://mlsanity.com/paper/2507.02591/) (2025)
- [A Survey of RWKV](https://mlsanity.com/paper/2412.14847/) (2024)
- [Pretraining Recurrent Networks without Recurrence](https://mlsanity.com/paper/2606.06479/) (2026)
- [RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling](https://mlsanity.com/paper/2507.04416/) (2025)
- [Memory makes computation universal, remember?](https://mlsanity.com/paper/2412.17794/) (2024)
- [MinGRU-Based Encoder for Turbo Autoencoder Frameworks](https://mlsanity.com/paper/2503.08451/) (2025)
- [Knowing When to Quit: Probabilistic Early Exits for Speech Separation](https://mlsanity.com/paper/2507.09768/) (2025)
- [From Small to Large Language Models: Revisiting the Federalist Papers](https://mlsanity.com/paper/2503.01869/) (2025)
- [Temporal horizons in forecasting: a performance-learnability trade-off](https://mlsanity.com/paper/2506.03889/) (2025)
- [Neuromorphic Intelligence](https://mlsanity.com/paper/2509.11940/) (2025)
- [Lipschitz Multiscale Deep Equilibrium Models: A Theoretically Guaranteed and Accelerated Approach](https://mlsanity.com/paper/2602.03297/) (2026)
- [The Impossibility Triangle of Long-Context Modeling](https://mlsanity.com/paper/2605.05066/) (2026)
- [Improving the Performance and Learning Stability of Parallelizable RNNs Designed for Ultra-Low Power Applications](https://mlsanity.com/paper/2605.11855/) (2026)
- [Machines of Meaning](https://mlsanity.com/paper/2412.07975/) (2024)
- [GraphMinNet: Learning Dependencies in Graphs with Light Complexity Minimal Architecture](https://mlsanity.com/paper/2502.00282/) (2025)
- [Exploring Performance-Complexity Trade-Offs in Sound Event Detection Models](https://mlsanity.com/paper/2503.11373/) (2025)
- [Bidirectional Linear Recurrent Models for Sequence-Level Multisource Fusion](https://mlsanity.com/paper/2504.08964/) (2025)
- [mGRADE: Minimal Recurrent Gating Meets Delay Convolutions for Lightweight Sequence Modeling](https://mlsanity.com/paper/2507.01829/) (2025)
- [Minimal Convolutional RNNs Accelerate Spatiotemporal Learning](https://mlsanity.com/paper/2508.03614/) (2025)
- [Generalized Orders of Magnitude for Scalable, Parallel, High-Dynamic-Range Computation](https://mlsanity.com/paper/2510.03426/) (2025)
- [EMTSF:Extraordinary Mixture of SOTA Models for Time Series Forecasting](https://mlsanity.com/paper/2510.23396/) (2025)
- [Hierarchical temporal receptive windows and zero-shot timescale generalization in biologically constrained scale-invariant deep networks](https://mlsanity.com/paper/2601.02618/) (2026)
- [Spark: Modular Spiking Neural Networks](https://mlsanity.com/paper/2602.02306/) (2026)
- [Improved state mixing in higher-order and block diagonal linear recurrent networks](https://mlsanity.com/paper/2602.12021/) (2026)
- [Efficient Sparse Selective-Update RNNs for Long-Range Sequence Modeling](https://mlsanity.com/paper/2603.02226/) (2026)
- [Scalable Memristive-Friendly Reservoir Computing for Time Series Classification](https://mlsanity.com/paper/2604.19343/) (2026)
- [Hidden States as Value Gradients: The Pontryagin Structure of Recurrent Policies](https://mlsanity.com/paper/2605.05373/) (2026)

## Abstract (arXiv)

The introduction of Transformers in 2017 reshaped the landscape of deep learning. Originally proposed for sequence modelling, Transformers have since achieved widespread success across various domains. However, the scalability limitations of Transformers - particularly with respect to sequence length - have sparked renewed interest in novel recurrent models that are parallelizable during training, offer comparable performance, and scale more effectively. In this work, we revisit sequence modelling from a historical perspective, focusing on Recurrent Neural Networks (RNNs), which dominated the field for two decades before the rise of Transformers. Specifically, we examine LSTMs (1997) and GRUs (2014). We demonstrate that by simplifying these models, we can derive minimal versions (minLSTMs and minGRUs) that (1) use fewer parameters than their traditional counterparts, (2) are fully parallelizable during training, and (3) achieve surprisingly competitive performance on a range of tasks, rivalling recent models including Transformers.