mlsanityData to 7 Oct 2026

Were RNNs All We Needed?

arXiv 2410.01201 ↗First posted 2 Oct 2024cs.LG cs.AIcited by 36

Not on the map. Why?

Abstract (arXiv)

The introduction of Transformers in 2017 reshaped the landscape of deep learning. Originally proposed for sequence modelling, Transformers have since achieved widespread success across various domains. However, the scalability limitations of Transformers - particularly with respect to sequence length - have sparked renewed interest in novel recurrent models that are parallelizable during training, offer comparable performance, and scale more effectively. In this work, we revisit sequence modelling from a historical perspective, focusing on Recurrent Neural Networks (RNNs), which dominated the field for two decades before the rise of Transformers. Specifically, we examine LSTMs (1997) and GRUs (2014). We demonstrate that by simplifying these models, we can derive minimal versions (minLSTMs and minGRUs) that (1) use fewer parameters than their traditional counterparts, (2) are fully parallelizable during training, and (3) achieve surprisingly competitive performance on a range of tasks, rivalling recent models including Transformers.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 31 of its 37 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

31 of 37 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  6. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  7. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  10. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  11. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  12. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
19 more
  1. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  2. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  3. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  4. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  5. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  6. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  7. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  8. 2024xLSTM: Extended Long Short-Term Memorycited by 320
  9. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  10. 2022Long Range Language Modeling via Gated State Spacescited by 265
  11. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  12. 2022Liquid Structural State-Space Modelscited by 134
  13. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  14. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  15. 2023GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modelingcited by 45
  16. 2024A Survey of Mambacited by 39
  17. 2023Decision S4: Efficient Sequence-Based RL via State Spaces Layerscited by 26
  18. 2024Decision Mamba: Reinforcement Learning via Sequence Modeling with Selective State Spacescited by 23
  19. 2024Attention as an RNNcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 15; 2026: 16.

36 papers.

Sort:

By area:

  1. 2025Transformers without Normalizationcited by 69
  2. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothingcited by 17
  3. 2024A Comprehensive Survey of Mamba Architectures for Medical Image Analysis: Classification, Segmentation, Restoration and Beyondcited by 16
  4. 2025AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understandingcited by 9
  5. 2024A Survey of RWKVcited by 5
  6. 2026Pretraining Recurrent Networks without Recurrencecited by 5
  7. 2025RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modelingcited by 3
  8. 2024Memory makes computation universal, remember?cited by 2
  9. 2025MinGRU-Based Encoder for Turbo Autoencoder Frameworkscited by 2
  10. 2025Knowing When to Quit: Probabilistic Early Exits for Speech Separationcited by 2
  11. 2025From Small to Large Language Models: Revisiting the Federalist Paperscited by 1
  12. 2025Temporal horizons in forecasting: a performance-learnability trade-offcited by 1
24 more
  1. 2025Neuromorphic Intelligencecited by 1
  2. 2026Lipschitz Multiscale Deep Equilibrium Models: A Theoretically Guaranteed and Accelerated Approachcited by 1
  3. 2026The Impossibility Triangle of Long-Context Modelingcited by 1
  4. 2026Improving the Performance and Learning Stability of Parallelizable RNNs Designed for Ultra-Low Power Applicationscited by 1
  5. 2024Machines of Meaningcited by 0
  6. 2025GraphMinNet: Learning Dependencies in Graphs with Light Complexity Minimal Architecturecited by 0
  7. 2025Exploring Performance-Complexity Trade-Offs in Sound Event Detection Modelscited by 0
  8. 2025Bidirectional Linear Recurrent Models for Sequence-Level Multisource Fusioncited by 0
  9. 2025mGRADE: Minimal Recurrent Gating Meets Delay Convolutions for Lightweight Sequence Modelingcited by 0
  10. 2025Minimal Convolutional RNNs Accelerate Spatiotemporal Learningcited by 0
  11. 2025Generalized Orders of Magnitude for Scalable, Parallel, High-Dynamic-Range Computationcited by 0
  12. 2025EMTSF:Extraordinary Mixture of SOTA Models for Time Series Forecastingcited by 0
  13. 2026Hierarchical temporal receptive windows and zero-shot timescale generalization in biologically constrained scale-invariant deep networkscited by 0
  14. 2026Spark: Modular Spiking Neural Networkscited by 0
  15. 2026Improved state mixing in higher-order and block diagonal linear recurrent networkscited by 0
  16. 2026Efficient Sparse Selective-Update RNNs for Long-Range Sequence Modelingcited by 0
  17. 2026Scalable Memristive-Friendly Reservoir Computing for Time Series Classificationcited by 0
  18. 2026Hidden States as Value Gradients: The Pontryagin Structure of Recurrent Policiescited by 0
  19. 2026On the Importance of Multistability for Horizon Generalization in Reinforcement Learningcited by 0
  20. 2026Trading Complexity for Expressivity Through Structured Generalized Linear Token Mixingcited by 0
  21. 2026Unveiling Transferability in Trajectory Prediction via Latent Scene Embeddingscited by 0
  22. 2026Agentic Anomaly Detection with ORCA-Style Dynamic Inductive Bias Adaptation in Multimodal Wearable Time Series Datacited by 0
  23. 2026Length-Independent State Tracking Under a Parallel Scancited by 0
  24. 2026Fast and Precise Learned Charged-Particle Trajectory Regression at the Large Hadron Collidercited by 0