mlsanityData to 7 Oct 2026

Resurrecting Recurrent Neural Networks for Long Sequences

arXiv 2303.06349 ↗First posted 11 Mar 2023cs.LGcited by 397

Not on the map. Why?

Cited by 54 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recurrent Neural Networks (RNNs) offer fast inference on long sequences but are hard to optimize and slow to train. Deep state-space models (SSMs) have recently been shown to perform remarkably well on long sequence modeling tasks, and have the added benefits of fast parallelizable training and RNN-like fast inference. However, while SSMs are superficially similar to RNNs, there are important differences that make it unclear where their performance boost over RNNs comes from. In this paper, we show that careful design of deep RNNs using standard signal propagation arguments can recover the impressive performance of deep SSMs on long-range reasoning tasks, while also matching their training speed. To achieve this, we analyze and ablate a series of changes to standard RNNs including linearizing and diagonalizing the recurrence, using better parameterizations and initializations, and ensuring proper normalization of the forward pass. Our results provide new insights on the origins of the impressive performance of deep SSMs, while also introducing an RNN block called the Linear Recurrent Unit that matches both their performance on the Long Range Arena benchmark and their computational efficiency.

Key facts. Cited by 397 papers in the map (data through 7 Oct 2026). Builds on 58 of its 115 references that are in the map. First posted on arXiv 11 Mar 2023.

Builds on

58 of 115 references.

References · 58

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2016Layer Normalizationcited by 5,579
  7. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  10. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  11. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,447
  12. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
38 more
  1. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  2. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  3. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  4. 2016Language Modeling with Gated Convolutional Networkscited by 688
  5. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  6. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  7. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  8. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  9. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  10. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  11. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  12. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  13. 2022Long Range Language Modeling via Gated State Spacescited by 265
  14. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  15. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  16. 2020Liquid Time-constant Networkscited by 160
  17. 2022FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecastingcited by 147
  18. 2018Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNNcited by 142
  19. 2022Liquid Structural State-Space Modelscited by 134
  20. 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 133
  21. 2015A Simple Way to Initialize Recurrent Networks of Rectified Linear Unitscited by 130
  22. 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 126
  23. 2022How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projectionscited by 121
  24. 2022Long Movie Clip Classification with State-Space Video Modelscited by 106
  25. 2016Full-Capacity Unitary Recurrent Neural Networkscited by 90
  26. 2020Lipschitz Recurrent Neural Networkscited by 88
  27. 2022What Makes Convolutional Models Great on Long Sequence Modeling?cited by 86
  28. 2017Orthogonal Recurrent Neural Networks with Scaled Cayley Transformcited by 82
  29. 2016Gradient Descent Learns Linear Dynamical Systemscited by 78
  30. 2017On Convergence of Extended Dynamic Mode Decomposition to the Koopman Operatorcited by 70
  31. 2022Learning Dynamical Systems via Koopman Operator Regression in Reproducing Kernel Hilbert Spacescited by 66
  32. 2016Neural Machine Translation in Linear Timecited by 65
  33. 2017Data-driven discovery of Koopman eigenfunctions for controlcited by 64
  34. 2016Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNscited by 62
  35. 2016Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflectionscited by 56
  36. 2016Generalizing Koopman Theory to allow for inputs and controlcited by 49
  37. 2017Unbiased Online Recurrent Optimizationcited by 46
  38. 2018Sparse Attentive Backtracking: Temporal CreditAssignment Through Remindingcited by 33

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 49; 2024: 129; 2025: 120; 2026: 99.

397 papers.

Sort:

By area:

  1. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  2. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  3. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  4. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  5. 2024Titans: Learning to Memorize at Test Timecited by 303
  6. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  7. 2024LocalMamba: Visual State Space Model with Windowed Selective Scancited by 235
  8. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  9. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  10. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  11. 2023Protein Design with Guided Discrete Diffusioncited by 154
  12. 2024EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mambacited by 146
38 more
  1. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  2. 2025TTT3R: 3D Reconstruction as Test-Time Trainingcited by 122
  3. 2024Transformers are Multi-State RNNscited by 120
  4. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  5. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  6. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  7. 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 76
  8. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 74
  9. 2024Theoretical Foundations of Deep Selective State-Space Modelscited by 73
  10. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 71
  11. 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Productscited by 69
  12. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  13. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 63
  14. 2024RecurrentGemma: Moving Past Transformers for Efficient Open Language Modelscited by 61
  15. 2024MoE-Mamba: Efficient Selective State Space Models with Mixture of Expertscited by 60
  16. 2025It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimizationcited by 60
  17. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  18. 2024MambaByte: Token-free Selective State Space Modelcited by 58
  19. 2024Longhorn: State Space Models are Amortized Online Learnerscited by 58
  20. 2024Vision Mamba: A Comprehensive Survey and Taxonomycited by 53
  21. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 51
  22. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 50
  23. 2023Generative Modeling of Regular and Irregular Time Series Data via Koopman VAEscited by 48
  24. 2023Parallelizing non-linear sequential models over the sequence lengthcited by 45
  25. 2023GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modelingcited by 45
  26. 2024Scalable Diffusion Models with State Space Backbonecited by 45
  27. 2024The Expressive Capacity of State Space Models: A Formal Language Perspectivecited by 45
  28. 2025MoM: Linear Sequence Modeling with Mixture-of-Memoriescited by 44
  29. 2024Hydra: Bidirectional State Space Models Through Generalized Matrix Mixerscited by 43
  30. 2025On Vanishing Gradients, Over-Smoothing, and Over-Squashing in GNNs: Bridging Recurrent and Graph Learningcited by 42
  31. 2024Visual Mamba: A Survey and New Outlookscited by 41
  32. 2023State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memorycited by 38
  33. 2024LION: Linear Group RNN for 3D Object Detection in Point Cloudscited by 38
  34. 2024Towards Scalable and Stable Parallelization of Nonlinear RNNscited by 38
  35. 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 38
  36. 2024Were RNNs All We Needed?cited by 36
  37. 2024Oscillatory State-Space Modelscited by 35
  38. 2023TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormercited by 34

50 most cited of 397 All 397 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024State-Space Modeling in Long Sequence Processing: A Survey on Recurrence in the Transformer Era
  2. 2023Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues
  3. 2023State space models can express n-gram languages
  4. 2025Implicit Language Models are RNNs: Balancing Parallelization and Expressivity
  5. 2025MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
  6. 2026M2RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces
  8. 2026DSSMs: State Space Models with Explicit Memory via Delay Differential Equations