mlsanityData to 7 Oct 2026

Recurrent neural networks: vanishing and exploding gradients are not the end of the story

arXiv 2405.21064 ↗First posted 31 May 2024cs.LG cs.AI math.OCcited by 25

Not on the map. Why?

Abstract (arXiv)

Recurrent neural networks (RNNs) notoriously struggle to learn long-term memories, primarily due to vanishing and exploding gradients. The recent success of state-space models (SSMs), a subclass of RNNs, to overcome such difficulties challenges our theoretical understanding. In this paper, we delve into the optimization challenges of RNNs and discover that, as the memory of a network increases, changes in its parameters result in increasingly large output variations, making gradient-based learning highly sensitive, even without exploding gradients. Our analysis further reveals the importance of the element-wise recurrence design pattern combined with careful parametrizations in mitigating this effect. This feature is present in SSMs, as well as in other architectures, such as LSTMs. Overall, our insights provide a new explanation for some of the difficulties in gradient-based learning of RNNs and why some architectures perform better than others.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 44 of its 67 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

44 of 67 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2016Layer Normalizationcited by 5,579
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  11. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
32 more
  1. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  2. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  3. 2016Language Modeling with Gated Convolutional Networkscited by 688
  4. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  5. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  6. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  7. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  8. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  9. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  10. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  11. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  12. 2019Recurrent Independent Mechanismscited by 219
  13. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  14. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  15. 2024The Illusion of State in State-Space Modelscited by 181
  16. 2015A Simple Way to Initialize Recurrent Networks of Rectified Linear Unitscited by 130
  17. 2023A Spectral Condition for Feature Learningcited by 101
  18. 2017On orthogonality and learning recurrent networks with long term dependenciescited by 82
  19. 2017Orthogonal Recurrent Neural Networks with Scaled Cayley Transformcited by 82
  20. 2020Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependenciescited by 81
  21. 2016Gradient Descent Learns Linear Dynamical Systemscited by 78
  22. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  23. 2018Can recurrent neural networks warp time?cited by 53
  24. 2023Toward Understanding Why Adam Converges Faster Than SGD for Transformerscited by 50
  25. 2018Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networkscited by 44
  26. 2016Recurrent Batch Normalizationcited by 35
  27. 2023Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvaluescited by 35
  28. 2023Online learning of long-range dependenciescited by 25
  29. 2017Fast-Slow Recurrent Neural Networkscited by 12
  30. 2023Inverse Approximation Theory for Nonlinear Recurrent Neural Networkscited by 10
  31. 2023Persistent learning signals and working memory without continuous attractorscited by 6
  32. 2017On the Computation of Complex-valued Gradients with Application to Statistically Optimum Beamformingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 13; 2026: 9.

25 papers.

Sort:

By area:

  1. 2025In Search of Adam's Secret Saucecited by 40
  2. 2025Generative Graph Pattern Machinecited by 11
  3. 2024Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Modelscited by 8
  4. 2025Predictability Enables Parallelization of Nonlinear State Space Modelscited by 7
  5. 2025Revisiting associative recall in modern recurrent modelscited by 7
  6. 2026HRM-Text: Efficient Pretraining Beyond Scalingcited by 7
  7. 2024Zero-Shot Temporal Resolution Domain Adaptation for Spiking Neural Networkscited by 6
  8. 2024DeNOTS: Stable Deep Neural ODEs for Time Seriescited by 5
  9. 2025Fixed-Point RNNs: Interpolating from Diagonal to Densecited by 5
  10. 2025Weight-Space Linear Recurrent Neural Networkscited by 5
  11. 2025Revisiting Glorot Initialization for Long-Range Linear Recurrencescited by 2
  12. 2025SiLIF: Structured State Space Model Dynamics and Parametrization for Spiking Neural Networkscited by 2
13 more
  1. 2025Time-Scale Coupling Between States and Parameters in Recurrent Neural Networkscited by 1
  2. 2025Design Principles for Sequence Models via Coefficient Dynamicscited by 1
  3. 2026Dissecting Linear Recurrent Models: How Different Gating Strategies Drive Selectivity and Generalizationcited by 1
  4. 2026Tuning the burn-in phase in training recurrent neural networks improves their performancecited by 1
  5. 2026Beyond a Single Explanation of the Adam--SGD Gapcited by 1
  6. 2025Lyapunov Learning at the Onset of Chaoscited by 0
  7. 2025Overcoming Low-Resource Barriers in Tulu: Neural Models and Corpus Creation for OffensiveLanguage Identificationcited by 0
  8. 2025Insights from Gradient Dynamics: Gradient Autoscaled Normalizationcited by 0
  9. 2026Improved state mixing in higher-order and block diagonal linear recurrent networkscited by 0
  10. 2026Extending SSMs with the Exponentially Weighted Signaturecited by 0
  11. 2026MsFormer: Enabling Robust Predictive Maintenance Services for Industrial Devicescited by 0
  12. 2026Parallel-in-Time Training of Recurrent Neural Networks for Dynamical Systems Reconstructioncited by 0
  13. 2026Dynamics and Representation Structure of Local Approximations to Gradient-Based Learning in Linear Recurrent Neural Networkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Memory Determines Learning Direction: A Theory of Gradient-Based Optimization in State Space Models
  2. 2026Improving the Performance and Learning Stability of Parallelizable RNNs Designed for Ultra-Low Power Applications
  3. 2025Fast Training of Recurrent Neural Networks with Stationary State Feedbacks
  4. 2024State-Space Modeling in Long Sequence Processing: A Survey on Recurrence in the Transformer Era
  5. 2025ParaRNN: Unlocking Parallel Training of Nonlinear RNNs for Large Language Models
  6. 2023Dynamic Analysis and an Eigen Initializer for Recurrent Neural Networks
  7. 2024Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks
  8. 2025StateX: Enhancing RNN Recall via Post-training State Expansion