mlsanityData to 7 Oct 2026

On the difficulty of learning chaotic dynamics with RNNs

arXiv 2110.07238 ↗First posted 14 Oct 2021cs.LG math.DS stat.MLcited by 64

Not on the map. Why?

Abstract (arXiv)

Recurrent neural networks (RNNs) are wide-spread machine learning tools for modeling sequential and time series data. They are notoriously hard to train because their loss gradients backpropagated in time tend to saturate or diverge during training. This is known as the exploding and vanishing gradient problem. Previous solutions to this issue either built on rather complicated, purpose-engineered architectures with gated memory buffers, or - more recently - imposed constraints that ensure convergence to a fixed point or restrict (the eigenspectrum of) the recurrence matrix. Such constraints, however, convey severe limitations on the expressivity of the RNN. Essential intrinsic dynamics such as multistability or chaos are disabled. This is inherently at disaccord with the chaotic nature of many, if not most, time series encountered in nature and society. It is particularly problematic in scientific applications where one aims to reconstruct the underlying dynamical system. Here we offer a comprehensive theoretical treatment of this problem by relating the loss gradients during RNN training to the Lyapunov spectrum of RNN-generated orbits. We mathematically prove that RNNs producing stable equilibrium or cyclic behavior have bounded gradients, whereas the gradients of RNNs with chaotic dynamics always diverge. Based on these analyses and insights we suggest ways of how to optimize the training process on chaotic data according to the system's Lyapunov spectrum, regardless of the employed RNN architecture.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Builds on 29 of its 114 references that are in the map. First posted on arXiv 14 Oct 2021.

Builds on

29 of 114 references.

References · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  5. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  6. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  7. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  8. 2015A Critical Review of Recurrent Neural Networks for Sequence Learningcited by 223
  9. 2018Data-Driven Forecasting of High-Dimensional Chaotic Systems with Long Short-Term Memory Networkscited by 155
  10. 2018Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNNcited by 142
  11. 2019AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networkscited by 112
  12. 2020Learning Stable Deep Dynamics Modelscited by 110
17 more
  1. 2020Lipschitz Recurrent Neural Networkscited by 88
  2. 2021Chaos as an interpretable benchmark for forecasting and data-driven modellingcited by 87
  3. 2017Orthogonal Recurrent Neural Networks with Scaled Cayley Transformcited by 82
  4. 2020Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependenciescited by 81
  5. 2018Learning Longer-term Dependencies in RNNs with Auxiliary Lossescited by 58
  6. 2021UnICORNN: A recurrent model for learning very long time dependenciescited by 56
  7. 2019Identifying nonlinear dynamical systems via generative recurrent neural networks with applications to fMRIcited by 39
  8. 2016Recurrent Orthogonal Networks and Long-Memory Taskscited by 38
  9. 2019Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamicscited by 38
  10. 2020Lyapunov spectra of chaotic recurrent neural networkscited by 37
  11. 2017Gated Orthogonal Recurrent Units: On Learning to Forgetcited by 31
  12. 2020On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Toolscited by 31
  13. 2019Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamicscited by 28
  14. 2015Synthesis of recurrent neural networks for dynamical system simulationcited by 16
  15. 2017Machine Learning, Deepest Learning: Statistical Data Assimilation Problemscited by 13
  16. 2019Improved memory in recurrent neural networks with sequential non-normal dynamicscited by 7
  17. 2021Transient Chaos in BERTcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 10; 2024: 16; 2025: 20; 2026: 15.

64 papers.

Sort:

By area:

  1. 2023PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solverscited by 205
  2. 2023Generalized Teacher Forcing for Learning Chaotic Dynamicscited by 55
  3. 2023Training neural operators to preserve invariant measures of chaotic attractorscited by 43
  4. 2024Oscillatory State-Space Modelscited by 35
  5. 2024Out-of-Domain Generalization in Dynamical Systems Reconstructioncited by 33
  6. 2024DySLIM: Dynamics Stable Learning by Invariant Measure for Chaotic Systemscited by 29
  7. 2024Zero-shot forecasting of chaotic systemscited by 25
  8. 2023Generative learning for nonlinear dynamicscited by 22
  9. 2025Thermalizer: Stable autoregressive neural emulation of spatiotemporal chaoscited by 21
  10. 2025True Zero-Shot Inference of Dynamical Systems Preserving Long-Term Statisticscited by 20
  11. 2023Benchmarking sparse system identification with low-dimensional chaoscited by 19
  12. 2024Almost-Linear RNNs Yield Highly Interpretable Symbolic Codes in Dynamical Systems Reconstructioncited by 18
38 more
  1. 2023Bifurcations and loss jumps in RNN trainingcited by 17
  2. 2022Homotopy-based training of NeuralODEs for accurate dynamics discoverycited by 14
  3. 2024Learning Interpretable Hierarchical Dynamical Systems Models from Time Series Datacited by 14
  4. 2024When are dynamical systems learned from time series data statistically accurate?cited by 14
  5. 2025Toward Large-Scale Distributed Quantum Long Short-Term Memory with Modular Quantum Computerscited by 14
  6. 2025Context parroting: A simple but tough-to-beat baseline for foundation models in scientific machine learningcited by 14
  7. 2022Integrating Multimodal Data for Joint Generative Modeling of Complex Dynamicscited by 13
  8. 2025Panda: A pretrained forecast model for chaotic dynamicscited by 12
  9. 2024A scalable generative model for dynamical system reconstruction from neuroimaging datacited by 9
  10. 2024Optimal Recurrent Network Topologies for Dynamical Systems Reconstructioncited by 8
  11. 2024Koopman-informed recurrent neural networkscited by 8
  12. 2023Persistent learning signals and working memory without continuous attractorscited by 6
  13. 2025POCO: Scalable Neural Forecasting through Population Conditioningcited by 6
  14. 2023Gradient Flossing: Improving Gradient Descent through Dynamic Control of Jacobianscited by 5
  15. 2025Uncovering the Computational Roles of Nonlinearity in Sequence Modeling Using Almost-Linear RNNscited by 4
  16. 2022Lyapunov-Guided Representation of Recurrent Neural Network Performancecited by 3
  17. 2024Mamba State-Space Models Are Lyapunov-Stable Learnerscited by 3
  18. 2024Designing Chaotic Attractors: A Semi-supervised Approachcited by 3
  19. 2024Toward Physics-guided Time Series Embeddingcited by 3
  20. 2025Uncovering the Spectral Bias in Diagonal State Space Modelscited by 3
  21. 2025Detecting Invariant Manifolds in ReLU-Based RNNscited by 3
  22. 2025INC: An Indirect Neural Corrector for Auto-Regressive Hybrid PDE Solverscited by 3
  23. 2026Continuous-Time Piecewise-Linear Recurrent Neural Networkscited by 3
  24. 2026Position: A Dynamical Systems Perspective is Needed to Advance Time Series Modelingcited by 3
  25. 2023Course Correcting Koopman Representationscited by 2
  26. 2025Neural Functions for Learning Periodic Signalcited by 2
  27. 2025ECO: Energy-Constrained Operator Learning for Chaotic Dynamics with Boundedness Guaranteescited by 2
  28. 2026Teacher Forcing as Generalized Bayes: Optimization Geometry Mismatch in Switching Surrogates for Chaotic Dynamicscited by 2
  29. 2023Message Propagation Through Time: An Algorithm for Sequence Dependency Retention in Time Series Modelingcited by 1
  30. 2024Systematic construction of continuous-time neural networks for linear dynamical systemscited by 1
  31. 2024Divide And Conquer: Learning Chaotic Dynamical Systems With Multistep Penalty Neural Ordinary Differential Equationscited by 1
  32. 2024Predicting Chaotic System Behavior using Machine Learning Techniquescited by 1
  33. 2025Investigating Memory in Model-Free RL with POPGym Arcadecited by 1
  34. 2025Temporal horizons in forecasting: a performance-learnability trade-offcited by 1
  35. 2025KPFlow: An Operator Perspective on Dynamic Collapse Under Gradient Descent Training of Recurrent Networkscited by 1
  36. 2025Double projection for reconstructing dynamical systems: between stochastic and deterministic regimescited by 1
  37. 2026Multimodal Scientific Learning Beyond Diffusions and Flowscited by 1
  38. 2026Stable Long-Horizon PDE Forecasting via Latent Structured Spectral Propagatorscited by 1

50 most cited of 64 All 64 (JSON)