mlsanityData to 7 Oct 2026

On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools

arXiv 2006.14123 ↗First posted 25 Jun 2020cs.LG math.DS nlin.CDcited by 31

Not on the map. Why?

Abstract (arXiv)

Recurrent neural networks (RNNs) have been successfully applied to a variety of problems involving sequential data, but their optimization is sensitive to parameter initialization, architecture, and optimizer hyperparameters. Considering RNNs as dynamical systems, a natural way to capture stability, i.e., the growth and decay over long iterates, are the Lyapunov Exponents (LEs), which form the Lyapunov spectrum. The LEs have a bearing on stability of RNN training dynamics because forward propagation of information is related to the backward propagation of error gradients. LEs measure the asymptotic rates of expansion and contraction of nonlinear system trajectories, and generalize stability analysis to the time-varying attractors structuring the non-autonomous dynamics of data-driven RNNs. As a tool to understand and exploit stability of training dynamics, the Lyapunov spectrum fills an existing gap between prescriptive mathematical approaches of limited scope and computationally-expensive empirical approaches. To leverage this tool, we implement an efficient way to compute LEs for RNNs during training, discuss the aspects specific to standard RNN architectures driven by typical sequential datasets, and show that the Lyapunov spectrum can serve as a robust readout of training stability across hyperparameters. With this exposition-oriented contribution, we hope to draw attention to this understudied, but theoretically grounded tool for understanding training stability in RNNs.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 2; 2023: 6; 2024: 6; 2025: 5; 2026: 7.

31 papers.

Sort:

By area:

  1. 2021On the difficulty of learning chaotic dynamics with RNNscited by 64
  2. 2021Noisy Recurrent Neural Networkscited by 42
  3. 2024Out-of-Domain Generalization in Dynamical Systems Reconstructioncited by 33
  4. 2023Physics-Informed Machine Learning for Modeling and Control of Dynamical Systemscited by 28
  5. 2024Almost-Linear RNNs Yield Highly Interpretable Symbolic Codes in Dynamical Systems Reconstructioncited by 18
  6. 2022Integrating Multimodal Data for Joint Generative Modeling of Complex Dynamicscited by 13
  7. 2023Finite-time Lyapunov exponents of deep neural networkscited by 12
  8. 2025Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuningcited by 12
  9. 2024Sparse Spiking Neural Network: Exploiting Heterogeneity in Timescales for Pruning Recurrent SNNcited by 10
  10. 2024A scalable generative model for dynamical system reconstruction from neuroimaging datacited by 9
  11. 2021RNNs of RNNs: Recursive Construction of Stable Assemblies of Recurrent Neural Networkscited by 7
  12. 2023Persistent learning signals and working memory without continuous attractorscited by 6
19 more
  1. 2023Gradient Flossing: Improving Gradient Descent through Dynamic Control of Jacobianscited by 5
  2. 2021On the Stochastic Stability of Deep Markov Modelscited by 4
  3. 2025Uncovering the Computational Roles of Nonlinearity in Sequence Modeling Using Almost-Linear RNNscited by 4
  4. 2021A Differential Geometry Perspective on Orthogonal Recurrent Modelscited by 3
  5. 2022Lyapunov-Guided Representation of Recurrent Neural Network Performancecited by 3
  6. 2024Mamba State-Space Models Are Lyapunov-Stable Learnerscited by 3
  7. 2023Fading memory as inductive bias in residual recurrent networkscited by 2
  8. 2023Leveraging Low-Rank and Sparse Recurrent Connectivity for Robust Closed-Loop Controlcited by 2
  9. 2024Exploiting Heterogeneity in Timescales for Sparse Recurrent Spiking Neural Networks for Energy-Efficient Edge Computingcited by 2
  10. 2026Quantizing Time-Series Models As Dynamical Systems: Trajectory-Based Quantization Sensitivity Scorecited by 2
  11. 2025Hyperpruning: Efficient Search through Pruned Variants of Recurrent Neural Networks Leveraging Lyapunov Spectrumcited by 1
  12. 2025Adversarial Data Augmentation for Single Domain Generalization via Lyapunov Exponent-Guided Optimizationcited by 1
  13. 2025KPFlow: An Operator Perspective on Dynamic Collapse Under Gradient Descent Training of Recurrent Networkscited by 1
  14. 2026Tracking Finite-Time Lyapunov Exponents to Robustify Neural ODEscited by 1
  15. 2026Contrastive and Multi-Task Learning on Noisy Brain Signals with Nonlinear Dynamical Signaturescited by 0
  16. 2026Verifying Physics-Informed Neural Network Fidelity using Classical Fisher Information from Differentiable Dynamical Systemcited by 0
  17. 2026Optimal Initialization in Depth: Lyapunov Initialization and Limit Theorems for Deep Leaky ReLU Networkscited by 0
  18. 2026JEDI: Jointly Embedded Inference of Neural Dynamicscited by 0
  19. 2026When Recursive Models Finish Computingcited by 0