# On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools

arXiv: https://arxiv.org/abs/2006.14123 · HTML: https://mlsanity.com/paper/2006.14123/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jun 2020.

## Built on by (most cited first, 30 of 31)

- [On the difficulty of learning chaotic dynamics with RNNs](https://mlsanity.com/paper/2110.07238/) (2021)
- [Noisy Recurrent Neural Networks](https://mlsanity.com/paper/2102.04877/) (2021)
- [Out-of-Domain Generalization in Dynamical Systems Reconstruction](https://mlsanity.com/paper/2402.18377/) (2024)
- [Physics-Informed Machine Learning for Modeling and Control of Dynamical Systems](https://mlsanity.com/paper/2306.13867/) (2023)
- [Almost-Linear RNNs Yield Highly Interpretable Symbolic Codes in Dynamical Systems Reconstruction](https://mlsanity.com/paper/2410.14240/) (2024)
- [Integrating Multimodal Data for Joint Generative Modeling of Complex Dynamics](https://mlsanity.com/paper/2212.07892/) (2022)
- [Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning](https://mlsanity.com/paper/2503.01329/) (2025)
- [Finite-time Lyapunov exponents of deep neural networks](https://mlsanity.com/paper/2306.12548/) (2023)
- [Sparse Spiking Neural Network: Exploiting Heterogeneity in Timescales for Pruning Recurrent SNN](https://mlsanity.com/paper/2403.03409/) (2024)
- [A scalable generative model for dynamical system reconstruction from neuroimaging data](https://mlsanity.com/paper/2411.02949/) (2024)
- [RNNs of RNNs: Recursive Construction of Stable Assemblies of Recurrent Neural Networks](https://mlsanity.com/paper/2106.08928/) (2021)
- [Persistent learning signals and working memory without continuous attractors](https://mlsanity.com/paper/2308.12585/) (2023)
- [Gradient Flossing: Improving Gradient Descent through Dynamic Control of Jacobians](https://mlsanity.com/paper/2312.17306/) (2023)
- [Uncovering the Computational Roles of Nonlinearity in Sequence Modeling Using Almost-Linear RNNs](https://mlsanity.com/paper/2506.07919/) (2025)
- [On the Stochastic Stability of Deep Markov Models](https://mlsanity.com/paper/2111.04601/) (2021)
- [A Differential Geometry Perspective on Orthogonal Recurrent Models](https://mlsanity.com/paper/2102.09589/) (2021)
- [Lyapunov-Guided Representation of Recurrent Neural Network Performance](https://mlsanity.com/paper/2204.04876/) (2022)
- [Mamba State-Space Models Are Lyapunov-Stable Learners](https://mlsanity.com/paper/2406.00209/) (2024)
- [Fading memory as inductive bias in residual recurrent networks](https://mlsanity.com/paper/2307.14823/) (2023)
- [Leveraging Low-Rank and Sparse Recurrent Connectivity for Robust Closed-Loop Control](https://mlsanity.com/paper/2310.03915/) (2023)
- [Exploiting Heterogeneity in Timescales for Sparse Recurrent Spiking Neural Networks for Energy-Efficient Edge Computing](https://mlsanity.com/paper/2407.06452/) (2024)
- [Quantizing Time-Series Models As Dynamical Systems: Trajectory-Based Quantization Sensitivity Score](https://mlsanity.com/paper/2606.13300/) (2026)
- [Hyperpruning: Efficient Search through Pruned Variants of Recurrent Neural Networks Leveraging Lyapunov Spectrum](https://mlsanity.com/paper/2506.07975/) (2025)
- [Adversarial Data Augmentation for Single Domain Generalization via Lyapunov Exponent-Guided Optimization](https://mlsanity.com/paper/2507.04302/) (2025)
- [KPFlow: An Operator Perspective on Dynamic Collapse Under Gradient Descent Training of Recurrent Networks](https://mlsanity.com/paper/2507.06381/) (2025)
- [Tracking Finite-Time Lyapunov Exponents to Robustify Neural ODEs](https://mlsanity.com/paper/2602.09613/) (2026)
- [When Recursive Models Finish Computing](https://mlsanity.com/paper/2609.26487/) (2026)
- [Contrastive and Multi-Task Learning on Noisy Brain Signals with Nonlinear Dynamical Signatures](https://mlsanity.com/paper/2601.08549/) (2026)
- [Verifying Physics-Informed Neural Network Fidelity using Classical Fisher Information from Differentiable Dynamical System](https://mlsanity.com/paper/2601.11638/) (2026)
- [Optimal Initialization in Depth: Lyapunov Initialization and Limit Theorems for Deep Leaky ReLU Networks](https://mlsanity.com/paper/2602.10949/) (2026)

## Abstract (arXiv)

Recurrent neural networks (RNNs) have been successfully applied to a variety of problems involving sequential data, but their optimization is sensitive to parameter initialization, architecture, and optimizer hyperparameters. Considering RNNs as dynamical systems, a natural way to capture stability, i.e., the growth and decay over long iterates, are the Lyapunov Exponents (LEs), which form the Lyapunov spectrum. The LEs have a bearing on stability of RNN training dynamics because forward propagation of information is related to the backward propagation of error gradients. LEs measure the asymptotic rates of expansion and contraction of nonlinear system trajectories, and generalize stability analysis to the time-varying attractors structuring the non-autonomous dynamics of data-driven RNNs. As a tool to understand and exploit stability of training dynamics, the Lyapunov spectrum fills an existing gap between prescriptive mathematical approaches of limited scope and computationally-expensive empirical approaches. To leverage this tool, we implement an efficient way to compute LEs for RNNs during training, discuss the aspects specific to standard RNN architectures driven by typical sequential datasets, and show that the Lyapunov spectrum can serve as a robust readout of training stability across hyperparameters. With this exposition-oriented contribution, we hope to draw attention to this understudied, but theoretically grounded tool for understanding training stability in RNNs.