# Generalized Teacher Forcing for Learning Chaotic Dynamics

arXiv: https://arxiv.org/abs/2306.04406 · HTML: https://mlsanity.com/paper/2306.04406/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 88 references that are in the map.
- First posted on arXiv 7 Jun 2023.

## Builds on (most cited first, 30 of 33)

- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Deep learning for universal linear embeddings of nonlinear dynamics](https://mlsanity.com/paper/1712.09707/) (2017)
- [Unitary Evolution Recurrent Neural Networks](https://mlsanity.com/paper/1511.06464/) (2015)
- [Modern Koopman Theory for Dynamical Systems](https://mlsanity.com/paper/2102.12086/) (2021)
- [Analyzing Inverse Problems with Invertible Neural Networks](https://mlsanity.com/paper/1808.04730/) (2018)
- [Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations](https://mlsanity.com/paper/1801.06637/) (2018)
- [Data-Driven Forecasting of High-Dimensional Chaotic Systems with Long Short-Term Memory Networks](https://mlsanity.com/paper/1802.07486/) (2018)
- [Linearly-Recurrent Autoencoder Networks for Learning Dynamics](https://mlsanity.com/paper/1712.01378/) (2017)
- [Backpropagation Algorithms and Reservoir Computing in Recurrent Neural Networks for the Forecasting of Complex Spatiotemporal Dynamics](https://mlsanity.com/paper/1910.05266/) (2019)
- [AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks](https://mlsanity.com/paper/1902.09689/) (2019)
- [Lipschitz Recurrent Neural Networks](https://mlsanity.com/paper/2006.12070/) (2020)
- [PySINDy: A comprehensive Python package for robust sparse system identification](https://mlsanity.com/paper/2111.08481/) (2021)
- [Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependencies](https://mlsanity.com/paper/2010.00951/) (2020)
- [On the difficulty of learning chaotic dynamics with RNNs](https://mlsanity.com/paper/2110.07238/) (2021)
- [Discovering Governing Equations from Partial Measurements with Deep Delay Autoencoders](https://mlsanity.com/paper/2201.05136/) (2022)
- [Generalizing to New Physical Systems via Context-Informed Dynamics Model](https://mlsanity.com/paper/2202.01889/) (2022)
- [Identifying nonlinear dynamical systems via generative recurrent neural networks with applications to fMRI](https://mlsanity.com/paper/1902.07186/) (2019)
- [Long Expressive Memory for Sequence Modeling](https://mlsanity.com/paper/2110.04744/) (2021)
- [Lyapunov spectra of chaotic recurrent neural networks](https://mlsanity.com/paper/2006.02427/) (2020)
- [Tractable Dendritic RNNs for Reconstructing Nonlinear Dynamical Systems](https://mlsanity.com/paper/2207.02542/) (2022)
- [Using Machine Learning to Anticipate Tipping Points and Extrapolate to Post-Tipping Dynamics of Non-Stationary Dynamical Systems](https://mlsanity.com/paper/2207.00521/) (2022)
- [A State Space Approach for Piecewise-Linear Recurrent Neural Networks for Reconstructing Nonlinear Dynamics from Neural Measurements](https://mlsanity.com/paper/1612.07846/) (2016)
- [DySMHO: Data-Driven Discovery of Governing Equations for Dynamical Systems via Moving Horizon Optimization](https://mlsanity.com/paper/2108.00069/) (2021)
- [Reverse engineering recurrent neural networks with Jacobian switching linear dynamical systems](https://mlsanity.com/paper/2111.01256/) (2021)
- [Reconstructing Nonlinear Dynamical Systems from Multi-Modal Time Series](https://mlsanity.com/paper/2111.02922/) (2021)

## Built on by (most cited first, 30 of 58)

- [Out-of-Domain Generalization in Dynamical Systems Reconstruction](https://mlsanity.com/paper/2402.18377/) (2024)
- [DySLIM: Dynamics Stable Learning by Invariant Measure for Chaotic Systems](https://mlsanity.com/paper/2402.04467/) (2024)
- [Zero-shot forecasting of chaotic systems](https://mlsanity.com/paper/2409.15771/) (2024)
- [True Zero-Shot Inference of Dynamical Systems Preserving Long-Term Statistics](https://mlsanity.com/paper/2505.13192/) (2025)
- [Almost-Linear RNNs Yield Highly Interpretable Symbolic Codes in Dynamical Systems Reconstruction](https://mlsanity.com/paper/2410.14240/) (2024)
- [Context parroting: A simple but tough-to-beat baseline for foundation models in scientific machine learning](https://mlsanity.com/paper/2505.11349/) (2025)
- [Bifurcations and loss jumps in RNN training](https://mlsanity.com/paper/2310.17561/) (2023)
- [Attractor Memory for Long-Term Time Series Forecasting: A Chaos Perspective](https://mlsanity.com/paper/2402.11463/) (2024)
- [Inferring stochastic low-rank recurrent neural networks from neural data](https://mlsanity.com/paper/2406.16749/) (2024)
- [Learning Interpretable Hierarchical Dynamical Systems Models from Time Series Data](https://mlsanity.com/paper/2410.04814/) (2024)
- [When are dynamical systems learned from time series data statistically accurate?](https://mlsanity.com/paper/2411.06311/) (2024)
- [A Practical Guide to Sample-based Statistical Distances for Evaluating Generative Models in Science](https://mlsanity.com/paper/2403.12636/) (2024)
- [Panda: A pretrained forecast model for chaotic dynamics](https://mlsanity.com/paper/2505.13755/) (2025)
- [Latent Diffusion for Neural Spiking Data](https://mlsanity.com/paper/2407.08751/) (2024)
- [A scalable generative model for dynamical system reconstruction from neuroimaging data](https://mlsanity.com/paper/2411.02949/) (2024)
- [Optimal Recurrent Network Topologies for Dynamical Systems Reconstruction](https://mlsanity.com/paper/2406.04934/) (2024)
- [Koopman-informed recurrent neural networks](https://mlsanity.com/paper/2410.23467/) (2024)
- [A call for embodied AI](https://mlsanity.com/paper/2402.03824/) (2024)
- [Towards Foundational Models for Dynamical System Reconstruction: Hierarchical Meta-Learning via Mixture of Experts](https://mlsanity.com/paper/2502.05335/) (2025)
- [Continuous-Time Piecewise-Linear Recurrent Neural Networks](https://mlsanity.com/paper/2602.15649/) (2026)
- [Position: A Dynamical Systems Perspective is Needed to Advance Time Series Modeling](https://mlsanity.com/paper/2602.16864/) (2026)
- [Learning Chaotic Systems and Long-Term Predictions with Neural Jump ODEs](https://mlsanity.com/paper/2407.18808/) (2024)
- [Toward Physics-guided Time Series Embedding](https://mlsanity.com/paper/2410.06651/) (2024)
- [A Dynamical Systems Perspective on the Analysis of Neural Networks](https://mlsanity.com/paper/2507.05164/) (2025)
- [Detecting Invariant Manifolds in ReLU-Based RNNs](https://mlsanity.com/paper/2510.03814/) (2025)
- [Is Flow Matching Just Trajectory Replay for Sequential Data?](https://mlsanity.com/paper/2602.08318/) (2026)
- [MENO: MeanFlow-Enhanced Neural Operators for Dynamical Systems](https://mlsanity.com/paper/2604.06881/) (2026)
- [RefreshNet: Learning Multiscale Dynamics through Hierarchical Refreshing](https://mlsanity.com/paper/2401.13282/) (2024)
- [Learning Beyond Experience: Generalizing to Unseen State Space with Reservoir Computing](https://mlsanity.com/paper/2506.05292/) (2025)
- [Neural Functions for Learning Periodic Signal](https://mlsanity.com/paper/2506.09526/) (2025)

## Abstract (arXiv)

Chaotic dynamical systems (DS) are ubiquitous in nature and society. Often we are interested in reconstructing such systems from observed time series for prediction or mechanistic insight, where by reconstruction we mean learning geometrical and invariant temporal properties of the system in question (like attractors). However, training reconstruction algorithms like recurrent neural networks (RNNs) on such systems by gradient-descent based techniques faces severe challenges. This is mainly due to exploding gradients caused by the exponential divergence of trajectories in chaotic systems. Moreover, for (scientific) interpretability we wish to have as low dimensional reconstructions as possible, preferably in a model which is mathematically tractable. Here we report that a surprisingly simple modification of teacher forcing leads to provably strictly all-time bounded gradients in training on chaotic systems, and, when paired with a simple architectural rearrangement of a tractable RNN design, piecewise-linear RNNs (PLRNNs), allows for faithful reconstruction in spaces of at most the dimensionality of the observed system. We show on several DS that with these amendments we can reconstruct DS better than current SOTA algorithms, in much lower dimensions. Performance differences were particularly compelling on real world data with which most other methods severely struggled. This work thus led to a simple yet powerful DS reconstruction algorithm which is highly interpretable at the same time.