# Almost-Linear RNNs Yield Highly Interpretable Symbolic Codes in Dynamical Systems Reconstruction

arXiv: https://arxiv.org/abs/2410.14240 · HTML: https://mlsanity.com/paper/2410.14240/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 107 references that are in the map.
- First posted on arXiv 18 Oct 2024.

## Builds on (most cited first, 30 of 36)

- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Deep learning for universal linear embeddings of nonlinear dynamics](https://mlsanity.com/paper/1712.09707/) (2017)
- [Modern Koopman Theory for Dynamical Systems](https://mlsanity.com/paper/2102.12086/) (2021)
- [Contemporary Symbolic Regression Methods and their Relative Performance](https://mlsanity.com/paper/2107.14351/) (2021)
- [Data-Driven Forecasting of High-Dimensional Chaotic Systems with Long Short-Term Memory Networks](https://mlsanity.com/paper/1802.07486/) (2018)
- [Linearly-Recurrent Autoencoder Networks for Learning Dynamics](https://mlsanity.com/paper/1712.01378/) (2017)
- [Sparse identification of nonlinear dynamics for model predictive control in the low-data limit](https://mlsanity.com/paper/1711.05501/) (2017)
- [Backpropagation Algorithms and Reservoir Computing in Recurrent Neural Networks for the Forecasting of Complex Spatiotemporal Dynamics](https://mlsanity.com/paper/1910.05266/) (2019)
- [Forecasting Sequential Data using Consistent Koopman Autoencoders](https://mlsanity.com/paper/2003.02236/) (2020)
- [Transformers for Modeling Physical Systems](https://mlsanity.com/paper/2010.03957/) (2020)
- [Interpretable Scientific Discovery with Symbolic Regression: A Review](https://mlsanity.com/paper/2211.10873/) (2022)
- [On the difficulty of learning chaotic dynamics with RNNs](https://mlsanity.com/paper/2110.07238/) (2021)
- [Generalized Teacher Forcing for Learning Chaotic Dynamics](https://mlsanity.com/paper/2306.04406/) (2023)
- [Identifying nonlinear dynamical systems via generative recurrent neural networks with applications to fMRI](https://mlsanity.com/paper/1902.07186/) (2019)
- [Long Expressive Memory for Sequence Modeling](https://mlsanity.com/paper/2110.04744/) (2021)
- [Automatic Differentiation to Simultaneously Identify Nonlinear Dynamics and Extract Noise Probability Distributions from Data](https://mlsanity.com/paper/2009.08810/) (2020)
- [Koopman Neural Forecaster for Time Series with Temporal Distribution Shifts](https://mlsanity.com/paper/2210.03675/) (2022)
- [Tractable Dendritic RNNs for Reconstructing Nonlinear Dynamical Systems](https://mlsanity.com/paper/2207.02542/) (2022)
- [Recurrent switching linear dynamical systems](https://mlsanity.com/paper/1610.08466/) (2016)
- [On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools](https://mlsanity.com/paper/2006.14123/) (2020)
- [A Systematic Exploration of Reservoir Computing for Forecasting Complex Spatiotemporal Dynamics](https://mlsanity.com/paper/2201.08910/) (2022)
- [A State Space Approach for Piecewise-Linear Recurrent Neural Networks for Reconstructing Nonlinear Dynamics from Neural Measurements](https://mlsanity.com/paper/1612.07846/) (2016)
- [Sparse Identification of Nonlinear Dynamical Systems via Reweighted ℓ1-regularized Least Squares](https://mlsanity.com/paper/2005.13232/) (2020)
- [DyNODE: Neural Ordinary Differential Equations for Dynamics Modeling in Continuous Control](https://mlsanity.com/paper/2009.04278/) (2020)
- [Bifurcations and loss jumps in RNN training](https://mlsanity.com/paper/2310.17561/) (2023)
- [Synthesis of recurrent neural networks for dynamical system simulation](https://mlsanity.com/paper/1512.05702/) (2015)
- [Reconstructing Nonlinear Dynamical Systems from Multi-Modal Time Series](https://mlsanity.com/paper/2111.02922/) (2021)
- [Homotopy-based training of NeuralODEs for accurate dynamics discovery](https://mlsanity.com/paper/2210.01407/) (2022)
- [Integrating Multimodal Data for Joint Generative Modeling of Complex Dynamics](https://mlsanity.com/paper/2212.07892/) (2022)

## Built on by (most cited first, 19 of 19)

- [True Zero-Shot Inference of Dynamical Systems Preserving Long-Term Statistics](https://mlsanity.com/paper/2505.13192/) (2025)
- [Context parroting: A simple but tough-to-beat baseline for foundation models in scientific machine learning](https://mlsanity.com/paper/2505.11349/) (2025)
- [Panda: A pretrained forecast model for chaotic dynamics](https://mlsanity.com/paper/2505.13755/) (2025)
- [Uncovering the Computational Roles of Nonlinearity in Sequence Modeling Using Almost-Linear RNNs](https://mlsanity.com/paper/2506.07919/) (2025)
- [ATLAS: Active Theory Learning for Automated Science](https://mlsanity.com/paper/2606.12386/) (2026)
- [Continuous-Time Piecewise-Linear Recurrent Neural Networks](https://mlsanity.com/paper/2602.15649/) (2026)
- [Position: A Dynamical Systems Perspective is Needed to Advance Time Series Modeling](https://mlsanity.com/paper/2602.16864/) (2026)
- [Detecting Invariant Manifolds in ReLU-Based RNNs](https://mlsanity.com/paper/2510.03814/) (2025)
- [Neural Functions for Learning Periodic Signal](https://mlsanity.com/paper/2506.09526/) (2025)
- [Teacher Forcing as Generalized Bayes: Optimization Geometry Mismatch in Switching Surrogates for Chaotic Dynamics](https://mlsanity.com/paper/2604.25904/) (2026)
- [Modeling Neural Activity with Conditionally Linear Dynamical Systems](https://mlsanity.com/paper/2502.18347/) (2025)
- [The Dynamic-Probabilistic Consistency Gap in Chaotic Surrogate Modeling](https://mlsanity.com/paper/2605.31547/) (2026)
- [Parallel-in-Time Training of Recurrent Neural Networks for Dynamical Systems Reconstruction](https://mlsanity.com/paper/2605.12683/) (2026)
- [AugRelNet: Relation-Augmented Dynamics for Structure Discovery and Forecasting from Limited Data](https://mlsanity.com/paper/2606.11251/) (2026)
- [Topological Out-of-Domain Generalization in Dynamical Systems Reconstruction](https://mlsanity.com/paper/2606.22969/) (2026)
- [A Minimal Interpretable Architecture for Zero-Shot Reconstruction of Dynamical Systems](https://mlsanity.com/paper/2607.14937/) (2026)
- [Continual Learning of Dynamical Systems in Recurrent Neural Networks through Recyclable Unit Gating](https://mlsanity.com/paper/2609.38356/) (2026)
- [From Redundancy to Minimality: Fixed-Point-Guided Hierarchical Reduction of Learned Piecewise-Linear Dynamics](https://mlsanity.com/paper/2610.01369/) (2026)
- [Learning Transferable Policies from Action-free Time Series Through Dynamical Embeddings](https://mlsanity.com/paper/2610.03065/) (2026)

## Abstract (arXiv)

Dynamical systems (DS) theory is fundamental for many areas of science and engineering. It can provide deep insights into the behavior of systems evolving in time, as typically described by differential or recursive equations. A common approach to facilitate mathematical tractability and interpretability of DS models involves decomposing nonlinear DS into multiple linear DS separated by switching manifolds, i.e. piecewise linear (PWL) systems. PWL models are popular in engineering and a frequent choice in mathematics for analyzing the topological properties of DS. However, hand-crafting such models is tedious and only possible for very low-dimensional scenarios, while inferring them from data usually gives rise to unnecessarily complex representations with very many linear subregions. Here we introduce Almost-Linear Recurrent Neural Networks (AL-RNNs) which automatically and robustly produce most parsimonious PWL representations of DS from time series data, using as few PWL nonlinearities as possible. AL-RNNs can be efficiently trained with any SOTA algorithm for dynamical systems reconstruction (DSR), and naturally give rise to a symbolic encoding of the underlying DS that provably preserves important topological properties. We show that for the Lorenz and R"ossler systems, AL-RNNs discover, in a purely data-driven way, the known topologically minimal PWL representations of the corresponding chaotic attractors. We further illustrate on two challenging empirical datasets that interpretable symbolic encodings of the dynamics can be achieved, tremendously facilitating mathematical and computational analysis of the underlying systems.