# Stable Recurrent Models

arXiv: https://arxiv.org/abs/1805.10369 · HTML: https://mlsanity.com/paper/1805.10369/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 May 2018.

## Built on by (most cited first, 30 of 66)

- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [Training Spiking Neural Networks Using Lessons From Deep Learning](https://mlsanity.com/paper/2109.12894/) (2021)
- [Recurrent Equilibrium Networks: Flexible Dynamic Models with Guaranteed Stability and Robustness](https://mlsanity.com/paper/2104.05942/) (2021)
- [Noisy Recurrent Neural Networks](https://mlsanity.com/paper/2102.04877/) (2021)
- [Deep networks for system identification: a Survey](https://mlsanity.com/paper/2301.12832/) (2023)
- [A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases](https://mlsanity.com/paper/2209.11208/) (2022)
- [Fine-Tuning Pre-trained Transformers into Decaying Fast Weights](https://mlsanity.com/paper/2210.04243/) (2022)
- [DeepDiagnosis: Automatically Diagnosing Faults and Recommending Actionable Fixes in Deep Learning Programs](https://mlsanity.com/paper/2112.04036/) (2021)
- [Near-optimal Offline and Streaming Algorithms for Learning Non-Linear Dynamical Systems](https://mlsanity.com/paper/2105.11558/) (2021)
- [Almost Surely Stable Deep Dynamics](https://mlsanity.com/paper/2103.14722/) (2021)
- [DeepFD: Automated Fault Diagnosis and Localization for Deep Learning Programs](https://mlsanity.com/paper/2205.01938/) (2022)
- [Temporally Consistent Koopman Autoencoders for Forecasting Dynamical Systems](https://mlsanity.com/paper/2403.12335/) (2024)
- [Learning over All Stabilizing Nonlinear Controllers for a Partially-Observed Linear System](https://mlsanity.com/paper/2112.04219/) (2021)
- [Homotopy-based training of NeuralODEs for accurate dynamics discovery](https://mlsanity.com/paper/2210.01407/) (2022)
- [Convergent Graph Solvers](https://mlsanity.com/paper/2106.01680/) (2021)
- [An Optimisation Framework for Unsupervised Environment Design](https://mlsanity.com/paper/2505.20659/) (2025)
- [RNNs of RNNs: Recursive Construction of Stable Assemblies of Recurrent Neural Networks](https://mlsanity.com/paper/2106.08928/) (2021)
- [Learning to optimize with convergence guarantees using nonlinear system theory](https://mlsanity.com/paper/2403.09389/) (2024)
- [Predictability Enables Parallelization of Nonlinear State Space Models](https://mlsanity.com/paper/2508.16817/) (2025)
- [Stable Long-Term Recurrent Video Super-Resolution](https://mlsanity.com/paper/2112.08950/) (2021)
- [Learning Over Contracting and Lipschitz Closed-Loops for Partially-Observed Nonlinear Systems (Extended Version)](https://mlsanity.com/paper/2304.06193/) (2023)
- [Adaptive recurrent vision performs zero-shot computation scaling to unseen difficulty levels](https://mlsanity.com/paper/2311.06964/) (2023)
- [On the Memory Mechanism of Tensor-Power Recurrent Models](https://mlsanity.com/paper/2103.01521/) (2021)
- [Recurrent Neural Network-based Internal Model Control design for stable nonlinear systems](https://mlsanity.com/paper/2108.04585/) (2021)
- [Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalability](https://mlsanity.com/paper/2311.04449/) (2023)
- [Pretraining Recurrent Networks without Recurrence](https://mlsanity.com/paper/2606.06479/) (2026)
- [Memory and attention in deep learning](https://mlsanity.com/paper/2107.01390/) (2021)
- [SDYN-GANs: Adversarial Learning Methods for Multistep Generative Models for General Order Stochastic Dynamics](https://mlsanity.com/paper/2302.03663/) (2023)
- [Structured state-space models are deep Wiener models](https://mlsanity.com/paper/2312.06211/) (2023)
- [Fuzzy Recurrent Stochastic Configuration Networks for Industrial Data Analytics](https://mlsanity.com/paper/2407.11038/) (2024)

## Abstract (arXiv)

Stability is a fundamental property of dynamical systems, yet to this date it has had little bearing on the practice of recurrent neural networks. In this work, we conduct a thorough investigation of stable recurrent models. Theoretically, we prove stable recurrent neural networks are well approximated by feed-forward networks for the purpose of both inference and training by gradient descent. Empirically, we demonstrate stable recurrent models often perform as well as their unstable counterparts on benchmark sequence tasks. Taken together, these findings shed light on the effective power of recurrent networks and suggest much of sequence learning happens, or can be made to happen, in the stable regime. Moreover, our results help to explain why in many cases practitioners succeed in replacing recurrent models by feed-forward models.