# A recurrent neural network without chaos

arXiv: https://arxiv.org/abs/1612.06212 · HTML: https://mlsanity.com/paper/1612.06212/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Dec 2016.

## Built on by (most cited first, 13 of 13)

- [UnICORNN: A recurrent model for learning very long time dependencies](https://mlsanity.com/paper/2103.05487/) (2021)
- [Fine-Tuning Pre-trained Transformers into Decaying Fast Weights](https://mlsanity.com/paper/2210.04243/) (2022)
- [Fusing Structure from Motion and Simulation-Augmented Pose Regression from Optical Flow for Challenging Indoor Environments](https://mlsanity.com/paper/2304.07250/) (2023)
- [Recurrent Neural Networks for Learning Long-term Temporal Dependencies with Reanalysis of Time Scale Representation](https://mlsanity.com/paper/2111.03282/) (2021)
- [A Convergence Analysis of Nesterov's Accelerated Gradient Method in Training Deep Linear Neural Networks](https://mlsanity.com/paper/2204.08306/) (2022)
- [Online Stochastic Gradient Descent Learns Linear Dynamical Systems from A Single Trajectory](https://mlsanity.com/paper/2102.11822/) (2021)
- [On Scrambling Phenomena for Randomly Initialized Recurrent Networks](https://mlsanity.com/paper/2210.05212/) (2022)
- [Mamba State-Space Models Are Lyapunov-Stable Learners](https://mlsanity.com/paper/2406.00209/) (2024)
- [Exploiting Chaotic Dynamics as Deep Neural Networks](https://mlsanity.com/paper/2406.02580/) (2024)
- [Impression-Aware Recommender Systems](https://mlsanity.com/paper/2308.07857/) (2023)
- [Hyperpruning: Efficient Search through Pruned Variants of Recurrent Neural Networks Leveraging Lyapunov Spectrum](https://mlsanity.com/paper/2506.07975/) (2025)
- [Algorithm Development in Neural Networks: Insights from the Streaming Parity Task](https://mlsanity.com/paper/2507.09897/) (2025)
- [Divisive Normalization Shapes Low-Rank Slow Manifolds for Continuous Working Memory](https://mlsanity.com/paper/2608.01947/) (2026)

## Abstract (arXiv)

We introduce an exceptionally simple gated recurrent neural network (RNN) that achieves performance comparable to well-known gated architectures, such as LSTMs and GRUs, on the word-level language modeling task. We prove that our model has simple, predicable and non-chaotic dynamics. This stands in stark contrast to more standard gated architectures, whose underlying dynamical systems exhibit chaotic behavior.