# RNNs of RNNs: Recursive Construction of Stable Assemblies of Recurrent Neural Networks

arXiv: https://arxiv.org/abs/2106.08928 · HTML: https://mlsanity.com/paper/2106.08928/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 94 references that are in the map.
- First posted on arXiv 16 Jun 2021.

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group](https://mlsanity.com/paper/1901.08428/) (2019)
- [A Simple Way to Initialize Recurrent Networks of Rectified Linear Units](https://mlsanity.com/paper/1504.00941/) (2015)
- [CKConv: Continuous Kernel Convolution For Sequential Data](https://mlsanity.com/paper/2102.02611/) (2021)
- [AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks](https://mlsanity.com/paper/1902.09689/) (2019)
- [Large Associative Memory Problem in Neurobiology and Machine Learning](https://mlsanity.com/paper/2008.06996/) (2020)
- [Lipschitz Recurrent Neural Networks](https://mlsanity.com/paper/2006.12070/) (2020)
- [The Lyapunov Neural Network: Adaptive Stability Certification for Safe Learning of Dynamical Systems](https://mlsanity.com/paper/1808.00924/) (2018)
- [Coordination Among Neural Modules Through a Shared Global Workspace](https://mlsanity.com/paper/2103.01197/) (2021)
- [Stable Recurrent Models](https://mlsanity.com/paper/1805.10369/) (2018)
- [Learning Longer-term Dependencies in RNNs with Auxiliary Losses](https://mlsanity.com/paper/1803.00144/) (2018)
- [Trellis Networks for Sequence Modeling](https://mlsanity.com/paper/1810.06682/) (2018)
- [Lipschitz Bounded Equilibrium Networks](https://mlsanity.com/paper/2010.01732/) (2020)
- [Overparameterized Neural Networks Implement Associative Memory](https://mlsanity.com/paper/1909.12362/) (2019)
- [Learning Stability Certificates from Data](https://mlsanity.com/paper/2008.05952/) (2020)
- [Control Contraction Metrics: Convex and Intrinsic Criteria for Nonlinear Feedback Design](https://mlsanity.com/paper/1503.03144/) (2015)
- [Lyapunov spectra of chaotic recurrent neural networks](https://mlsanity.com/paper/2006.02427/) (2020)
- [Deep Learning and the Global Workspace Theory](https://mlsanity.com/paper/2012.10390/) (2020)
- [On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools](https://mlsanity.com/paper/2006.14123/) (2020)
- [Robust Implicit Networks via Non-Euclidean Contractions](https://mlsanity.com/paper/2106.03194/) (2021)
- [Beyond Convexity -- Contraction and Global Convergence of Gradient Descent](https://mlsanity.com/paper/1806.06655/) (2018)
- [Contracting Implicit Recurrent Neural Networks: Stable Models with Improved Trainability](https://mlsanity.com/paper/1912.10402/) (2019)
- [An Ode to an ODE](https://mlsanity.com/paper/2006.11421/) (2020)
- [Attention as Activation](https://mlsanity.com/paper/2007.07729/) (2020)

## Built on by (most cited first, 7 of 7)

- [Neural Contractive Dynamical Systems](https://mlsanity.com/paper/2401.09352/) (2024)
- [Predictability Enables Parallelization of Nonlinear State Space Models](https://mlsanity.com/paper/2508.16817/) (2025)
- [R2DN: Scalable Parameterization of Contracting and Lipschitz Recurrent Deep Networks](https://mlsanity.com/paper/2504.01250/) (2025)
- [Characterizing control between interacting subsystems with deep Jacobian estimation](https://mlsanity.com/paper/2507.01946/) (2025)
- [Tuning the burn-in phase in training recurrent neural networks improves their performance](https://mlsanity.com/paper/2602.10911/) (2026)
- [Contraction Properties of the Global Workspace Primitive](https://mlsanity.com/paper/2310.01571/) (2023)
- [A Nonlinear Separation Principle via Contraction Theory: Applications to Neural Networks, Control, and Learning](https://mlsanity.com/paper/2604.15238/) (2026)

## Abstract (arXiv)

Recurrent neural networks (RNNs) are widely used throughout neuroscience as models of local neural activity. Many properties of single RNNs are well characterized theoretically, but experimental neuroscience has moved in the direction of studying multiple interacting areas, and RNN theory needs to be likewise extended. We take a constructive approach towards this problem, leveraging tools from nonlinear control theory and machine learning to characterize when combinations of stable RNNs will themselves be stable. Importantly, we derive conditions which allow for massive feedback connections between interacting RNNs. We parameterize these conditions for easy optimization using gradient-based techniques, and show that stability-constrained "networks of networks" can perform well on challenging sequential-processing benchmark tasks. Altogether, our results provide a principled approach towards understanding distributed, modular function in the brain.