mlsanityData to 7 Oct 2026

RNNs of RNNs: Recursive Construction of Stable Assemblies of Recurrent Neural Networks

arXiv 2106.08928 ↗First posted 16 Jun 2021cs.LG math.DS q-bio.NCcited by 7

Not on the map. Why?

Abstract (arXiv)

Recurrent neural networks (RNNs) are widely used throughout neuroscience as models of local neural activity. Many properties of single RNNs are well characterized theoretically, but experimental neuroscience has moved in the direction of studying multiple interacting areas, and RNN theory needs to be likewise extended. We take a constructive approach towards this problem, leveraging tools from nonlinear control theory and machine learning to characterize when combinations of stable RNNs will themselves be stable. Importantly, we derive conditions which allow for massive feedback connections between interacting RNNs. We parameterize these conditions for easy optimization using gradient-based techniques, and show that stability-constrained "networks of networks" can perform well on challenging sequential-processing benchmark tasks. Altogether, our results provide a principled approach towards understanding distributed, modular function in the brain.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 27 of its 94 references that are in the map. First posted on arXiv 16 Jun 2021.

Builds on

27 of 94 references.

References · 27

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2020Hopfield Networks is All You Needcited by 461
  4. 2017Stable Architectures for Deep Neural Networkscited by 340
  5. 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 133
  6. 2015A Simple Way to Initialize Recurrent Networks of Rectified Linear Unitscited by 130
  7. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 117
  8. 2019AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networkscited by 112
  9. 2020Large Associative Memory Problem in Neurobiology and Machine Learningcited by 110
  10. 2020Lipschitz Recurrent Neural Networkscited by 88
  11. 2018The Lyapunov Neural Network: Adaptive Stability Certification for Safe Learning of Dynamical Systemscited by 87
  12. 2021Coordination Among Neural Modules Through a Shared Global Workspacecited by 83
15 more
  1. 2018Stable Recurrent Modelscited by 65
  2. 2018Learning Longer-term Dependencies in RNNs with Auxiliary Lossescited by 58
  3. 2018Trellis Networks for Sequence Modelingcited by 50
  4. 2020Lipschitz Bounded Equilibrium Networkscited by 49
  5. 2019Overparameterized Neural Networks Implement Associative Memorycited by 45
  6. 2020Learning Stability Certificates from Datacited by 45
  7. 2015Control Contraction Metrics: Convex and Intrinsic Criteria for Nonlinear Feedback Designcited by 38
  8. 2020Lyapunov spectra of chaotic recurrent neural networkscited by 37
  9. 2020Deep Learning and the Global Workspace Theorycited by 35
  10. 2020On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Toolscited by 31
  11. 2021Robust Implicit Networks via Non-Euclidean Contractionscited by 19
  12. 2018Beyond Convexity -- Contraction and Global Convergence of Gradient Descentcited by 17
  13. 2019Contracting Implicit Recurrent Neural Networks: Stable Models with Improved Trainabilitycited by 17
  14. 2020An Ode to an ODEcited by 17
  15. 2020Attention as Activationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 3; 2026: 2.

7 papers.

  1. 2024Neural Contractive Dynamical Systemscited by 14
  2. 2025Predictability Enables Parallelization of Nonlinear State Space Modelscited by 7
  3. 2025R2DN: Scalable Parameterization of Contracting and Lipschitz Recurrent Deep Networkscited by 2
  4. 2025Characterizing control between interacting subsystems with deep Jacobian estimationcited by 2
  5. 2026Tuning the burn-in phase in training recurrent neural networks improves their performancecited by 1
  6. 2023Contraction Properties of the Global Workspace Primitivecited by 0
  7. 2026A Nonlinear Separation Principle via Contraction Theory: Applications to Neural Networks, Control, and Learningcited by 0