# Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1806.05394 · HTML: https://mlsanity.com/paper/1806.05394/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jun 2018.

## Built on by (most cited first, 30 of 45)

- [An Empirical Investigation of the Role of Pre-training in Lifelong Learning](https://mlsanity.com/paper/2112.09153/) (2021)
- [Barren Plateaus in Variational Quantum Computing](https://mlsanity.com/paper/2405.00781/) (2024)
- [Noisy Recurrent Neural Networks](https://mlsanity.com/paper/2102.04877/) (2021)
- [Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn features](https://mlsanity.com/paper/2212.13881/) (2022)
- [Recurrent neural networks: vanishing and exploding gradients are not the end of the story](https://mlsanity.com/paper/2405.21064/) (2024)
- [The edge of chaos: quantum field theory and deep neural networks](https://mlsanity.com/paper/2109.13247/) (2021)
- [The Law of Parsimony in Gradient Descent for Learning Deep Linear Networks](https://mlsanity.com/paper/2306.01154/) (2023)
- [Lightweight Online Adaption for Time Series Foundation Model Forecasts](https://mlsanity.com/paper/2502.12920/) (2025)
- [NoT: Federated Unlearning via Weight Negation](https://mlsanity.com/paper/2503.05657/) (2025)
- [Towards quantifying information flows: relative entropy in deep neural networks and the renormalization group](https://mlsanity.com/paper/2107.06898/) (2021)
- [Implicit Bias of Linear RNNs](https://mlsanity.com/paper/2101.07833/) (2021)
- [Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applications](https://mlsanity.com/paper/2111.12143/) (2021)
- [Analysis of Convolutions, Non-linearity and Depth in Graph Neural Networks using Neural Tangent Kernel](https://mlsanity.com/paper/2210.09809/) (2022)
- [On the Neural Tangent Kernel of Equilibrium Models](https://mlsanity.com/paper/2310.14062/) (2023)
- [Persistent learning signals and working memory without continuous attractors](https://mlsanity.com/paper/2308.12585/) (2023)
- [AutoInit: Analytic Signal-Preserving Weight Initialization for Neural Networks](https://mlsanity.com/paper/2109.08958/) (2021)
- [Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networks](https://mlsanity.com/paper/2210.01245/) (2022)
- [On the Initialisation of Wide Low-Rank Feedforward Neural Networks](https://mlsanity.com/paper/2301.13710/) (2023)
- [Gradient Flossing: Improving Gradient Descent through Dynamic Control of Jacobians](https://mlsanity.com/paper/2312.17306/) (2023)
- [Deep Neural Network Initialization with Sparsity Inducing Activations](https://mlsanity.com/paper/2402.16184/) (2024)
- [Infinite-Width Limit of a Single Attention Layer: Analysis via Tensor Programs](https://mlsanity.com/paper/2506.00846/) (2025)
- [Preprint: Norm Loss: An efficient yet effective regularization method for deep neural networks](https://mlsanity.com/paper/2103.06583/) (2021)
- [Recency Dropout for Recurrent Recommender Systems](https://mlsanity.com/paper/2201.11016/) (2022)
- [Lyapunov-Guided Representation of Recurrent Neural Network Performance](https://mlsanity.com/paper/2204.04876/) (2022)
- [Criticality versus uniformity in deep neural networks](https://mlsanity.com/paper/2304.04784/) (2023)
- [Efficient Compression of Overparameterized Deep Models through Low-Dimensional Learning Dynamics](https://mlsanity.com/paper/2311.05061/) (2023)
- [Sparser, Better, Deeper, Stronger: Improving Sparse Training with Exact Orthogonal Initialization](https://mlsanity.com/paper/2406.01755/) (2024)
- [Statistical Physics of Deep Neural Networks: Initialization toward Optimal Channels](https://mlsanity.com/paper/2212.01744/) (2022)
- [Revisiting Glorot Initialization for Long-Range Linear Recurrences](https://mlsanity.com/paper/2505.19827/) (2025)
- [Time-Scale Coupling Between States and Parameters in Recurrent Neural Networks](https://mlsanity.com/paper/2508.12121/) (2025)

## Abstract (arXiv)

Recurrent neural networks have gained widespread use in modeling sequence data across various domains. While many successful recurrent architectures employ a notion of gating, the exact mechanism that enables such remarkable performance is not well understood. We develop a theory for signal propagation in recurrent networks after random initialization using a combination of mean field theory and random matrix theory. To simplify our discussion, we introduce a new RNN cell with a simple gating mechanism that we call the minimalRNN and compare it with vanilla RNNs. Our theory allows us to define a maximum timescale over which RNNs can remember an input. We show that this theory predicts trainability for both recurrent architectures. We show that gated recurrent networks feature a much broader, more robust, trainable region than vanilla RNNs, which corroborates recent experimental findings. Finally, we develop a closed-form critical initialization scheme that achieves dynamical isometry in both vanilla RNNs and minimalRNNs. We show that this results in significantly improvement in training dynamics. Finally, we demonstrate that the minimalRNN achieves comparable performance to its more complex counterparts, such as LSTMs or GRUs, on a language modeling task.