# Theory of gating in recurrent neural networks

arXiv: https://arxiv.org/abs/2007.14823 · HTML: https://mlsanity.com/paper/2007.14823/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jul 2020.

## Built on by (most cited first, 19 of 19)

- [Generative learning for nonlinear dynamics](https://mlsanity.com/paper/2311.04128/) (2023)
- [The Influence of Learning Rule on Representation Dynamics in Wide Neural Networks](https://mlsanity.com/paper/2210.02157/) (2022)
- [Introduction to dynamical mean-field theory of randomly connected neural networks with bidirectionally correlated couplings](https://mlsanity.com/paper/2305.08459/) (2023)
- [Gradient Flossing: Improving Gradient Descent through Dynamic Control of Jacobians](https://mlsanity.com/paper/2312.17306/) (2023)
- [Path sampling of recurrent neural networks by incorporating known physics](https://mlsanity.com/paper/2203.00597/) (2022)
- [Spectrum of non-Hermitian deep-Hebbian neural networks](https://mlsanity.com/paper/2208.11411/) (2022)
- [Neural timescales from a computational perspective](https://mlsanity.com/paper/2409.02684/) (2024)
- [Parallelizable memory recurrent units](https://mlsanity.com/paper/2601.09495/) (2026)
- [Abrupt and spontaneous strategy switches emerge in simple regularised neural networks](https://mlsanity.com/paper/2302.11351/) (2023)
- [Complexity-calibrated Benchmarks for Machine Learning Reveal When Next-Generation Reservoir Computer Predictions Succeed and Mislead](https://mlsanity.com/paper/2303.14553/) (2023)
- [Trainability, Expressivity and Interpretability in Gated Neural ODEs](https://mlsanity.com/paper/2307.06398/) (2023)
- [Phase codes emerge in recurrent neural networks optimized for modular arithmetic](https://mlsanity.com/paper/2310.07908/) (2023)
- [Time-Scale Coupling Between States and Parameters in Recurrent Neural Networks](https://mlsanity.com/paper/2508.12121/) (2025)
- [Hybrid Associative Memories](https://mlsanity.com/paper/2603.22325/) (2026)
- [Synergistic pathways of modulation enable robust task packing within neural dynamics](https://mlsanity.com/paper/2408.01316/) (2024)
- [Neuronal correlations shape the scaling behavior of memory capacity and nonlinear computational capability of reservoir recurrent neural networks](https://mlsanity.com/paper/2504.19657/) (2025)
- [Learnability Window in Gated Recurrent Neural Networks](https://mlsanity.com/paper/2512.05790/) (2025)
- [Visualising Information Flow in Word Embeddings with Diffusion Tensor Imaging](https://mlsanity.com/paper/2601.05713/) (2026)
- [Gating Enables Curvature: A Geometric Expressivity Gap in Attention](https://mlsanity.com/paper/2604.14702/) (2026)

## Abstract (arXiv)

Recurrent neural networks (RNNs) are powerful dynamical models, widely used in machine learning (ML) and neuroscience. Prior theoretical work has focused on RNNs with additive interactions. However, gating - i.e. multiplicative - interactions are ubiquitous in real neurons and also the central feature of the best-performing RNNs in ML. Here, we show that gating offers flexible control of two salient features of the collective dynamics: i) timescales and ii) dimensionality. The gate controlling timescales leads to a novel, marginally stable state, where the network functions as a flexible integrator. Unlike previous approaches, gating permits this important function without parameter fine-tuning or special symmetries. Gates also provide a flexible, context-dependent mechanism to reset the memory trace, thus complementing the memory function. The gate modulating the dimensionality can induce a novel, discontinuous chaotic transition, where inputs push a stable system to strong chaotic activity, in contrast to the typically stabilizing effect of inputs. At this transition, unlike additive RNNs, the proliferation of critical points (topological complexity) is decoupled from the appearance of chaotic dynamics (dynamical complexity). The rich dynamics are summarized in phase diagrams, thus providing a map for principled parameter initialization choices to ML practitioners.