# Learnability Window in Gated Recurrent Neural Networks

arXiv: https://arxiv.org/abs/2512.05790 · HTML: https://mlsanity.com/paper/2512.05790/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 64 references that are in the map.
- First posted on arXiv 5 Dec 2025.

## Builds on (most cited first, 30 of 33)

- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://mlsanity.com/paper/2110.13985/) (2021)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [Unitary Evolution Recurrent Neural Networks](https://mlsanity.com/paper/1511.06464/) (2015)
- [Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer](https://mlsanity.com/paper/2203.03466/) (2022)
- [An Empirical Model of Large-Batch Training](https://mlsanity.com/paper/1812.06162/) (2018)
- [Three Factors Influencing Minima in SGD](https://mlsanity.com/paper/1711.04623/) (2017)
- [A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks](https://mlsanity.com/paper/1901.06053/) (2019)
- [Implicit Gradient Regularization](https://mlsanity.com/paper/2009.11162/) (2020)
- [A Bayesian Perspective on Generalization and Stochastic Gradient Descent](https://mlsanity.com/paper/1710.06451/) (2017)
- [Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice](https://mlsanity.com/paper/1711.04735/) (2017)
- [A Simple Way to Initialize Recurrent Networks of Rectified Linear Units](https://mlsanity.com/paper/1504.00941/) (2015)
- [Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning](https://mlsanity.com/paper/2010.05627/) (2020)
- [AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks](https://mlsanity.com/paper/1902.09689/) (2019)
- [Hierarchical Multiscale Recurrent Neural Networks](https://mlsanity.com/paper/1609.01704/) (2016)
- [Full-Capacity Unitary Recurrent Neural Networks](https://mlsanity.com/paper/1611.00035/) (2016)
- [Are All Layers Created Equal?](https://mlsanity.com/paper/1902.01996/) (2019)
- [Dilated Recurrent Neural Networks](https://mlsanity.com/paper/1710.02224/) (2017)
- [A Clockwork RNN](https://mlsanity.com/paper/1402.3511/) (2014)
- [Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs](https://mlsanity.com/paper/1612.05231/) (2016)
- [Can recurrent neural networks warp time?](https://mlsanity.com/paper/1804.11188/) (2018)
- [From Gradient Clipping to Normalization for Heavy Tailed SGD](https://mlsanity.com/paper/2410.13849/) (2024)
- [Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks](https://mlsanity.com/paper/1806.05394/) (2018)
- [Recurrent Batch Normalization](https://mlsanity.com/paper/1603.09025/) (2016)
- [Theory of gating in recurrent neural networks](https://mlsanity.com/paper/2007.14823/) (2020)

## Abstract (arXiv)

We develop a statistical theory of temporal learnability in recurrent neural networks, quantifying the maximal temporal horizon HN over which gradient-based learning can recover lag-dependent structure at finite sample size N. The theory is built on the effective learning rate envelope f(ℓ), a function that captures how gating mechanisms and adaptive optimizers jointly shape the coupling between state-space dynamics and parameter updates during Backpropagation Through Time. Under heavy-tailed (α-stable) fluctuations, where empirical averages concentrate at rate N-1/kappaα with kappaα = α/(α-1), the interplay between envelope decay and statistical concentration yields explicit scaling laws for the growth of HN: logarithmic, polynomial, and exponential temporal learning regimes emerge according to the decay law of f(ℓ). These results identify envelope decay as the key determinant of temporal learnability. Slower attenuation of f(ℓ) enlarges HN, while heavy-tailed fluctuations compress it by weakening statistical concentration. Moreover, envelope geometry outweighs dataset size: slowing the envelope's decay enlarges HN more than adding data, so more complex architectures that realize slower-decaying envelopes can be more data-efficient than simpler ones. Experiments across multiple gated architectures and optimizers corroborate these structural predictions.