# Dynamics of learning to integrate in linear recurrent neural networks

arXiv: https://arxiv.org/abs/2503.18754 · HTML: https://mlsanity.com/paper/2503.18754/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Mar 2025.

## Built on by (most cited first, 4 of 4)

- [Learning Dynamics of RNNs in Closed-Loop Environments](https://mlsanity.com/paper/2505.13567/) (2025)
- [The Global Empirical NTK: Self-Referential Bias and Dimensionality of Gradient Descent Learning](https://mlsanity.com/paper/2605.08746/) (2026)
- [KPFlow: An Operator Perspective on Dynamic Collapse Under Gradient Descent Training of Recurrent Networks](https://mlsanity.com/paper/2507.06381/) (2025)
- [Learning reveals invisible structure in low-rank RNNs](https://mlsanity.com/paper/2605.04115/) (2026)

## Abstract (arXiv)

Learning recurrent connectivity that supports memory over long intrinsic timescales is a basic problem in the theory of dynamical computation. While continuous attractor and integrator models describe how tuned recurrent circuits can maintain information, less is known about how such slow modes are acquired by gradient-based learning. Here we study this question in an analytically tractable setting: we build a mathematical theory of the learning dynamics of linear RNNs trained to integrate white noise. We show that when the initial recurrent weights are small, the dynamics of learning are described by a low-dimensional system that tracks a single outlier eigenvalue of the recurrent weights. This reveals the precise manner in which the long timescale associated with white noise integration is learned. We extend our analyses to RNNs learning a damped oscillatory filter, and find low-dimensional effective dynamical equations for the evolution of a conjugate pair of outlier eigenvalues. Taken together, our analyses build a rich mathematical framework for studying dynamical learning problems relevant to both machine learning and neuroscience.