# The interplay between randomness and structure during learning in RNNs

arXiv: https://arxiv.org/abs/2006.11036 · HTML: https://mlsanity.com/paper/2006.11036/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jun 2020.

## Built on by (most cited first, 20 of 20)

- [A Rainbow in Deep Network Black Boxes](https://mlsanity.com/paper/2305.18512/) (2023)
- [How connectivity structure shapes rich and lazy learning in neural circuits](https://mlsanity.com/paper/2310.08513/) (2023)
- [Low Tensor Rank Learning of Neural Dynamics](https://mlsanity.com/paper/2308.11567/) (2023)
- [Provable Emergence of Deep Neural Collapse and Low-Rank Bias in L2-Regularized Nonlinear Networks](https://mlsanity.com/paper/2402.03991/) (2024)
- [Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rules](https://mlsanity.com/paper/2206.00823/) (2022)
- [Bayesian reconstruction of memories stored in neural networks from their connectivity](https://mlsanity.com/paper/2105.07416/) (2021)
- [Leveraging Low-Rank and Sparse Recurrent Connectivity for Robust Closed-Loop Control](https://mlsanity.com/paper/2310.03915/) (2023)
- [Learning Dynamics of RNNs in Closed-Loop Environments](https://mlsanity.com/paper/2505.13567/) (2025)
- [The Global Empirical NTK: Self-Referential Bias and Dimensionality of Gradient Descent Learning](https://mlsanity.com/paper/2605.08746/) (2026)
- [The impact of memory on learning sequence-to-sequence tasks](https://mlsanity.com/paper/2205.14683/) (2022)
- [A ghost mechanism: An analytical model of abrupt learning in recurrent networks](https://mlsanity.com/paper/2501.02378/) (2025)
- [Can Biologically Plausible Temporal Credit Assignment Rules Match BPTT for Neural Similarity? E-prop as an Example](https://mlsanity.com/paper/2506.06904/) (2025)
- [KPFlow: An Operator Perspective on Dynamic Collapse Under Gradient Descent Training of Recurrent Networks](https://mlsanity.com/paper/2507.06381/) (2025)
- [A Factorized Low-Rank RNN Framework for Uncovering Independent Neural Latent Dynamics and Connectivity](https://mlsanity.com/paper/2511.13899/) (2025)
- [RNNs perform task computations by dynamically warping neural representations](https://mlsanity.com/paper/2512.04310/) (2025)
- [Learning reveals invisible structure in low-rank RNNs](https://mlsanity.com/paper/2605.04115/) (2026)
- [Memory Determines Learning Direction: A Theory of Gradient-Based Optimization in State Space Models](https://mlsanity.com/paper/2510.00563/) (2025)
- [Center-Manifold Reduction of Learning at Bifurcations: Interference and Rich Learning in Recurrent Neural Networks](https://mlsanity.com/paper/2605.12763/) (2026)
- [Dynamics and Representation Structure of Local Approximations to Gradient-Based Learning in Linear Recurrent Neural Networks](https://mlsanity.com/paper/2606.00243/) (2026)
- [Harnessing cortical geometry, wiring, and function as inductive biases for recurrent neural networks](https://mlsanity.com/paper/2606.14975/) (2026)

## Abstract (arXiv)

Recurrent neural networks (RNNs) trained on low-dimensional tasks have been widely used to model functional biological networks. However, the solutions found by learning and the effect of initial connectivity are not well understood. Here, we examine RNNs trained using gradient descent on different tasks inspired by the neuroscience literature. We find that the changes in recurrent connectivity can be described by low-rank matrices, despite the unconstrained nature of the learning algorithm. To identify the origin of the low-rank structure, we turn to an analytically tractable setting: training a linear RNN on a simplified task. We show how the low-dimensional task structure leads to low-rank changes to connectivity. This low-rank structure allows us to explain and quantify the phenomenon of accelerated learning in the presence of random initial connectivity. Altogether, our study opens a new perspective to understanding trained RNNs in terms of both the learning process and the resulting network structure.