# Low Tensor Rank Learning of Neural Dynamics

arXiv: https://arxiv.org/abs/2308.11567 · HTML: https://mlsanity.com/paper/2308.11567/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 60 references that are in the map.
- First posted on arXiv 22 Aug 2023.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Implicit Regularization in Deep Matrix Factorization](https://mlsanity.com/paper/1905.13655/) (2019)
- [Scalable Gradients for Stochastic Differential Equations](https://mlsanity.com/paper/2001.01328/) (2020)
- [Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning](https://mlsanity.com/paper/1810.01075/) (2018)
- [Universality and individuality in neural dynamics across large populations of recurrent networks](https://mlsanity.com/paper/1907.08549/) (2019)
- [Linking connectivity, dynamics and computations in low-rank recurrent neural networks](https://mlsanity.com/paper/1711.09672/) (2017)
- [Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations](https://mlsanity.com/paper/2205.13571/) (2022)
- [A Hebbian/Anti-Hebbian Neural Network for Linear Subspace Learning: A Derivation from Multidimensional Scaling of Streaming Data](https://mlsanity.com/paper/1503.00669/) (2015)
- [The interplay between randomness and structure during learning in RNNs](https://mlsanity.com/paper/2006.11036/) (2020)
- [Robust alignment of cross-session recordings of neural population activity by behaviour via unsupervised domain adaptation](https://mlsanity.com/paper/2202.06159/) (2022)
- [Connectivity Inference from Neural Recording Data: Challenges, Mathematical Bases and Research Directions](https://mlsanity.com/paper/1708.01888/) (2017)
- [Emergence of functional and structural properties of the head direction system by optimization of recurrent neural networks](https://mlsanity.com/paper/1912.10189/) (2019)
- [Deep inference of latent dynamics with spatio-temporal super-resolution using selective backpropagation through time](https://mlsanity.com/paper/2111.00070/) (2021)
- [A framework for studying synaptic plasticity with neural spike train data](https://mlsanity.com/paper/1411.4077/) (2014)

## Built on by (most cited first, 12 of 12)

- [Provable Emergence of Deep Neural Collapse and Low-Rank Bias in L2-Regularized Nonlinear Networks](https://mlsanity.com/paper/2402.03991/) (2024)
- [Meta-Dynamical State Space Models for Integrative Neural Data Analysis](https://mlsanity.com/paper/2410.05454/) (2024)
- [Fast Generalized Neural Tangent Kernel Statistics via Trace Estimation](https://mlsanity.com/paper/2511.10796/) (2025)
- [Emergent Riemannian geometry over learning discrete computations on continuous manifolds](https://mlsanity.com/paper/2512.00196/) (2025)
- [Curl Descent: Non-Gradient Learning Dynamics with Sign-Diverse Plasticity](https://mlsanity.com/paper/2510.02765/) (2025)
- [The Global Empirical NTK: Self-Referential Bias and Dimensionality of Gradient Descent Learning](https://mlsanity.com/paper/2605.08746/) (2026)
- [Active learning of neural population dynamics using two-photon holographic optogenetics](https://mlsanity.com/paper/2412.02529/) (2024)
- [RNNs perform task computations by dynamically warping neural representations](https://mlsanity.com/paper/2512.04310/) (2025)
- [DCHO: A Decomposition-Composition Framework for Predicting Higher-Order Brain Connectivity to Enhance Diverse Downstream Applications](https://mlsanity.com/paper/2509.09696/) (2025)
- [JEDI: Jointly Embedded Inference of Neural Dynamics](https://mlsanity.com/paper/2603.10489/) (2026)
- [Identifying Connectivity Distributions from Neural Dynamics Using Flows](https://mlsanity.com/paper/2603.26506/) (2026)
- [Center-Manifold Reduction of Learning at Bifurcations: Interference and Rich Learning in Recurrent Neural Networks](https://mlsanity.com/paper/2605.12763/) (2026)

## Abstract (arXiv)

Learning relies on coordinated synaptic changes in recurrently connected populations of neurons. Therefore, understanding the collective evolution of synaptic connectivity over learning is a key challenge in neuroscience and machine learning. In particular, recent work has shown that the weight matrices of task-trained RNNs are typically low rank, but how this low rank structure unfolds over learning is unknown. To address this, we investigate the rank of the 3-tensor formed by the weight matrices throughout learning. By fitting RNNs of varying rank to large-scale neural recordings during a motor learning task, we find that the inferred weights are low-tensor-rank and therefore evolve over a fixed low-dimensional subspace throughout the entire course of learning. We next validate the observation of low-tensor-rank learning on an RNN trained to solve the same task. Finally, we present a set of mathematical results bounding the matrix and tensor ranks of gradient descent learning dynamics which show that low-tensor-rank weights emerge naturally in RNNs trained to solve low-dimensional tasks. Taken together, our findings provide insight on the evolution of population connectivity over learning in both biological and artificial neural networks, and enable reverse engineering of learning-induced changes in recurrent dynamics from large-scale neural recordings.