# Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers

arXiv: https://arxiv.org/abs/1910.05505 · HTML: https://mlsanity.com/paper/1910.05505/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Oct 2019.

## Built on by (most cited first, 30 of 61)

- [Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction](https://mlsanity.com/paper/2106.15013/) (2021)
- [Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flows](https://mlsanity.com/paper/2307.00144/) (2023)
- [Symmetry in Neural Network Parameter Spaces](https://mlsanity.com/paper/2506.13018/) (2025)
- [Geometry of Linear Convolutional Networks](https://mlsanity.com/paper/2108.01538/) (2021)
- [Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks](https://mlsanity.com/paper/2201.11729/) (2022)
- [The loss landscape of deep linear neural networks: a second-order analysis](https://mlsanity.com/paper/2107.13289/) (2021)
- [Convergence of gradient descent for learning linear neural networks](https://mlsanity.com/paper/2108.02040/) (2021)
- [On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensions](https://mlsanity.com/paper/2311.03794/) (2023)
- [Infinite-width limit of deep linear neural networks](https://mlsanity.com/paper/2211.16980/) (2022)
- [Function Space and Critical Points of Linear Convolutional Networks](https://mlsanity.com/paper/2304.05752/) (2023)
- [Optimization Dynamics of Equivariant and Augmented Neural Networks](https://mlsanity.com/paper/2303.13458/) (2023)
- [Neural (Tangent Kernel) Collapse](https://mlsanity.com/paper/2305.16427/) (2023)
- [Plastic Learning with Deep Fourier Features](https://mlsanity.com/paper/2410.20634/) (2024)
- [Robust low-rank training via approximate orthonormal constraints](https://mlsanity.com/paper/2306.01485/) (2023)
- [Robust Implicit Regularization via Weight Normalization](https://mlsanity.com/paper/2305.05448/) (2023)
- [Geometry-aware training of factorized layers in tensor Tucker format](https://mlsanity.com/paper/2305.19059/) (2023)
- [Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation](https://mlsanity.com/paper/2107.04479/) (2021)
- [Provable Emergence of Deep Neural Collapse and Low-Rank Bias in L2-Regularized Nonlinear Networks](https://mlsanity.com/paper/2402.03991/) (2024)
- [Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flows](https://mlsanity.com/paper/2405.12888/) (2024)
- [Principal Components Bias in Over-parameterized Linear Models, and its Manifestation in Deep Neural Networks](https://mlsanity.com/paper/2105.05553/) (2021)
- [Stable Recovery of Entangled Weights: Towards Robust Identification of Deep Neural Networks from Minimal Samples](https://mlsanity.com/paper/2101.07150/) (2021)
- [Deep Linear Networks for Matrix Completion -- An Infinite Depth Limit](https://mlsanity.com/paper/2210.12497/) (2022)
- [Understanding the training of infinitely deep and wide ResNets with Conditional Optimal Transport](https://mlsanity.com/paper/2403.12887/) (2024)
- [An entropy formula for the Deep Linear Network](https://mlsanity.com/paper/2509.09088/) (2025)
- [Critical Points and Convergence Analysis of Generative Deep Linear Networks Trained with Bures-Wasserstein Loss](https://mlsanity.com/paper/2303.03027/) (2023)
- [Federated Dynamical Low-Rank Training with Global Loss Convergence Guarantees](https://mlsanity.com/paper/2406.17887/) (2024)
- [Transformative or Conservative? Conservation laws for ResNets and Transformers](https://mlsanity.com/paper/2506.06194/) (2025)
- [Side Effects of Learning from Low-dimensional Data Embedded in a Euclidean Space](https://mlsanity.com/paper/2203.00614/) (2022)
- [How and When Random Feedback Works: A Case Study of Low-Rank Matrix Factorization](https://mlsanity.com/paper/2111.08706/) (2021)
- [Gradient descent provably escapes saddle points in the training of shallow ReLU networks](https://mlsanity.com/paper/2208.02083/) (2022)

## Abstract (arXiv)

We study the convergence of gradient flows related to learning deep linear neural networks (where the activation function is the identity map) from data. In this case, the composition of the network layers amounts to simply multiplying the weight matrices of all layers together, resulting in an overparameterized problem. The gradient flow with respect to these factors can be re-interpreted as a Riemannian gradient flow on the manifold of rank-r matrices endowed with a suitable Riemannian metric. We show that the flow always converges to a critical point of the underlying functional. Moreover, we establish that, for almost all initializations, the flow converges to a global minimum on the manifold of rank k matrices for some k≤ r.