mlsanityData to 14 Oct 2026

Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers

arXiv 1910.05505 ↗First posted 12 Oct 2019math.OC cs.LGcited by 61

Not on the map. Why?

Abstract (arXiv)

We study the convergence of gradient flows related to learning deep linear neural networks (where the activation function is the identity map) from data. In this case, the composition of the network layers amounts to simply multiplying the weight matrices of all layers together, resulting in an overparameterized problem. The gradient flow with respect to these factors can be re-interpreted as a Riemannian gradient flow on the manifold of rank-r matrices endowed with a suitable Riemannian metric. We show that the flow always converges to a critical point of the underlying functional. Moreover, we establish that, for almost all initializations, the flow converges to a global minimum on the manifold of rank k matrices for some k≤ r.

Key facts. Cited by 61 papers in the map (data through 14 Oct 2026). First posted on arXiv 12 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 9; 2022: 8; 2023: 12; 2024: 10; 2025: 15; 2026: 7.

61 papers.

Sort:

By area:

  1. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 78
  2. 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 45
  3. 2025Symmetry in Neural Network Parameter Spacescited by 36
  4. 2021Geometry of Linear Convolutional Networkscited by 27
  5. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  6. 2021The loss landscape of deep linear neural networks: a second-order analysiscited by 24
  7. 2021Convergence of gradient descent for learning linear neural networkscited by 22
  8. 2023On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensionscited by 21
  9. 2022Infinite-width limit of deep linear neural networkscited by 20
  10. 2023Function Space and Critical Points of Linear Convolutional Networkscited by 19
  11. 2023Optimization Dynamics of Equivariant and Augmented Neural Networkscited by 18
  12. 2023Neural (Tangent Kernel) Collapsecited by 18
38 more
  1. 2024Plastic Learning with Deep Fourier Featurescited by 14
  2. 2023Robust low-rank training via approximate orthonormal constraintscited by 13
  3. 2023Robust Implicit Regularization via Weight Normalizationcited by 12
  4. 2023Geometry-aware training of factorized layers in tensor Tucker formatcited by 12
  5. 2021Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activationcited by 11
  6. 2024Provable Emergence of Deep Neural Collapse and Low-Rank Bias in L2-Regularized Nonlinear Networkscited by 11
  7. 2024Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flowscited by 11
  8. 2021Principal Components Bias in Over-parameterized Linear Models, and its Manifestation in Deep Neural Networkscited by 10
  9. 2021Stable Recovery of Entangled Weights: Towards Robust Identification of Deep Neural Networks from Minimal Samplescited by 8
  10. 2022Deep Linear Networks for Matrix Completion -- An Infinite Depth Limitcited by 8
  11. 2024Understanding the training of infinitely deep and wide ResNets with Conditional Optimal Transportcited by 8
  12. 2025An entropy formula for the Deep Linear Networkcited by 8
  13. 2023Critical Points and Convergence Analysis of Generative Deep Linear Networks Trained with Bures-Wasserstein Losscited by 7
  14. 2024Federated Dynamical Low-Rank Training with Global Loss Convergence Guaranteescited by 7
  15. 2025Transformative or Conservative? Conservation laws for ResNets and Transformerscited by 7
  16. 2022Side Effects of Learning from Low-dimensional Data Embedded in a Euclidean Spacecited by 6
  17. 2021How and When Random Feedback Works: A Case Study of Low-Rank Matrix Factorizationcited by 5
  18. 2022Gradient descent provably escapes saddle points in the training of shallow ReLU networkscited by 5
  19. 2022Finite Sample Identification of Wide Shallow Neural Networks with Biasescited by 5
  20. 2023Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Spacecited by 5
  21. 2021First-Order Methods for Convex Optimizationcited by 4
  22. 2023Combining Explicit and Implicit Regularization for Efficient Learning in Deep Networkscited by 4
  23. 2023Implicit regularization in AI meets generalized hardness of approximation in optimization -- Sharp results for diagonal linear networkscited by 4
  24. 2024Convergence of continuous-time stochastic gradient descent with applications to deep neural networkscited by 4
  25. 2025Intrinsic training dynamics of deep neural networkscited by 4
  26. 2025Regularization Implies balancedness in the deep linear networkcited by 4
  27. 2024On the Convergence of Gradient Descent for Large Learning Ratescited by 3
  28. 2025Understanding Learning Invariance in Deep Linear Networkscited by 3
  29. 2024On the Role of Initialization on the Implicit Bias in Deep Linear Networkscited by 2
  30. 2025The Riemannian Geometry Associated to Gradient Flows of Linear Convolutional Networkscited by 2
  31. 2025Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamicscited by 2
  32. 2022A Dynamics Theory of Implicit Regularization in Deep Low-Rank Matrix Factorizationcited by 1
  33. 2024Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learningcited by 1
  34. 2025Error Bound Analysis for the Regularized Loss of Deep Linear Neural Networkscited by 1
  35. 2025The Effect of Depth on the Expressivity of Deep Linear State-Space Modelscited by 1
  36. 2025Wormhole Dynamics in Deep Neural Networkscited by 1
  37. 2026Does SGD Seek Flatness or Sharpness? An Exactly Solvable Modelcited by 1
  38. 2026Conservation Laws for Modern Neural Architecturescited by 1

50 most cited of 61 All 61 (JSON)