mlsanityData to 7 Oct 2026

Convergence of gradient descent for learning linear neural networks

arXiv 2108.02040 ↗First posted 4 Aug 2021cs.LG math.OCcited by 22

Not on the map. Why?

Abstract (arXiv)

We study the convergence properties of gradient descent for training deep linear neural networks, i.e., deep matrix factorizations, by extending a previous analysis for the related gradient flow. We show that under suitable conditions on the step sizes gradient descent converges to a critical point of the loss function, i.e., the square loss in this article. Furthermore, we demonstrate that for almost all initializations gradient descent converges to a global minimum in the case of two layers. In the case of three or more layers we show that gradient descent converges to a global minimum on the manifold matrices of some fixed rank, where the rank cannot be determined a priori.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 26 of its 31 references that are in the map. First posted on arXiv 4 Aug 2021.

Builds on

26 of 31 references.

References · 26

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  3. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  4. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  5. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  6. 2017Implicit Regularization in Matrix Factorizationcited by 318
  7. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  8. 2016Deep Learning without Poor Local Minimacited by 233
  9. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  10. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  11. 2018Stochastic subgradient method converges on tame functionscited by 120
  12. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
14 more
  1. 2020On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problemscited by 79
  2. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 74
  3. 2020Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networkscited by 73
  4. 2019Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizerscited by 61
  5. 2019Width Provably Matters in Optimization for Deep Linear Neural Networkscited by 52
  6. 2018Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networkscited by 44
  7. 2020Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rankcited by 42
  8. 2019First-order methods almost always avoid saddle points: the case of vanishing step-sizescited by 35
  9. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  10. 2019Pure and Spurious Critical Points: a Geometric Study of Linear Networkscited by 33
  11. 2017Global optimality conditions for deep neural networkscited by 31
  12. 2020On the Global Convergence of Training Deep Linear ResNetscited by 29
  13. 2018Exponential Convergence Time of Gradient Descent for One-Dimensional Deep Linear Neural Networkscited by 22
  14. 2019Global Convergence of Gradient Descent for Deep Linear Residual Networkscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 4; 2025: 8; 2026: 3.

22 papers.

Sort:

  1. 2023Function Space and Critical Points of Linear Convolutional Networkscited by 19
  2. 2024How do Transformers perform In-Context Autoregressive Learning?cited by 16
  3. 2023Robust Implicit Regularization via Weight Normalizationcited by 10
  4. 2025An entropy formula for the Deep Linear Networkcited by 8
  5. 2023Critical Points and Convergence Analysis of Generative Deep Linear Networks Trained with Bures-Wasserstein Losscited by 7
  6. 2022Side Effects of Learning from Low-dimensional Data Embedded in a Euclidean Spacecited by 6
  7. 2023Geometry of Linear Neural Networks: Equivariance and Invariance under Permutation Groupscited by 5
  8. 2024Convergence of continuous-time stochastic gradient descent with applications to deep neural networkscited by 4
  9. 2025A Local Polyak-Lojasiewicz and Descent Lemma of Gradient Descent For Overparametrized Linear Modelscited by 4
  10. 2024Three Mechanisms of Feature Learning in a Linear Networkcited by 3
  11. 2023Implicit Regularization Makes Overparameterized Asymmetric Matrix Sensing Robust to Perturbationscited by 2
  12. 2025Gradient Descent Converges Linearly to Flatter Minima than Gradient Flow in Shallow Linear Networkscited by 2
10 more
  1. 2025The Riemannian Geometry Associated to Gradient Flows of Linear Convolutional Networkscited by 2
  2. 2026Balanced LoRA: Removing Parameter Invariance to Accelerate Convergencecited by 2
  3. 2024Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learningcited by 1
  4. 2025Error Bound Analysis for the Regularized Loss of Deep Linear Neural Networkscited by 1
  5. 2025Wormhole Dynamics in Deep Neural Networkscited by 1
  6. 2026Does SGD Seek Flatness or Sharpness? An Exactly Solvable Modelcited by 1
  7. 2023Implicit Balancing and Regularization: Generalization and Convergence Guarantees for Overparameterized Asymmetric Matrix Sensingcited by 0
  8. 2025GradES: Significantly Faster Training in Transformers with Gradient-Based Early Stoppingcited by 0
  9. 2025The Interplay of Statistics and Noisy Optimization: Learning Linear Predictors with Random Data Weightscited by 0
  10. 2026Convergence of gradient flow for learning convolutional neural networkscited by 0