mlsanityData to 14 Oct 2026

A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks

arXiv 1810.02281 ↗First posted 4 Oct 2018cs.LG cs.NE stat.MLcited by 184

Not on the map. Why?

Abstract (arXiv)

We analyze speed of convergence to global optimum for gradient descent training a deep linear neural network (parameterized as x mapsto WN WN-1 cdots W1 x) by minimizing the ℓ2 loss over whitened data. Convergence at a linear rate is guaranteed when the following hold: (i) dimensions of hidden layers are at least the minimum of the input and output dimensions; (ii) weight matrices at initialization are approximately balanced; and (iii) the initial loss is smaller than the loss of any rank-deficient solution. The assumptions on initialization (conditions (ii) and (iii)) are necessary, in the sense that violating any one of them may lead to convergence failure. Moreover, in the important case of output dimension 1, i.e. scalar regression, they are met, and thus convergence to global optimum holds, with constant probability under a random initialization scheme. Our results significantly extend previous analyses, e.g., of deep linear residual networks (Bartlett et al., 2018).

Key facts. Cited by 184 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 34; 2022: 30; 2023: 20; 2024: 34; 2025: 40; 2026: 26.

184 papers.

Sort:

By area:

  1. 2021Towards Understanding Knowledge Distillationcited by 158
  2. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
  3. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 102
  4. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  5. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 85
  6. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 78
  7. 2021Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depthcited by 55
  8. 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
  9. 2023JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attentioncited by 52
  10. 2021Learning Theory Can (Sometimes) Explain Generalisation in Graph Neural Networkscited by 51
  11. 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 51
  12. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 49
38 more
  1. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 46
  2. 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 45
  3. 2024Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficientcited by 40
  4. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
  5. 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
  6. 2025Symmetry in Neural Network Parameter Spacescited by 36
  7. 2021On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layerscited by 35
  8. 2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
  9. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  10. 2021Batch Normalization Orthogonalizes Representations in Deep Random Networkscited by 29
  11. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 29
  12. 2023Dynamical versus Bayesian Phase Transitions in a Toy Model of Superpositioncited by 29
  13. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  14. 2023Early Neuron Alignment in Two-layer ReLU Networks with Small Initializationcited by 28
  15. 2021Geometry of Linear Convolutional Networkscited by 27
  16. 2022Explicit Regularization in Overparametrized Models via Noise Injectioncited by 26
  17. 2022Implicit Full Waveform Inversion with Deep Neural Representationcited by 26
  18. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  19. 2021The loss landscape of deep linear neural networks: a second-order analysiscited by 24
  20. 2022Convergence of gradient descent for deep neural networkscited by 24
  21. 2021Towards Demystifying Representation Learning with Non-contrastive Self-supervisioncited by 23
  22. 2021Convergence of gradient descent for learning linear neural networkscited by 22
  23. 2022Bayesian Interpolation with Deep Linear Networkscited by 21
  24. 2023Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyondcited by 21
  25. 2024Geometry of Polynomial Neural Networkscited by 21
  26. 2022Infinite-width limit of deep linear neural networkscited by 20
  27. 2023The Law of Parsimony in Gradient Descent for Learning Deep Linear Networkscited by 20
  28. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 20
  29. 2021Noether: The More Things Change, the More Stay the Samecited by 19
  30. 2022Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learningcited by 19
  31. 2023Understanding Unimodal Bias in Multimodal Deep Linear Networkscited by 19
  32. 2024Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimescited by 18
  33. 2021A Theoretical Analysis of Fine-tuning with Linear Teacherscited by 17
  34. 2021A global convergence theory for deep ReLU implicit networks via over-parameterizationcited by 17
  35. 2024Deep linear networks for regression are implicitly regularized towards flat minimacited by 17
  36. 2021Scaling Properties of Deep Residual Networkscited by 16
  37. 2024Linear Recursive Feature Machines provably recover low-rank matricescited by 16
  38. 2024Masks, Signs, And Learning Rate Rewindingcited by 15

50 most cited of 184 All 184 (JSON)