mlsanityData to 7 Oct 2026

Global Convergence Analysis of Deep Linear Networks with A One-neuron Layer

arXiv 2201.02761 ↗First posted 8 Jan 2022cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we follow Eftekhari's work to give a non-local convergence analysis of deep linear networks. Specifically, we consider optimizing deep linear networks which have a layer with one neuron under quadratic loss. We describe the convergent point of trajectories with arbitrary starting point under gradient flow, including the paths which converge to one of the saddle points or the original point. We also show specific convergence rates of trajectories that converge to the global minimizer by stages. To achieve these results, this paper mainly extends the machinery in Eftekhari's work to provably identify the rank-stable set and the global minimizer convergent set. We also give specific examples to show the necessity of our definitions. Crucially, as far as we know, our results appear to be the first to give a non-local global analysis of linear neural networks from arbitrary initialized points, rather than the lazy training regime which has dominated the literature of neural networks, and restricted benign initialization in Eftekhari's work. We also note that extending our results to general linear networks without one hidden neuron assumption remains a challenging open problem.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 34 of its 45 references that are in the map. First posted on arXiv 8 Jan 2022.

Builds on

34 of 45 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  4. 2018On Lazy Training in Differentiable Programmingcited by 671
  5. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  6. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  7. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  8. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  9. 2016Deep Learning without Poor Local Minimacited by 233
  10. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  11. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  12. 2020Directional convergence and alignment in deep learningcited by 172
22 more
  1. 2018Convergence of Gradient Descent on Separable Datacited by 108
  2. 2016Identity Matters in Deep Learningcited by 107
  3. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  4. 2020Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networkscited by 73
  5. 2017Deep linear neural networks with arbitrary loss: All local minima are globalcited by 64
  6. 2019Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizerscited by 61
  7. 2020The Global Landscape of Neural Networks: An Overviewcited by 57
  8. 2019Width Provably Matters in Optimization for Deep Linear Neural Networkscited by 52
  9. 2018Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networkscited by 44
  10. 2018On the loss landscape of a class of deep neural networks with no bad local valleyscited by 38
  11. 2017Depth Creates No Bad Local Minimacited by 29
  12. 2020On the Global Convergence of Training Deep Linear ResNetscited by 29
  13. 2018Understanding the Loss Surface of Neural Networks for Binary Classificationcited by 27
  14. 2018Exponential Convergence Time of Gradient Descent for One-Dimensional Deep Linear Neural Networkscited by 22
  15. 2018Adding One Neuron Can Eliminate All Bad Local Minimacited by 21
  16. 2018Spurious Valleys in Two-layer Neural Network Optimization Landscapescited by 20
  17. 2020Training Linear Neural Networks: Non-Local Convergence and Complexity Resultscited by 19
  18. 2018On the Benefit of Width for Neural Networks: Disappearance of Bad Basinscited by 16
  19. 2019Global Convergence of Gradient Descent for Deep Linear Residual Networkscited by 16
  20. 2018Learning Deep Models: Critical Points and Local Opennesscited by 10
  21. 2019Revisiting Landscape Analysis in Deep Neural Networks: Eliminating Decreasing Paths to Infinitycited by 6
  22. 2019Depth creates no more spurious local minimacited by 5

Built on by

Not cited yet.