mlsanityData to 14 Oct 2026

Gradient descent provably escapes saddle points in the training of shallow ReLU networks

arXiv 2208.02083 ↗First posted 3 Aug 2022cs.LG math.DS math.OCcited by 5

Not on the map. Why?

Abstract (arXiv)

Dynamical systems theory has recently been applied in optimization to prove that gradient descent algorithms bypass so-called strict saddle points of the loss function. However, in many modern machine learning applications, the required regularity conditions are not satisfied. In this paper, we prove a variant of the relevant dynamical systems result, a center-stable manifold theorem, in which we relax some of the regularity requirements. We explore its relevance for various machine learning tasks, with a particular focus on shallow rectified linear unit (ReLU) and leaky ReLU networks with scalar input. Building on a detailed examination of critical points of the square integral loss function for shallow ReLU and leaky ReLU networks relative to an affine target function, we show that gradient descent circumvents most saddle points. Furthermore, we prove convergence to global minima under favourable initialization conditions, quantified by an explicit threshold on the limiting loss.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 36 of its 54 references that are in the map. First posted on arXiv 3 Aug 2022.

Builds on

36 of 54 references.

References · 36

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  3. 2018On Lazy Training in Differentiable Programmingcited by 675
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  5. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  6. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  7. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 289
  8. 2017How to Escape Saddle Points Efficientlycited by 285
  9. 2014The Loss Surfaces of Multilayer Networkscited by 280
  10. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  11. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 243
  12. 2016Deep Learning without Poor Local Minimacited by 234
24 more
  1. 2017No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysiscited by 129
  2. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  3. 2018Stochastic subgradient method converges on tame functionscited by 124
  4. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  5. 2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
  6. 2018The Limit Points of (Optimistic) Gradient Descent in Min-Max Optimizationcited by 101
  7. 2016A Geometric Analysis of Phase Retrievalcited by 92
  8. 2016Global Optimality of Local Search for Low Rank Matrix Recoverycited by 82
  9. 2017Gradient Descent Can Take Exponential Time to Escape Saddle Pointscited by 74
  10. 2019Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizerscited by 61
  11. 2015Complete Dictionary Recovery over the Sphere I: Overview and the Geometric Picturecited by 49
  12. 2019A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamicscited by 49
  13. 2019Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptionscited by 48
  14. 2020On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regimecited by 39
  15. 2020On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamicscited by 39
  16. 2019First-order methods almost always avoid saddle points: the case of vanishing step-sizescited by 35
  17. 2016Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regionscited by 34
  18. 2014Splitting methods with variable metric for KL functionscited by 26
  19. 2015On the Quality of the Initial Basin in Overspecified Neural Networkscited by 20
  20. 2017The nonsmooth landscape of phase retrievalcited by 16
  21. 2021A proof of convergence for gradient descent in the training of artificial neural networks for constant target functionscited by 14
  22. 2021Landscape analysis for shallow neural networks: complete classification of critical points for affine target functionscited by 11
  23. 2021A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functionscited by 10
  24. 2017Behavior of Accelerated Gradient Methods Near Critical Points of Nonconvex Functionscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 0; 2025: 2; 2026: 1.

5 papers.

  1. 2022Global Convergence of SGD On Two Layer Neural Netscited by 5
  2. 2023Global Convergence of SGD For Logistic Loss on Two Layer Neural Netscited by 3
  3. 2025Langevin Monte-Carlo Provably Learns Depth Two Neural Nets at Any Size and Datacited by 1
  4. 2025Non-Singularity of the Gradient Descent map for Neural Networks with Piecewise Analytic Activationscited by 0
  5. 2026Endogenous Regime Switching Driven by Scalar-Irreducible Learning Dynamicscited by 0