mlsanityData to 7 Oct 2026

A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functions

arXiv 2108.04620 ↗First posted 10 Aug 2021math.OC cs.LG cs.NAcited by 10

Not on the map. Why?

Abstract (arXiv)

Gradient descent (GD) type optimization methods are the standard instrument to train artificial neural networks (ANNs) with rectified linear unit (ReLU) activation. Despite the great success of GD type optimization methods in numerical simulations for the training of ANNs with ReLU activation, it remains - even in the simplest situation of the plain vanilla GD optimization method with random initializations and ANNs with one hidden layer - an open problem to prove (or disprove) the conjecture that the risk of the GD optimization method converges in the training of such ANNs to zero as the width of the ANNs, the number of independent random initializations, and the number of GD steps increase to infinity. In this article we prove this conjecture in the situation where the probability distribution of the input data is equivalent to the continuous uniform distribution on a compact interval, where the probability distributions for the random initializations of the ANN parameters are standard normal distributions, and where the target function under consideration is continuous and piecewise affine linear. Roughly speaking, the key ingredients in our mathematical convergence analysis are (i) to prove that suitable sets of global minima of the risk functions are twice continuously differentiable submanifolds of the ANN parameter spaces, (ii) to prove that the Hessians of the risk functions on these sets of global minima satisfy an appropriate maximal rank condition, and, thereafter, (iii) to apply the machinery in [Fehrman, B., Gess, B., Jentzen, A., Convergence rates for the stochastic gradient descent method for non-convex objective functions. J. Mach. Learn. Res. 21(136): 1--48, 2020] to establish convergence of the GD optimization method with random initializations.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 32 of its 59 references that are in the map. First posted on arXiv 10 Aug 2021.

Builds on

32 of 59 references.

References · 32

  1. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  2. 2016An overview of gradient descent optimization algorithmscited by 1,021
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  5. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  6. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  7. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  8. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  9. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  10. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 241
  11. 2013Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)cited by 110
  12. 2020Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don'tcited by 94
20 more
  1. 2019Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networkscited by 76
  2. 2019Convergence rates for the stochastic gradient descent method for non-convex objective functionscited by 66
  3. 2013Fast Convergence of Stochastic Gradient Descent under a Strong Growth Conditioncited by 54
  4. 2019A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamicscited by 49
  5. 2019Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptionscited by 48
  6. 2019The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descentcited by 46
  7. 2021Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysiscited by 38
  8. 2019First-order methods almost always avoid saddle points: the case of vanishing step-sizescited by 35
  9. 2016Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regionscited by 33
  10. 2021Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapescited by 27
  11. 2018Exponential Convergence Time of Gradient Descent for One-Dimensional Deep Linear Neural Networkscited by 22
  12. 2020Taming neural networks with TUSLA: Non-convex learning via adaptive stochastic gradient Langevin algorithmscited by 21
  13. 2020Non-convergence of stochastic gradient descent in the training of deep neural networkscited by 14
  14. 2021A proof of convergence for gradient descent in the training of artificial neural networks for constant target functionscited by 14
  15. 2020Nonasymptotic analysis of Stochastic Gradient Hamiltonian Monte Carlo under local conditions for nonconvex optimizationcited by 13
  16. 2020Stopping Criteria for, and Strong Convergence of, Stochastic Gradient Descent on Bottou-Curtis-Nocedal Functionscited by 13
  17. 2018Lower error bounds for the stochastic gradient descent optimization algorithm: Sharp convergence rates for slowly and fast decaying learning ratescited by 11
  18. 2021Landscape analysis for shallow neural networks: complete classification of critical points for affine target functionscited by 11
  19. 2021Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activationcited by 11
  20. 2021A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functionscited by 9

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 2; 2023: 1; 2024: 2; 2025: 2.

10 papers.

  1. 2021On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networkscited by 12
  2. 2021Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activationcited by 9
  3. 2021Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functionscited by 9
  4. 2024Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networkscited by 9
  5. 2022Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networkscited by 5
  6. 2022Gradient descent provably escapes saddle points in the training of shallow ReLU networkscited by 5
  7. 2023Approximation Results for Gradient Descent trained Neural Networkscited by 2
  8. 2024Approximation and Gradient Descent Training with Neural Networkscited by 1
  9. 2025Constructive Universal Approximation and Sure Convergence for Multi-Layer Neural Networkscited by 0
  10. 2025Non-Singularity of the Gradient Descent map for Neural Networks with Piecewise Analytic Activationscited by 0