mlsanityData to 14 Oct 2026

A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions

arXiv 2102.09924 ↗First posted 19 Feb 2021math.NA cs.LG cs.NAcited by 14

Not on the map. Why?

Abstract (arXiv)

Gradient descent optimization algorithms are the standard ingredients that are used to train artificial neural networks (ANNs). Even though a huge number of numerical simulations indicate that gradient descent optimization methods do indeed convergence in the training of ANNs, until today there is no rigorous theoretical analysis which proves (or disproves) this conjecture. In particular, even in the case of the most basic variant of gradient descent optimization algorithms, the plain vanilla gradient descent method, it remains an open problem to prove or disprove the conjecture that gradient descent converges in the training of ANNs. In this article we solve this problem in the special situation where the target function under consideration is a constant function. More specifically, in the case of constant target functions we prove in the training of rectified fully-connected feedforward ANNs with one-hidden layer that the risk function of the gradient descent method does indeed converge to zero. Our mathematical analysis strongly exploits the property that the rectifier function is the activation function used in the considered ANNs. A key contribution of this work is to explicitly specify a Lyapunov function for the gradient flow system of the ANN parameters. This Lyapunov function is the central tool in our convergence proof of the gradient descent method.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 22 of its 27 references that are in the map. First posted on arXiv 19 Feb 2021.

Builds on

22 of 27 references.

References · 22

  1. 2016An overview of gradient descent optimization algorithmscited by 1,033
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  3. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  4. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  5. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  6. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  7. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  8. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  9. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 141
  10. 2013Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)cited by 111
  11. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  12. 2018Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?cited by 94
10 more
  1. 2019Convergence rates for the stochastic gradient descent method for non-convex objective functionscited by 66
  2. 2019Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networkscited by 60
  3. 2019A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamicscited by 49
  4. 2019Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptionscited by 48
  5. 2019The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descentcited by 46
  6. 2018Collapse of Deep and Narrow Neural Netscited by 37
  7. 2020Taming neural networks with TUSLA: Non-convex learning via adaptive stochastic gradient Langevin algorithmscited by 21
  8. 2020Non-convergence of stochastic gradient descent in the training of deep neural networkscited by 14
  9. 2020Nonasymptotic analysis of Stochastic Gradient Hamiltonian Monte Carlo under local conditions for nonconvex optimizationcited by 13
  10. 2018Lower error bounds for the stochastic gradient descent optimization algorithm: Sharp convergence rates for slowly and fast decaying learning ratescited by 11

Built on by

Cited by year

Citing papers per year: 2021: 7; 2022: 3; 2023: 0; 2024: 1; 2025: 3.

14 papers.

Sort:

  1. 2021On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networkscited by 12
  2. 2021Landscape analysis for shallow neural networks: complete classification of critical points for affine target functionscited by 11
  3. 2021Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activationcited by 11
  4. 2021A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functionscited by 10
  5. 2021A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functionscited by 9
  6. 2021Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activationcited by 9
  7. 2021Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functionscited by 9
  8. 2024Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networkscited by 9
  9. 2025Non-convergence to the optimal risk for Adam and stochastic gradient descent optimization in the training of deep neural networkscited by 6
  10. 2022Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networkscited by 5
  11. 2022Gradient descent provably escapes saddle points in the training of shallow ReLU networkscited by 5
  12. 2025Compound AI Systems Optimization: A Survey of Methods, Challenges, and Future Directionscited by 4
2 more
  1. 2025Physics Informed Constrained Learning of Dynamics from Static Datacited by 1
  2. 2022Normalized gradient flow optimization in the training of ReLU artificial neural networkscited by 0