mlsanityData to 7 Oct 2026

A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions

arXiv 2104.00277 ↗First posted 1 Apr 2021math.NA cs.LG cs.NAcited by 9

Not on the map. Why?

Abstract (arXiv)

In this article we study the stochastic gradient descent (SGD) optimization method in the training of fully-connected feedforward artificial neural networks with ReLU activation. The main result of this work proves that the risk of the SGD process converges to zero if the target function under consideration is constant. In the established convergence result the considered artificial neural networks consist of one input layer, one hidden layer, and one output layer (with d ∈ mathbbN neurons on the input layer, H ∈ mathbbN neurons on the hidden layer, and one neuron on the output layer). The learning rates of the SGD process are assumed to be sufficiently small and the input data used in the SGD process to train the artificial neural networks is assumed to be independent and identically distributed.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 33 of its 47 references that are in the map. First posted on arXiv 1 Apr 2021.

Builds on

33 of 47 references.

References · 33

  1. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  2. 2016An overview of gradient descent optimization algorithmscited by 1,021
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  5. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  6. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  7. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  8. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  9. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 241
  10. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  11. 2013Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)cited by 110
  12. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
21 more
  1. 2018Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?cited by 94
  2. 2019Convergence rates for the stochastic gradient descent method for non-convex objective functionscited by 66
  3. 2019A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamicscited by 49
  4. 2019Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptionscited by 48
  5. 2019The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descentcited by 46
  6. 2019Non-asymptotic Analysis of Biased Stochastic Approximation Schemecited by 45
  7. 2018Solving the Kolmogorov PDE by means of deep learningcited by 41
  8. 2018Collapse of Deep and Narrow Neural Netscited by 37
  9. 2019First-order methods almost always avoid saddle points: the case of vanishing step-sizescited by 35
  10. 2016Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regionscited by 33
  11. 2021Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapescited by 27
  12. 2018Exponential Convergence Time of Gradient Descent for One-Dimensional Deep Linear Neural Networkscited by 22
  13. 2020Taming neural networks with TUSLA: Non-convex learning via adaptive stochastic gradient Langevin algorithmscited by 21
  14. 2019Full error analysis for the training of deep neural networkscited by 17
  15. 2020Non-convergence of stochastic gradient descent in the training of deep neural networkscited by 14
  16. 2021A proof of convergence for gradient descent in the training of artificial neural networks for constant target functionscited by 14
  17. 2020Nonasymptotic analysis of Stochastic Gradient Hamiltonian Monte Carlo under local conditions for nonconvex optimizationcited by 13
  18. 2020Stopping Criteria for, and Strong Convergence of, Stochastic Gradient Descent on Bottou-Curtis-Nocedal Functionscited by 13
  19. 2018Lower error bounds for the stochastic gradient descent optimization algorithm: Sharp convergence rates for slowly and fast decaying learning ratescited by 11
  20. 2021Landscape analysis for shallow neural networks: complete classification of critical points for affine target functionscited by 11
  21. 2020Overall error analysis for the training of deep neural networks via stochastic gradient descent with random initialisationcited by 7

Built on by

Cited by year

Citing papers per year: 2021: 5; 2022: 1; 2023: 1; 2024: 0; 2025: 2.

9 papers.

  1. 2021On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networkscited by 12
  2. 2021Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activationcited by 11
  3. 2021A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functionscited by 10
  4. 2021Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activationcited by 9
  5. 2021Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functionscited by 9
  6. 2025Non-convergence to the optimal risk for Adam and stochastic gradient descent optimization in the training of deep neural networkscited by 6
  7. 2023Operator theory, kernels, and Feedforward Neural Networkscited by 2
  8. 2022Identical Image Retrieval using Deep Learningcited by 0
  9. 2025Why Does Stochastic Gradient Descent Slow Down in Low-Precision Training?cited by 0