mlsanityData to 7 Oct 2026

Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functions

arXiv 2112.07369 ↗First posted 13 Dec 2021cs.LG cs.NA math.NAcited by 9

Not on the map. Why?

Abstract (arXiv)

In many numerical simulations stochastic gradient descent (SGD) type optimization methods perform very effectively in the training of deep neural networks (DNNs) but till this day it remains an open problem of research to provide a mathematical convergence analysis which rigorously explains the success of SGD type optimization methods in the training of DNNs. In this work we study SGD type optimization methods in the training of fully-connected feedforward DNNs with rectified linear unit (ReLU) activation. We first establish general regularity properties for the risk functions and their generalized gradient functions appearing in the training of such DNNs and, thereafter, we investigate the plain vanilla SGD optimization method in the training of such DNNs under the assumption that the target function under consideration is a constant function. Specifically, we prove under the assumption that the learning rates (the step sizes of the SGD optimization method) are sufficiently small but not L1-summable and under the assumption that the target function is a constant function that the expectation of the riskof the considered SGD process converges in the training of such DNNs to zero as the number of SGD steps increases to infinity.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 39 of its 54 references that are in the map. First posted on arXiv 13 Dec 2021.

Builds on

39 of 54 references.

References · 39

  1. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  2. 2016An overview of gradient descent optimization algorithmscited by 1,021
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  4. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  5. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  6. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  7. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  8. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  9. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 241
  10. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  11. 2013Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)cited by 110
  12. 2020Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don'tcited by 94
27 more
  1. 2019Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networkscited by 76
  2. 2019Convergence rates for the stochastic gradient descent method for non-convex objective functionscited by 66
  3. 2013Fast Convergence of Stochastic Gradient Descent under a Strong Growth Conditioncited by 54
  4. 2019A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamicscited by 49
  5. 2019Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptionscited by 48
  6. 2019The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descentcited by 46
  7. 2019Non-asymptotic Analysis of Biased Stochastic Approximation Schemecited by 45
  8. 2018Solving the Kolmogorov PDE by means of deep learningcited by 41
  9. 2021Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysiscited by 38
  10. 2018Collapse of Deep and Narrow Neural Netscited by 37
  11. 2019First-order methods almost always avoid saddle points: the case of vanishing step-sizescited by 35
  12. 2016Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regionscited by 33
  13. 2021Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapescited by 27
  14. 2018Exponential Convergence Time of Gradient Descent for One-Dimensional Deep Linear Neural Networkscited by 22
  15. 2020A Dynamical Central Limit Theorem for Shallow Neural Networkscited by 22
  16. 2020Taming neural networks with TUSLA: Non-convex learning via adaptive stochastic gradient Langevin algorithmscited by 21
  17. 2019Full error analysis for the training of deep neural networkscited by 17
  18. 2020Non-convergence of stochastic gradient descent in the training of deep neural networkscited by 14
  19. 2021A proof of convergence for gradient descent in the training of artificial neural networks for constant target functionscited by 14
  20. 2020Nonasymptotic analysis of Stochastic Gradient Hamiltonian Monte Carlo under local conditions for nonconvex optimizationcited by 13
  21. 2020Stopping Criteria for, and Strong Convergence of, Stochastic Gradient Descent on Bottou-Curtis-Nocedal Functionscited by 13
  22. 2018Lower error bounds for the stochastic gradient descent optimization algorithm: Sharp convergence rates for slowly and fast decaying learning ratescited by 11
  23. 2021Landscape analysis for shallow neural networks: complete classification of critical points for affine target functionscited by 11
  24. 2021Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activationcited by 11
  25. 2021A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functionscited by 10
  26. 2021A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functionscited by 9
  27. 2021Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activationcited by 9

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 3; 2023: 0; 2024: 1; 2025: 4.

9 papers.

  1. 2021On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networkscited by 12
  2. 2024Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networkscited by 9
  3. 2025Non-convergence to the optimal risk for Adam and stochastic gradient descent optimization in the training of deep neural networkscited by 6
  4. 2022Global Convergence of SGD On Two Layer Neural Netscited by 5
  5. 2025Compound AI Systems Optimization: A Survey of Methods, Challenges, and Future Directionscited by 4
  6. 2022On bounds for norms of reparameterized ReLU artificial neural network parameters: sums of fractional powers of the Lipschitz norm control the network parameter vectorcited by 0
  7. 2022Normalized gradient flow optimization in the training of ReLU artificial neural networkscited by 0
  8. 2025Error analysis for the deep Kolmogorov methodcited by 0
  9. 2025Gradient Flow Convergence Guarantee for General Neural Network Architecturescited by 0