mlsanityData to 14 Oct 2026

Gradient Descent Finds Global Minima for Generalizable Deep Neural Networks of Practical Sizes

arXiv 1908.02419 ↗First posted 5 Aug 2019stat.ML cs.LG cs.NEcited by 29

Not on the map. Why?

Abstract (arXiv)

In this paper, we theoretically prove that gradient descent can find a global minimum of non-convex optimization of all layers for nonlinear deep neural networks of sizes commonly encountered in practice. The theory developed in this paper only requires the practical degrees of over-parameterization unlike previous theories. Our theory only requires the number of trainable parameters to increase linearly as the number of training samples increases. This allows the size of the deep neural networks to be consistent with practice and to be several orders of magnitude smaller than that required by the previous theories. Moreover, we prove that the linear increase of the size of the network is the optimal rate and that it cannot be improved, except by a logarithmic factor. Furthermore, deep neural networks with the trainability guarantee are shown to generalize well to unseen test samples with a natural dataset but not a random dataset.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). First posted on arXiv 5 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 8; 2022: 10; 2023: 4; 2024: 2; 2025: 3; 2026: 2.

29 papers.

Sort:

  1. 2021Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depthcited by 55
  2. 2021Meta-learning PINN loss functionscited by 51
  3. 2021On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layerscited by 35
  4. 2021Subquadratic Overparameterization for Shallow Neural Networkscited by 26
  5. 2022Robustness Implies Generalization via Data-Dependent Generalization Boundscited by 23
  6. 2023Six Lectures on Linearized Neural Networkscited by 20
  7. 2022Overcoming the Spectral Bias of Neural Value Approximationcited by 19
  8. 2022Bounding the Width of Neural Networks via Coupled Initialization -- A Worst Case Analysiscited by 15
  9. 2023Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Reviewcited by 9
  10. 2021A Recipe for Global Convergence Guarantee in Deep Neural Networkscited by 8
  11. 2022Approximation results for Gradient Descent trained Shallow Neural Networks in 1dcited by 7
  12. 2021On the Double Descent of Random Features Models Trained with SGDcited by 5
17 more
  1. 2022Global Convergence of SGD On Two Layer Neural Netscited by 5
  2. 2021A Convergence Theory Towards Practical Over-parameterized Deep Neural Networkscited by 4
  3. 2021Learning distinct features helps, provablycited by 4
  4. 2022Learning Proximal Operators to Discover Multiple Optimacited by 4
  5. 2023Analysis of the expected L2 error of an over-parametrized deep neural network estimate learned by gradient descent without regularizationcited by 4
  6. 2023Approximation Results for Gradient Descent trained Neural Networkscited by 2
  7. 2022Improved Overparametrization Bounds for Global Convergence of Stochastic Gradient Descent for Shallow Neural Networkscited by 1
  8. 2022Parameter Convex Neural Networkscited by 1
  9. 2024Approximation and Gradient Descent Training with Neural Networkscited by 1
  10. 2025From Sublinear to Linear: Local Convergence in Finite-Width Networks via Locally Polyak-Lojasiewicz Regionscited by 1
  11. 2026On damage of interpolation to adversarial robustness in regressioncited by 1
  12. 2022Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networkscited by 0
  13. 2022Demystifying the Global Convergence Puzzle of Learning Over-parameterized ReLU Nets in Very High Dimensionscited by 0
  14. 2024Analysis of the rate of convergence of an over-parametrized convolutional neural network image classifier learned by gradient descentcited by 0
  15. 2025Statistically guided deep learningcited by 0
  16. 2025Distributionally Robust Wireless Semantic Communication with Large AI Modelscited by 0
  17. 2026Upper Bounds on the Generalization Error of Deep Learning Models via Local Robustness and Stabilitycited by 0