mlsanityData to 14 Oct 2026

Gradient Descent Finds Global Minima of Deep Neural Networks

arXiv 1811.03804 ↗First posted 9 Nov 2018cs.LG cs.AI cs.CVcited by 574

Not on the map. Why?

Abstract (arXiv)

Gradient descent finds a global minimum in training deep neural networks despite the objective function being non-convex. The current paper proves gradient descent achieves zero training loss in polynomial time for a deep over-parameterized neural network with residual connections (ResNet). Our analysis relies on the particular structure of the Gram matrix induced by the neural network architecture. This structure allows us to show the Gram matrix is stable throughout the training process and this stability implies the global optimality of the gradient descent algorithm. We further extend our analysis to deep residual convolutional neural networks and obtain a similar convergence result.

Key facts. Cited by 574 papers in the map (data through 14 Oct 2026). First posted on arXiv 9 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 153; 2022: 131; 2023: 100; 2024: 78; 2025: 56; 2026: 55.

574 papers.

Sort:

By area:

  1. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  2. 2022U-NO: U-shaped Neural Operatorscited by 230
  3. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  4. 2021The Values Encoded in Machine Learning Researchcited by 172
  5. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  6. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  7. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 154
  8. 2021Dash: Semi-Supervised Learning with Dynamic Thresholdingcited by 148
  9. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 132
  10. 2021On the Convergence and Robustness of Adversarial Trainingcited by 130
  11. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 127
  12. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
38 more
  1. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 121
  2. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  3. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  4. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 110
  5. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  6. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
  7. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 102
  8. 2021Theory of overparametrization in quantum neural networkscited by 100
  9. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 91
  10. 2021Learning with invariances in random features and kernel modelscited by 89
  11. 2021Improved architectures and training algorithms for deep operator networkscited by 86
  12. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 83
  13. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 80
  14. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 78
  15. 2023A Mathematical Guide to Operator Learningcited by 76
  16. 2022Universality of empirical risk minimizationcited by 75
  17. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 69
  18. 2021Improved OOD Generalization via Adversarial Training and Pre-trainingcited by 66
  19. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
  20. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  21. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 63
  22. 2021Training Multi-Layer Over-Parametrized Neural Network in Subquadratic Timecited by 62
  23. 2022Theory of Graph Neural Networks: Representation and Learningcited by 61
  24. 2021Score-based Generative Neural Networks for Large-Scale Optimal Transportcited by 60
  25. 2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 58
  26. 2023Max-Margin Token Selection in Attention Mechanismcited by 58
  27. 2021Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depthcited by 55
  28. 2021Exploring Memorization in Adversarial Trainingcited by 55
  29. 2023JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attentioncited by 52
  30. 2021Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptationcited by 50
  31. 2022Predicting Out-of-Distribution Error with the Projection Normcited by 50
  32. 2022Mechanistic Mode Connectivitycited by 50
  33. 2021Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functionscited by 49
  34. 2021What can linearized neural networks actually say about generalization?cited by 49
  35. 2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 49
  36. 2021Faster Single-loop Algorithms for Minimax Optimization without Strong Concavitycited by 49
  37. 2022AutoBalance: Optimized Loss Functions for Imbalanced Datacited by 49
  38. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 49

50 most cited of 574 All 574 (JSON)