mlsanityData to 7 Oct 2026

SGD Converges to Global Minimum in Deep Learning via Star-convex Path

arXiv 1901.00451 ↗First posted 2 Jan 2019cs.LG stat.MLcited by 39

Not on the map. Why?

Abstract (arXiv)

Stochastic gradient descent (SGD) has been found to be surprisingly effective in training a variety of deep neural networks. However, there is still a lack of understanding on how and why SGD can train these complex networks towards a global minimum. In this study, we establish the convergence of SGD to a global minimum for nonconvex optimization problems that are commonly encountered in neural network training. Our argument exploits the following two important properties: 1) the training loss can achieve zero value (approximately), which has been widely observed in deep learning; 2) SGD follows a star-convex path, which is verified by various experiments in this paper. In such a context, our analysis shows that SGD, although has long been considered as a randomized algorithm, converges in an intrinsically deterministic manner to a global minimum.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 7; 2023: 4; 2024: 3; 2025: 8; 2026: 8.

39 papers.

Sort:

  1. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  2. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 81
  3. 2021Local Stochastic Gradient Descent Ascent: Convergence Analysis and Communication Efficiencycited by 40
  4. 2021Decentralized Local Stochastic Extra-Gradient for Variational Inequalitiescited by 25
  5. 2021SGD: The Role of Implicit Regularization, Batch-size and Multiple-epochscited by 22
  6. 2021Stochastic Polyak Stepsize with a Moving Targetcited by 19
  7. 2025SGD with Adaptive Preconditioning: Unified Analysis and Momentum Accelerationcited by 12
  8. 2024How to Make the Gradients Small Privately: Improved Rates for Differentially Private Non-Convex Optimizationcited by 10
  9. 2021A Decentralized Adaptive Momentum Method for Solving a Class of Min-Max Optimization Problemscited by 9
  10. 2023Learning to Generalize Provably in Learning to Optimizecited by 8
  11. 2021Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activationscited by 7
  12. 2024Loss Landscape Characterization of Neural Networks without Over-Parametrizationcited by 7
27 more
  1. 2022Optimization for Amortized Inverse Problemscited by 6
  2. 2026On the Role of Batch Size in Stochastic Conditional Gradient Methodscited by 6
  3. 2023PRISE: Demystifying Deep Lucas-Kanade with Strongly Star-Convex Constraints for Multimodel Image Alignmentcited by 4
  4. 2026Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizerscited by 4
  5. 2022A Stochastic Bundle Method for Interpolating Networkscited by 3
  6. 2022Sharper Utility Bounds for Differentially Private Modelscited by 3
  7. 2024Deep Loss Convexification for Learning Iterative Modelscited by 3
  8. 2025The Ball-Proximal (="Broximal") Point Method: a New Algorithm, Convergence Theory, and Applicationscited by 3
  9. 2025Deterministic Global Optimization of the Acquisition Function in Bayesian Optimization: To Do or Not To Do?cited by 3
  10. 2025Minimisation of Quasar-Convex Functions Using Random Zeroth-Order Oraclescited by 3
  11. 2026Convex Dominance in Deep Learning I: A Scaling Law of Loss and Learning Ratecited by 3
  12. 2022Tackling benign nonconvexity with smoothing and stochastic gradientscited by 2
  13. 2022On the Convergence to a Global Solution of Shuffling-Type Gradient Algorithmscited by 2
  14. 2022Invariant Aggregator for Defending against Federated Backdoor Attackscited by 2
  15. 2023Efficient Quantum Algorithms for Quantum Optimal Controlcited by 1
  16. 2025On the Convergence Analysis of Muoncited by 1
  17. 2025Smooth Quasar-Convex Optimization with Constraintscited by 1
  18. 2021Training Deep Neural Networks via Branch-and-Boundcited by 0
  19. 2021Improved Learning Rates for Stochastic Optimizationcited by 0
  20. 2021Non-Asymptotic Analysis of Online Multiplicative Stochastic Gradient Descentcited by 0
  21. 2022Prediction intervals for neural network models using weighted asymmetric loss functionscited by 0
  22. 2025Path Regularization: A Near-Complete and Optimal Nonasymptotic Generalization Theory for Multilayer Neural Networks and Double Descent Phenomenoncited by 0
  23. 2026Neural network optimization strategies and the topography of the loss landscapecited by 0
  24. 2026Stochastic Trust-Region Methods for Over-parameterized Modelscited by 0
  25. 2026Stochastic Non-Smooth Convex Optimization with Unbounded Gradientscited by 0
  26. 2026Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuningcited by 0
  27. 2026Sublinearly Structured Deep Neural Networks Achieve Feature Learning Consistency for Compositional Functionscited by 0