mlsanityData to 14 Oct 2026

Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks

arXiv 1811.08888 ↗First posted 21 Nov 2018cs.LG cs.AI math.OCcited by 141

Not on the map. Why?

Abstract (arXiv)

We study the problem of training deep neural networks with Rectified Linear Unit (ReLU) activation function using gradient descent and stochastic gradient descent. In particular, we study the binary classification problem and show that for a broad family of loss functions, with proper random weight initialization, both gradient descent and stochastic gradient descent can find the global minima of the training loss for an over-parameterized deep ReLU network, under mild assumption on the training data. The key idea of our proof is that Gaussian random initialization followed by (stochastic) gradient descent produces a sequence of iterates that stay inside a small perturbation region centering around the initial weights, in which the empirical loss function of deep ReLU networks enjoys nice local curvature properties that ensure the global convergence of (stochastic) gradient descent. Our theoretical results shed light on understanding the optimization for deep learning, and pave the way for studying the optimization dynamics of training modern deep neural networks.

Key facts. Cited by 141 papers in the map (data through 14 Oct 2026). First posted on arXiv 21 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 56; 2022: 35; 2023: 17; 2024: 21; 2025: 7; 2026: 5.

141 papers.

Sort:

By area:

  1. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  2. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  3. 2021On the Convergence and Robustness of Adversarial Trainingcited by 130
  4. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  5. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  6. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 110
  7. 2021Learning with invariances in random features and kernel modelscited by 89
  8. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  9. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 83
  10. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  11. 2021Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functionscited by 49
  12. 2022A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Featurescited by 49
38 more
  1. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
  2. 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 43
  3. 2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 38
  4. 2021On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layerscited by 35
  5. 2024Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiencycited by 35
  6. 2021Early-stopped neural networks are consistentcited by 34
  7. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  8. 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
  9. 2021Scaling Neural Tangent Kernels via Sketching and Random Featurescited by 32
  10. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  11. 2021Subquadratic Overparameterization for Shallow Neural Networkscited by 26
  12. 2022On the Generalization Mystery in Deep Learningcited by 25
  13. 2021Overparameterization of deep ResNet: zero loss and mean-field analysiscited by 24
  14. 2021Embedding Principle: a hierarchical structure of loss landscape of deep neural networkscited by 22
  15. 2022Characterizing the Spectrum of the NTK via a Power Series Expansioncited by 22
  16. 2021Proxy Convexity: A Unified Framework for the Analysis of Neural Networks Trained by Gradient Descentcited by 21
  17. 2021Learning with convolution and pooling operations in kernel methodscited by 21
  18. 2022Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networkscited by 21
  19. 2021Understanding Deflation Process in Over-parametrized Tensor Decompositioncited by 20
  20. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
  21. 2021Noether: The More Things Change, the More Stay the Samecited by 19
  22. 2021Provable Generalization of SGD-trained Neural Networks of Any Width in the Presence of Adversarial Label Noisecited by 17
  23. 2021A global convergence theory for deep ReLU implicit networks via over-parameterizationcited by 17
  24. 2021Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networkscited by 17
  25. 2022Empirical Phase Diagram for Three-layer Neural Networks with Infinite Widthcited by 17
  26. 2021Learning with Gradient Descent and Weakly Convex Lossescited by 16
  27. 2021Global Convergence of Three-layer Neural Networks in the Mean Field Regimecited by 16
  28. 2021On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Settingcited by 16
  29. 2022An Analysis of Attention via the Lens of Exchangeability and Latent Variable Modelscited by 16
  30. 2023Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networkscited by 16
  31. 2021Neural Tangent Kernel Maximum Mean Discrepancycited by 15
  32. 2023Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networkscited by 15
  33. 2024Exploring the Frontiers of Softmax: Provable Optimization, Applications in Diffusion Model, and Beyondcited by 15
  34. 2021A proof of convergence for gradient descent in the training of artificial neural networks for constant target functionscited by 14
  35. 2021Towards Deepening Graph Neural Networks: A GNTK-based Optimization Perspectivecited by 14
  36. 2021Generalization Guarantees for Neural Architecture Search with Train-Validation Splitcited by 14
  37. 2023A Unified Approach to Controlling Implicit Regularization via Mirror Descentcited by 14
  38. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 14

50 most cited of 141 All 141 (JSON)