mlsanityData to 14 Oct 2026

Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networks

arXiv 2206.02139 ↗First posted 5 Jun 2022cs.LG math.OCcited by 5

Not on the map. Why?

Abstract (arXiv)

The convergence of GD and SGD when training mildly parameterized neural networks starting from random initialization is studied. For a broad range of models and loss functions, including the most commonly used square loss and cross entropy loss, we prove an ``early stage convergence'' result. We show that the loss is decreased by a significant amount in the early stage of the training, and this decrease is fast. Furthurmore, for exponential type loss functions, and under some assumptions on the training data, we show global convergence of GD. Instead of relying on extreme over-parameterization, our study is based on a microscopic analysis of the activation patterns for the neurons, which helps us derive more powerful lower bounds for the gradient. The results on activation patterns, which we call ``neuron partition'', help build intuitions for understanding the behavior of neural networks' training dynamics, and may be of independent interest.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 41 of its 58 references that are in the map. First posted on arXiv 5 Jun 2022.

Builds on

41 of 58 references.

References · 41

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  2. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  4. 2018On Lazy Training in Differentiable Programmingcited by 675
  5. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  6. 2014Convex Optimization: Algorithms and Complexitycited by 575
  7. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  8. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  9. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  10. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  11. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  12. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
29 more
  1. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  2. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  3. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  4. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  5. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  6. 2020Directional convergence and alignment in deep learningcited by 178
  7. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 141
  8. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  9. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  10. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  11. 2017Recovery Guarantees for One-hidden-layer Neural Networkscited by 94
  12. 2022Understanding the unstable convergence of gradient descentcited by 86
  13. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 83
  14. 2014On the Computational Efficiency of Training Neural Networkscited by 71
  15. 2020The Global Landscape of Neural Networks: An Overviewcited by 57
  16. 2017SGD Learns the Conjugate Kernel Class of the Networkcited by 54
  17. 2017An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysiscited by 52
  18. 2020Learning a Single Neuron with Gradient Methodscited by 50
  19. 2019A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamicscited by 49
  20. 2021A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Networkcited by 47
  21. 2020The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networkscited by 33
  22. 2021A proof of convergence for gradient descent in the training of artificial neural networks for constant target functionscited by 14
  23. 2022Learning a Single Neuron for Non-monotonic Activation Functionscited by 13
  24. 2020When does gradient descent with logistic loss find interpolating two-layer networks?cited by 12
  25. 2021A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functionscited by 10
  26. 2021When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?cited by 9
  27. 2020The Quenching-Activation Behavior of the Gradient Descent Dynamics for Two-layer Neural Network Modelscited by 8
  28. 2019Analysis of the Gradient Descent Algorithm for a Deep Neural Network Model with Skip-connectionscited by 7
  29. 2020Deep Gated Networks: A framework to understand training and generalisation in deep learningcited by 1

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 2; 2025: 1.

5 papers.

  1. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 42
  2. 2023Early Neuron Alignment in Two-layer ReLU Networks with Small Initializationcited by 28
  3. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  4. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 14
  5. 2024Directional Convergence Near Small Initializations and Saddles in Two-Homogeneous Neural Networkscited by 11