mlsanityData to 7 Oct 2026

Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Network

arXiv 1902.07111 ↗First posted 19 Feb 2019cs.LG math.OC stat.MLcited by 27

Not on the map. Why?

Abstract (arXiv)

Adaptive gradient methods like AdaGrad are widely used in optimizing neural networks. Yet, existing convergence guarantees for adaptive gradient methods require either convexity or smoothness, and, in the smooth setting, only guarantee convergence to a stationary point. We propose an adaptive gradient method and show that for two-layer over-parameterized neural networks -- if the width is sufficiently large (polynomially) -- then the proposed method converges to the global minimum in polynomial time, and convergence is robust, without the need to fine-tune hyper-parameters such as the step-size schedule and with the level of over-parametrization independent of the training error. Our analysis indicates in particular that over-parametrization is crucial for the harnessing the full potential of adaptive gradient methods in the setting of neural networks.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 11; 2023: 1; 2024: 3; 2025: 2; 2026: 1.

27 papers.

Sort:

  1. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
  2. 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
  3. 2023Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedupcited by 23
  4. 2022Training Overparametrized Neural Networks in Sublinear Timecited by 22
  5. 2021Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networkscited by 21
  6. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
  7. 2022AsyncFedED: Asynchronous Federated Learning with Euclidean Distance based Adaptive Weight Aggregationcited by 19
  8. 2021Ghosts in Neural Networks: Existence, Structure and Role of Infinite-Dimensional Null Spacecited by 7
  9. 2021Provable Convergence of Nesterov's Accelerated Gradient Method for Over-Parameterized Neural Networkscited by 7
  10. 2021Provable Regret Bounds for Deep Online Learning and Controlcited by 6
  11. 2024Regularized DeepIV with Model Selectioncited by 6
  12. 2022Riemannian Natural Gradient Methodscited by 5
15 more
  1. 2022Global Convergence of SGD On Two Layer Neural Netscited by 5
  2. 2022Finite Sample Identification of Wide Shallow Neural Networks with Biasescited by 5
  3. 2021A Convergence Theory Towards Practical Over-parameterized Deep Neural Networkscited by 4
  4. 2021AdaLoss: A computationally-efficient and provably convergent adaptive gradient methodcited by 3
  5. 2022Mean-field analysis for heavy ball methods: Dropout-stability, connectivity, and global convergencecited by 3
  6. 2021AutoDrop: Training Deep Learning Models with Automatic Learning Rate Dropcited by 2
  7. 2022A Framework for Overparameterized Learningcited by 2
  8. 2024Regularized Gradient Clipping Provably Trains Wide and Deep Neural Networkscited by 2
  9. 2026SHANG++: Robust Stochastic Acceleration under Multiplicative Noisecited by 2
  10. 2022Parameter Convex Neural Networkscited by 1
  11. 2022Improved Convergence Guarantees for Shallow Neural Networkscited by 1
  12. 2025MAC: An Efficient Gradient Preconditioning using Mean Activation Approximated Curvaturecited by 1
  13. 2025Quantitative convergence of trained single layer neural networks to Gaussian processescited by 1
  14. 2021Dynamics of Local Elasticity During Training of Neural Netscited by 0
  15. 2022Demystifying the Global Convergence Puzzle of Learning Over-parameterized ReLU Nets in Very High Dimensionscited by 0