mlsanityData to 14 Oct 2026

Mean-field analysis for heavy ball methods: Dropout-stability, connectivity, and global convergence

arXiv 2210.06819 ↗First posted 13 Oct 2022cs.LG stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

The stochastic heavy ball method (SHB), also known as stochastic gradient descent (SGD) with Polyak's momentum, is widely used in training neural networks. However, despite the remarkable success of such algorithm in practice, its theoretical characterization remains limited. In this paper, we focus on neural networks with two and three layers and provide a rigorous understanding of the properties of the solutions found by SHB: (i) stability after dropping out part of the neurons, (ii) connectivity along a low-loss path, and (iii) convergence to the global optimum. To achieve this goal, we take a mean-field view and relate the SHB dynamics to a certain partial differential equation in the limit of large network widths. This mean-field perspective has inspired a recent line of work focusing on SGD while, in contrast, our paper considers an algorithm with momentum. More specifically, after proving existence and uniqueness of the limit differential equations, we show convergence to the global optimum and give a quantitative bound between the mean-field limit and the SHB dynamics of a finite-width network. Armed with this last bound, we are able to establish the dropout-stability and connectivity of SHB solutions.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 43 of its 63 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

43 of 63 references.

References · 43

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  4. 2018On Lazy Training in Differentiable Programmingcited by 675
  5. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  6. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  7. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  8. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  9. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
  10. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 291
  11. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
  12. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 206
31 more
  1. 2014Qualitatively characterizing neural network optimization problemscited by 188
  2. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  3. 2016A Variational Perspective on Accelerated Methods in Optimizationcited by 110
  4. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  5. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  6. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 96
  7. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  8. 2018Understanding the Acceleration Phenomenon via High-Resolution Differential Equationscited by 69
  9. 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68
  10. 2019Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Netscited by 63
  11. 2022Mean-Field Langevin Dynamics: Exponential Convergence and Annealingcited by 61
  12. 2020A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depthcited by 56
  13. 2022Convex Analysis of the Mean Field Langevin Dynamicscited by 55
  14. 2019Mean-Field Langevin Dynamics and Energy Landscape of Neural Networkscited by 50
  15. 2019Gradient Dynamics of Shallow Univariate ReLU Networkscited by 46
  16. 2020Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networkscited by 44
  17. 2022Towards understanding how momentum improves generalization in deep learningcited by 40
  18. 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
  19. 2019Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networkscited by 35
  20. 2019Continuous Time Analysis of Momentum Methodscited by 32
  21. 2018Small nonlinearities in activation functions create bad local minima in neural networkscited by 28
  22. 2019Analysis of a Two-Layer Neural Network via Displacement Convexitycited by 27
  23. 2019Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Networkcited by 27
  24. 2020A Dynamical Central Limit Theorem for Shallow Neural Networkscited by 23
  25. 2020A Modular Analysis of Provable Acceleration via Polyak's Momentum: Training a Wide ReLU Network and a Deep Linear Networkcited by 19
  26. 2021Global Convergence of Three-layer Neural Networks in the Mean Field Regimecited by 16
  27. 2021Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networkscited by 13
  28. 2020Ergodicity of the underdamped mean-field Langevin dynamicscited by 7
  29. 2021Training Integrable Parameterizations of Deep Neural Networks in the Infinite-Width Limitcited by 7
  30. 2021Limiting fluctuation and trajectorial stability of multilayer neural networks with mean field trainingcited by 5
  31. 2020Global Convergence of Second-order Dynamics in Two-layer Neural Networkscited by 1

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 0; 2025: 1.

3 papers.

  1. 2023Fundamental limits of overparametrized shallow neural networks for supervised learningcited by 13
  2. 2023Exponential convergence rates for momentum stochastic gradient descent in the overparametrized settingcited by 8
  3. 2025Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regimecited by 6