Langevin Monte-Carlo Provably Learns Depth Two Neural Nets at Any Size and Data
Not on the map. Why?
Abstract (arXiv)
In this work, we will establish that the Langevin Monte-Carlo algorithm can learn depth-2 neural nets of any size and for any data and we give non-asymptotic convergence rates for it. We achieve this via showing that in q-Renyi divergence, the iterates of Langevin Monte Carlo converge to the Gibbs distribution of Frobenius norm regularized losses for any of these nets, when using smooth activations and in both classification and regression settings. Most critically, the amount of regularization needed for our results is independent of the size of the net. This result achieves a synthesis of several recent observations about isoperimetry conditions under which LMC converges and that two-layer neural loss functions can always be regularized by a certain constant amount such that they satisfy the Villani conditions, and thus their Gibbs measures satisfy a Poincare inequality.
Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Builds on 37 of its 57 references that are in the map. First posted on arXiv 13 Mar 2025.
Builds on
37 of 57 references.
References · 37
- 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 786
- 2018On Lazy Training in Differentiable Programmingcited by 675
- 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 658
- 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
- 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 565
- 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 465
- 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 425
- 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
- 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
- 2019Rapid Convergence of the Unadjusted Langevin Algorithm: Isoperimetry Sufficescited by 224
- 2015Adding Gradient Noise Improves Learning for Very Deep Networkscited by 144
- 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
25 more
- 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 121
- 2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
- 2017Recovery Guarantees for One-hidden-layer Neural Networkscited by 94
- 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
- 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
- 2022Towards a Theory of Non-Log-Concave Sampling: First-Order Stationarity Guarantees for Langevin Monte Carlocited by 63
- 2022Mean-Field Langevin Dynamics: Exponential Convergence and Annealingcited by 60
- 2020On the Convergence of Langevin Monte Carlo: The Interplay between Tail Growth and Smoothnesscited by 55
- 2021A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Networkcited by 47
- 2018Global Non-convex Optimization with Discretized Diffusionscited by 44
- 2020Convergence of Langevin Monte Carlo in Chi-Squared and Renyi Divergencecited by 42
- 2024Numerical analysis of physics-informed neural networks and related models in physics-informed machine learningcited by 39
- 2019Improved Bounds for Discretization of Langevin Diffusions: Near-Optimal Rates without Convexitycited by 38
- 2020On Learning Rates and Schr"odinger Operatorscited by 33
- 2023Towards a Complete Analysis of Langevin Monte Carlo: Beyond Poincar'e Inequalitycited by 31
- 2019On Learning Over-parameterized Neural Networks: A Functional Approximation Perspectivecited by 28
- 2021Subquadratic Overparameterization for Shallow Neural Networkscited by 26
- 2018Learning Two-layer Neural Networks with Symmetric Inputscited by 22
- 2021Efficient Algorithms for Learning Depth-2 Neural Networks with General ReLU Activationscited by 15
- 2020Depth-2 Neural Networks Under a Data-Poisoning Attackcited by 6
- 2022Gradient descent provably escapes saddle points in the training of shallow ReLU networkscited by 5
- 2022Global Convergence of SGD On Two Layer Neural Netscited by 5
- 2022Concentration of the Langevin Algorithm's Stationary Distributioncited by 5
- 2023Non-Log-Concave and Nonsmooth Sampling via Langevin Monte Carlo Algorithmscited by 4
- 2023Global Convergence of SGD For Logistic Loss on Two Layer Neural Netscited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.