mlsanityData to 7 Oct 2026

Convergence of Stochastic Gradient Langevin Dynamics in the Lazy Training Regime

arXiv 2510.21245 ↗First posted 24 Oct 2025cs.LG math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

Continuous-time models provide important insights into the training dynamics of optimization algorithms in deep learning. In this work, we establish a non-asymptotic convergence analysis of stochastic gradient Langevin dynamics (SGLD), which is an Ito stochastic differential equation (SDE) approximation of stochastic gradient descent in continuous time, in the lazy training regime. We show that, under regularity conditions on the Hessian of the loss function, SGLD with multiplicative and state-dependent noise (i) yields a non-degenerate kernel throughout the training process with high probability, and (ii) achieves exponential convergence to the empirical risk minimizer in expectation, and we establish finite-time and finite-width bounds on the optimality gap. We corroborate our theoretical findings with numerical examples in the regression setting.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 19 references that are in the map. First posted on arXiv 24 Oct 2025.

Builds on

16 of 19 references.

References · 16

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  3. 2018On Lazy Training in Differentiable Programmingcited by 671
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  5. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  6. 2014Convex Optimization: Algorithms and Complexitycited by 569
  7. 2019Benign Overfitting in Linear Regressioncited by 544
  8. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  9. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 213
  10. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  11. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
  12. 2020The Heavy-Tail Phenomenon in SGDcited by 122
4 more
  1. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 100
  2. 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
  3. 2024Convergence of Gradient Descent for Recurrent Neural Networks: A Nonasymptotic Analysiscited by 7
  4. 2024Convergence of continuous-time stochastic gradient descent with applications to deep neural networkscited by 4

Built on by

Not cited yet.