A sharp uniform-in-time error estimate for Stochastic Gradient Langevin Dynamics
Not on the map. Why?
Abstract (arXiv)
We establish a sharp uniform-in-time error estimate for the Stochastic Gradient Langevin Dynamics (SGLD), which is a widely-used sampling algorithm. Under mild assumptions, we obtain a uniform-in-time O(η2) bound for the KL-divergence between the SGLD iteration and the Langevin diffusion, where η is the step size (or learning rate). Our analysis is also valid for varying step sizes. Consequently, we are able to derive an O(η) bound for the distance between the invariant measures of the SGLD iteration and the Langevin diffusion, in terms of Wasserstein or total variation distances. Our result can be viewed as a significant improvement compared with existing analysis for SGLD in related literature.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 21 of its 63 references that are in the map. First posted on arXiv 19 Jul 2022.
Builds on
21 of 63 references.
References · 21
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
- 2014Theoretical guarantees for approximate sampling from smooth and log-concave densitiescited by 206
- 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
- 2015Adding Gradient Noise Improves Learning for Very Deep Networkscited by 144
- 2017User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradientcited by 116
- 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
- 2017Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpointscited by 82
- 2020On the Generalization Benefit of Noise in Stochastic Gradient Descentcited by 81
- 2017A Hitting Time Analysis of Stochastic Gradient Langevin Dynamicscited by 70
- 2014Consistency and fluctuations for stochastic gradient Langevin dynamicscited by 65
- 2018The promises and pitfalls of Stochastic Gradient Langevin Dynamicscited by 56
9 more
- 2019Improved Bounds for Discretization of Langevin Diffusions: Near-Optimal Rates without Convexitycited by 38
- 2019Nonasymptotic estimates for Stochastic Gradient Langevin Dynamics under local conditions in nonconvex optimizationcited by 38
- 2018Non-asymptotic bounds for sampling algorithms without log-concavitycited by 35
- 2020Faster Convergence of Stochastic Gradient Langevin Dynamics for Non-Log-Concave Samplingcited by 32
- 2021Time-independent Generalization Bounds for SGLD in Non-convex Settingscited by 27
- 2019Uniform-in-Time Weak Error Analysis for Stochastic Gradient Descent Algorithms via Diffusion Approximationcited by 14
- 2021SGD with a Constant Large Learning Rate Can Converge to Local Maximacited by 8
- 2015Scalable MCMC for Mixed Membership Stochastic Blockmodelscited by 7
- 2022On uniform-in-time diffusion approximation for stochastic gradient descentcited by 5
Built on by
Citing papers per year: 2022: 1; 2023: 1; 2024: 1.
3 papers.