mlsanityData to 7 Oct 2026

Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis

arXiv 1702.03849 ↗First posted 13 Feb 2017cs.LG math.OC math.PRcited by 242

Not on the map. Why?

Abstract (arXiv)

Stochastic Gradient Langevin Dynamics (SGLD) is a popular variant of Stochastic Gradient Descent, where properly scaled isotropic Gaussian noise is added to an unbiased estimate of the gradient at each iteration. This modest change allows SGLD to escape local minima and suffices to guarantee asymptotic convergence to global minimizers for sufficiently regular non-convex objectives (Gelfand and Mitter, 1991). The present work provides a nonasymptotic analysis in the context of non-convex learning problems, giving finite-time guarantees for SGLD to find approximate minimizers of both empirical and population risks. As in the asymptotic setting, our analysis relates the discrete-time SGLD Markov chain to a continuous-time diffusion process. A new tool that drives the results is the use of weighted transportation cost inequalities to quantify the rate of convergence of SGLD to a stationary distribution in the Euclidean 2-Wasserstein distance.

Key facts. Cited by 242 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Feb 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 38; 2022: 54; 2023: 48; 2024: 33; 2025: 36; 2026: 32.

242 papers.

Sort:

By area:

  1. 2021Information-Theoretic Generalization Bounds for Stochastic Gradient Descentcited by 84
  2. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 77
  3. 2021Manipulating SGD with Data Ordering Attackscited by 73
  4. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 69
  5. 2022Towards a Theory of Non-Log-Concave Sampling: First-Order Stationarity Guarantees for Langevin Monte Carlocited by 63
  6. 2021Optimizing Functionals on the Space of Probabilities with Input Convex Neural Networkscited by 53
  7. 2021Generalization Bounds For Meta-Learning: An Information-Theoretic Analysiscited by 45
  8. 2023Reverse Diffusion Monte Carlocited by 41
  9. 2021Differentiable Annealed Importance Sampling and the Perils of Gradient Noisecited by 36
  10. 2023Revisiting the Noise Model of Stochastic Gradient Descentcited by 35
  11. 2024Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Datacited by 35
  12. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
38 more
  1. 2023Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behaviorcited by 33
  2. 2024Diffusion Models as Constrained Samplers for Optimization with Unknown Constraintscited by 33
  3. 2023Towards a Complete Analysis of Langevin Monte Carlo: Beyond Poincar'e Inequalitycited by 31
  4. 2021PAC-Bayes Information Bottleneckcited by 28
  5. 2024Conformalized-DeepONet: A Distribution-Free Framework for Uncertainty Quantification in Deep Operator Networkscited by 28
  6. 2021Time-independent Generalization Bounds for SGLD in Non-convex Settingscited by 27
  7. 2023Improved Discretization Analysis for Underdamped Langevin Monte Carlocited by 26
  8. 2023Provably Convergent Schr"odinger Bridge with Applications to Probabilistic Time Series Imputationcited by 26
  9. 2021Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysiscited by 25
  10. 2022Theoretical Perspectives on Deep Learning Methods in Inverse Problemscited by 22
  11. 2022A Proximal Algorithm for Samplingcited by 21
  12. 2022Resolving the Mixing Time of the Langevin Algorithm to its Stationary Distribution for Log-Concave Samplingcited by 21
  13. 2023Tighter Information-Theoretic Generalization Bounds from Supersamplescited by 21
  14. 2021Efficient Reinforced Feature Selection via Early Stopping Traverse Strategycited by 19
  15. 2022Mean-field Variational Inference via Wasserstein Gradient Flowcited by 19
  16. 2024A Unified and General Framework for Continual Learningcited by 19
  17. 2023Guiding Energy-based Models via Contrastive Latent Variablescited by 17
  18. 2023Interacting Particle Langevin Algorithm for Maximum Marginal Likelihood Estimationcited by 17
  19. 2021On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Settingcited by 16
  20. 2022Improved Convergence Rate of Stochastic Gradient Langevin Dynamics with Variance Reduction and its Application to Optimizationcited by 16
  21. 2022Convergence of Stein Variational Gradient Descent under a Weaker Smoothness Conditioncited by 16
  22. 2023Renormalizing Diffusion Modelscited by 16
  23. 2021Generalization Bounds for Noisy Iterative Algorithms Using Properties of Additive Noise Channelscited by 15
  24. 2023Algorithmic Stability of Heavy-Tailed SGD with General Loss Functionscited by 15
  25. 2021Lazy OCO: Online Convex Optimization on a Switching Budgetcited by 14
  26. 2021Provable Compressed Sensing with Generative Priors via Langevin Dynamicscited by 14
  27. 2021Significance tests of feature relevance for a black-box learnercited by 14
  28. 2021Characterizing and Understanding the Generalization Error of Transfer Learning with Gibbs Algorithmcited by 14
  29. 2021On Convergence of Federated Averaging Langevin Dynamicscited by 14
  30. 2022Generalization Error Bounds for Deep Neural Networks Trained by SGDcited by 14
  31. 2022Two-Scale Gradient Descent Ascent Dynamics Finds Mixed Nash Equilibria of Continuous Games: A Mean-Field Perspectivecited by 13
  32. 2023Tighter PAC-Bayes Bounds Through Coin-Bettingcited by 13
  33. 2024Improved Finite-Particle Convergence Rates for Stein Variational Gradient Descentcited by 13
  34. 2021Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injectionscited by 12
  35. 2021Deep Bayesian inference for seismic imaging with taskscited by 12
  36. 2022Algorithmic Stability of Heavy-Tailed Stochastic Gradient Descent on Least Squarescited by 12
  37. 2023Kinetic Langevin MCMC Sampling Without Gradient Lipschitz Continuity -- the Strongly Convex Casecited by 12
  38. 2023On the different regimes of Stochastic Gradient Descentcited by 12

50 most cited of 242 All 242 (JSON)