# Consistency and fluctuations for stochastic gradient Langevin dynamics

arXiv: https://arxiv.org/abs/1409.0578 · HTML: https://mlsanity.com/paper/1409.0578/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Sep 2014.

## Built on by (most cited first, 30 of 65)

- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Differentiable Annealed Importance Sampling and the Perils of Gradient Noise](https://mlsanity.com/paper/2107.10211/) (2021)
- [Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlo](https://mlsanity.com/paper/2305.18246/) (2023)
- [Time-independent Generalization Bounds for SGLD in Non-convex Settings](https://mlsanity.com/paper/2111.12876/) (2021)
- [Bayesian autoencoders for data-driven discovery of coordinates, governing equations and fundamental constants](https://mlsanity.com/paper/2211.10575/) (2022)
- [On Convergence of Federated Averaging Langevin Dynamics](https://mlsanity.com/paper/2112.05120/) (2021)
- [Enhanced DACER Algorithm with High Diffusion Efficiency](https://mlsanity.com/paper/2505.23426/) (2025)
- [Deep Bayesian inference for seismic imaging with tasks](https://mlsanity.com/paper/2110.04825/) (2021)
- [Scalable Bayesian Uncertainty Quantification for Neural Network Potentials: Promise and Pitfalls](https://mlsanity.com/paper/2212.07959/) (2022)
- [Nonlinear Sufficient Dimension Reduction with a Stochastic Neural Network](https://mlsanity.com/paper/2210.04349/) (2022)
- [Accelerating Approximate Thompson Sampling with Underdamped Langevin Monte Carlo](https://mlsanity.com/paper/2401.11665/) (2024)
- [Interacting Contour Stochastic Gradient Langevin Dynamics](https://mlsanity.com/paper/2202.09867/) (2022)
- [Energy-based Hopfield Boosting for Out-of-Distribution Detection](https://mlsanity.com/paper/2405.08766/) (2024)
- [The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions](https://mlsanity.com/paper/2506.13234/) (2025)
- [Taming Score-Based Diffusion Priors for Infinite-Dimensional Nonlinear Inverse Problems](https://mlsanity.com/paper/2405.15676/) (2024)
- [Non-Reversible Langevin Algorithms for Constrained Sampling](https://mlsanity.com/paper/2501.11743/) (2025)
- [Stein ICP for Uncertainty Estimation in Point Cloud Matching](https://mlsanity.com/paper/2106.03287/) (2021)
- [Energy-Based Models for Functional Data using Path Measure Tilting](https://mlsanity.com/paper/2202.01929/) (2022)
- [Sampling and estimation on manifolds using the Langevin diffusion](https://mlsanity.com/paper/2312.14882/) (2023)
- [From Global to Local: A Scalable Benchmark for Local Posterior Sampling](https://mlsanity.com/paper/2507.21449/) (2025)
- [Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory](https://mlsanity.com/paper/2510.12077/) (2025)
- [Decentralized Bayesian Learning with Metropolis-Adjusted Hamiltonian Monte Carlo](https://mlsanity.com/paper/2107.07211/) (2021)
- [Geometry-informed irreversible perturbations for accelerated convergence of Langevin dynamics](https://mlsanity.com/paper/2108.08247/) (2021)
- [A sharp uniform-in-time error estimate for Stochastic Gradient Langevin Dynamics](https://mlsanity.com/paper/2207.09304/) (2022)
- [Tuning Stochastic Gradient Algorithms for Statistical Inference via Large-Sample Asymptotics](https://mlsanity.com/paper/2207.12395/) (2022)
- [Langevin Quasi-Monte Carlo](https://mlsanity.com/paper/2309.12664/) (2023)
- [Variational Learning of Gaussian Process Latent Variable Models through Stochastic Gradient Annealed Importance Sampling](https://mlsanity.com/paper/2408.06710/) (2024)
- [DIGing--SGLD: Decentralized and Scalable Langevin Sampling over Time--Varying Networks](https://mlsanity.com/paper/2511.12836/) (2025)
- [Stagewise Reinforcement Learning and the Geometry of the Regret Landscape](https://mlsanity.com/paper/2601.07524/) (2026)
- [Bayesian Learning of Parameterised Quantum Circuits](https://mlsanity.com/paper/2206.07559/) (2022)

## Abstract (arXiv)

Applying standard Markov chain Monte Carlo (MCMC) algorithms to large data sets is computationally expensive. Both the calculation of the acceptance probability and the creation of informed proposals usually require an iteration through the whole data set. The recently proposed stochastic gradient Langevin dynamics (SGLD) method circumvents this problem by generating proposals which are only based on a subset of the data, by skipping the accept-reject step and by using decreasing step-sizes sequence (δm)m ≥ 0. %Under appropriate Lyapunov conditions, We provide in this article a rigorous mathematical framework for analysing this algorithm. We prove that, under verifiable assumptions, the algorithm is consistent, satisfies a central limit theorem (CLT) and its asymptotic bias-variance decomposition can be characterized by an explicit functional of the step-sizes sequence (δm)m ≥ 0. We leverage this analysis to give practical recommendations for the notoriously difficult tuning of this algorithm: it is asymptotically optimal to use a step-size sequence of the type δm asymp m-1/3, leading to an algorithm whose mean squared error (MSE) decreases at rate O(m-1/3)