# Global Convergence of Stochastic Gradient Hamiltonian Monte Carlo for Non-Convex Stochastic Optimization: Non-Asymptotic Performance Bounds and Momentum-Based Acceleration

arXiv: https://arxiv.org/abs/1809.04618 · HTML: https://mlsanity.com/paper/1809.04618/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Sep 2018.

## Built on by (most cited first, 30 of 31)

- [Improved Discretization Analysis for Underdamped Langevin Monte Carlo](https://mlsanity.com/paper/2302.08049/) (2023)
- [AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networks](https://mlsanity.com/paper/2303.00565/) (2023)
- [Algorithmic Stability of Heavy-Tailed SGD with General Loss Functions](https://mlsanity.com/paper/2301.11885/) (2023)
- [Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections](https://mlsanity.com/paper/2102.07006/) (2021)
- [Kinetic Langevin MCMC Sampling Without Gradient Lipschitz Continuity -- the Strongly Convex Case](https://mlsanity.com/paper/2301.08039/) (2023)
- [Memory-Efficient Optimization with Factorized Hamiltonian Descent](https://mlsanity.com/paper/2406.09958/) (2024)
- [Nonlinear Sufficient Dimension Reduction with a Stochastic Neural Network](https://mlsanity.com/paper/2210.04349/) (2022)
- [Kinetic Interacting Particle Langevin Monte Carlo](https://mlsanity.com/paper/2407.05790/) (2024)
- [Uniform-in-Time Wasserstein Stability Bounds for (Noisy) Stochastic Gradient Descent](https://mlsanity.com/paper/2305.12056/) (2023)
- [Neural Operator Variational Inference based on Regularized Stein Discrepancy for Deep Gaussian Processes](https://mlsanity.com/paper/2309.12658/) (2023)
- [A Unifying and Canonical Description of Measure-Preserving Diffusions](https://mlsanity.com/paper/2105.02845/) (2021)
- [Decentralized Bayesian Learning with Metropolis-Adjusted Hamiltonian Monte Carlo](https://mlsanity.com/paper/2107.07211/) (2021)
- [Accelerating Langevin Monte Carlo Sampling: A Large Deviations Analysis](https://mlsanity.com/paper/2503.19066/) (2025)
- [Revisiting the Effects of Stochasticity for Hamiltonian Samplers](https://mlsanity.com/paper/2106.16200/) (2021)
- [Privacy of SGD under Gaussian or Heavy-Tailed Noise: Guarantees without Gradient Clipping](https://mlsanity.com/paper/2403.02051/) (2024)
- [Causal-StoNet: Causal Inference for High-Dimensional Complex Data](https://mlsanity.com/paper/2403.18994/) (2024)
- [Regime-Switching Langevin Monte Carlo Algorithms](https://mlsanity.com/paper/2509.00941/) (2025)
- [Adaptive Momentum and Nonlinear Damping for Neural Network Training](https://mlsanity.com/paper/2602.00334/) (2026)
- [Global convergence of optimized adaptive importance samplers](https://mlsanity.com/paper/2201.00409/) (2022)
- [Cyclic and Randomized Stepsizes Invoke Heavier Tails in SGD than Constant Stepsize](https://mlsanity.com/paper/2302.05516/) (2023)
- [Accelerated Gradient Methods for Nonconvex Optimization: Escape Trajectories From Strict Saddle Points and Convergence to Local Minima](https://mlsanity.com/paper/2307.07030/) (2023)
- [Variance Reduction for Stochastic Gradient Generalized Non-reversible Langevin Monte Carlo Algorithms](https://mlsanity.com/paper/2606.28808/) (2026)
- [Hamiltonian Monte Carlo with Asymmetrical Momentum Distributions](https://mlsanity.com/paper/2110.12907/) (2021)
- [Subsampling Error in Stochastic Gradient Langevin Diffusions](https://mlsanity.com/paper/2305.13882/) (2023)
- [Error estimates between SGD with momentum and underdamped Langevin diffusion](https://mlsanity.com/paper/2410.17297/) (2024)
- [Contractive kinetic Langevin samplers beyond global Lipschitz continuity](https://mlsanity.com/paper/2509.12031/) (2025)
- [Improved Analysis for Hessian-free High-resolution Monte Carlo Sampling](https://mlsanity.com/paper/2608.25052/) (2026)
- [A New Paradigm for Generative Adversarial Networks based on Randomized Decision Rules](https://mlsanity.com/paper/2306.13641/) (2023)
- [A General Continuous-Time Formulation of Stochastic ADMM and Its Variants](https://mlsanity.com/paper/2404.14358/) (2024)
- [On Convergence of the Alternating Directions SGHMC Algorithm](https://mlsanity.com/paper/2405.13140/) (2024)

## Abstract (arXiv)

Stochastic gradient Hamiltonian Monte Carlo (SGHMC) is a variant of stochastic gradient with momentum where a controlled and properly scaled Gaussian noise is added to the stochastic gradients to steer the iterates towards a global minimum. Many works reported its empirical success in practice for solving stochastic non-convex optimization problems, in particular it has been observed to outperform overdamped Langevin Monte Carlo-based methods such as stochastic gradient Langevin dynamics (SGLD) in many applications. Although asymptotic global convergence properties of SGHMC are well known, its finite-time performance is not well-understood. In this work, we study two variants of SGHMC based on two alternative discretizations of the underdamped Langevin diffusion. We provide finite-time performance bounds for the global convergence of both SGHMC variants for solving stochastic non-convex optimization problems with explicit constants. Our results lead to non-asymptotic guarantees for both population and empirical risk minimization problems. For a fixed target accuracy level, on a class of non-convex problems, we obtain complexity bounds for SGHMC that can be tighter than those for SGLD. These results show that acceleration with momentum is possible in the context of global non-convex optimization.