# Bridging the Gap between Stochastic Gradient MCMC and Stochastic Optimization

arXiv: https://arxiv.org/abs/1512.07962 · HTML: https://mlsanity.com/paper/1512.07962/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Dec 2015.

## Built on by (most cited first, 17 of 17)

- [Is MC Dropout Bayesian?](https://mlsanity.com/paper/2110.04286/) (2021)
- [Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect](https://mlsanity.com/paper/2106.06596/) (2021)
- [Efficient and Generalizable Tuning Strategies for Stochastic Gradient MCMC](https://mlsanity.com/paper/2105.13059/) (2021)
- [Tuning-Free Maximum Likelihood Training of Latent Variable Models via Coin Betting](https://mlsanity.com/paper/2305.14916/) (2023)
- [Langevin Quasi-Monte Carlo](https://mlsanity.com/paper/2309.12664/) (2023)
- [Uncertainty quantification in non-rigid image registration via stochastic gradient Markov chain Monte Carlo](https://mlsanity.com/paper/2110.13289/) (2021)
- [AdamMCMC: Combining Metropolis Adjusted Langevin with Momentum-based Optimization](https://mlsanity.com/paper/2312.14027/) (2023)
- [Generative vs. Discriminative modeling under the lens of uncertainty quantification](https://mlsanity.com/paper/2406.09172/) (2024)
- [Generalized Bayesian deep reinforcement learning](https://mlsanity.com/paper/2412.11743/) (2024)
- [Density Regression and Uncertainty Quantification with Bayesian Deep Noise Neural Networks](https://mlsanity.com/paper/2206.05643/) (2022)
- [Bayesian Computation in Deep Learning](https://mlsanity.com/paper/2502.18300/) (2025)
- [Bayesian analysis of the prevalence bias: learning and predicting from imbalanced data](https://mlsanity.com/paper/2108.00250/) (2021)
- [Dense Uncertainty Estimation via an Ensemble-based Conditional Latent Variable Model](https://mlsanity.com/paper/2111.11055/) (2021)
- [Approximate Inference via Clustering](https://mlsanity.com/paper/2111.14219/) (2021)
- [ADASR: An Adversarial Auto-Augmentation Framework for Hyperspectral and Multispectral Data Fusion](https://mlsanity.com/paper/2310.07255/) (2023)
- [MCMC: Bridging Rendering, Optimization and Generative AI](https://mlsanity.com/paper/2510.09078/) (2025)
- [Prequential posteriors](https://mlsanity.com/paper/2511.17721/) (2025)

## Abstract (arXiv)

Stochastic gradient Markov chain Monte Carlo (SG-MCMC) methods are Bayesian analogs to popular stochastic optimization methods; however, this connection is not well studied. We explore this relationship by applying simulated annealing to an SGMCMC algorithm. Furthermore, we extend recent SG-MCMC methods with two key components: i) adaptive preconditioners (as in ADAgrad or RMSprop), and ii) adaptive element-wise momentum weights. The zero-temperature limit gives a novel stochastic optimization method with adaptive element-wise momentum weights, while conventional optimization methods only have a shared, static momentum weight. Under certain assumptions, our theoretical analysis suggests the proposed simulated annealing approach converges close to the global optima. Experiments on several deep neural network models show state-of-the-art results compared to related stochastic optimization algorithms.