# AMAGOLD: Amortized Metropolis Adjustment for Efficient Stochastic Gradient MCMC

arXiv: https://arxiv.org/abs/2003.00193 · HTML: https://mlsanity.com/paper/2003.00193/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Feb 2020.

## Built on by (most cited first, 13 of 13)

- [What Are Bayesian Neural Network Posteriors Really Like?](https://mlsanity.com/paper/2104.14421/) (2021)
- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [Exact Langevin Dynamics with Stochastic Gradients](https://mlsanity.com/paper/2102.01691/) (2021)
- [Scalable Bayesian Uncertainty Quantification for Neural Network Potentials: Promise and Pitfalls](https://mlsanity.com/paper/2212.07959/) (2022)
- [Training Bayesian Neural Networks with Sparse Subspace Variational Inference](https://mlsanity.com/paper/2402.11025/) (2024)
- [DiffHybrid-UQ: Uncertainty Quantification for Differentiable Hybrid Neural Modeling](https://mlsanity.com/paper/2401.00161/) (2023)
- [On Feynman--Kac training of partial Bayesian neural networks](https://mlsanity.com/paper/2310.19608/) (2023)
- [A Langevin sampling algorithm inspired by the Adam optimizer](https://mlsanity.com/paper/2504.18911/) (2025)
- [Transgressing the boundaries: towards a rigorous understanding of deep learning and its (non-)robustness](https://mlsanity.com/paper/2307.02454/) (2023)
- [JaxSGMC: Modular stochastic gradient MCMC in JAX](https://mlsanity.com/paper/2505.11190/) (2025)
- [Accelerating Hamiltonian Monte Carlo for Bayesian Inference in Neural Networks and Neural Operators](https://mlsanity.com/paper/2507.14652/) (2025)
- [One-Step Diffusion Samplers via Self-Distillation and Deterministic Flow](https://mlsanity.com/paper/2512.05251/) (2025)
- [Cyclical Kernel Adaptive Metropolis](https://mlsanity.com/paper/2206.14421/) (2022)

## Abstract (arXiv)

Stochastic gradient Hamiltonian Monte Carlo (SGHMC) is an efficient method for sampling from continuous distributions. It is a faster alternative to HMC: instead of using the whole dataset at each iteration, SGHMC uses only a subsample. This improves performance, but introduces bias that can cause SGHMC to converge to the wrong distribution. One can prevent this using a step size that decays to zero, but such a step size schedule can drastically slow down convergence. To address this tension, we propose a novel second-order SG-MCMC algorithm---AMAGOLD---that infrequently uses Metropolis-Hastings (M-H) corrections to remove bias. The infrequency of corrections amortizes their cost. We prove AMAGOLD converges to the target distribution with a fixed, rather than a diminishing, step size, and that its convergence rate is at most a constant factor slower than a full-batch baseline. We empirically demonstrate AMAGOLD's effectiveness on synthetic distributions, Bayesian logistic regression, and Bayesian neural networks.