# AdamMCMC: Combining Metropolis Adjusted Langevin with Momentum-based Optimization

arXiv: https://arxiv.org/abs/2312.14027 · HTML: https://mlsanity.com/paper/2312.14027/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 21 Dec 2023.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [Stochastic Gradient Hamiltonian Monte Carlo](https://mlsanity.com/paper/1402.4102/) (2014)
- [User-friendly introduction to PAC-Bayes bounds](https://mlsanity.com/paper/2110.11216/) (2021)
- [A Complete Recipe for Stochastic Gradient MCMC](https://mlsanity.com/paper/1506.04696/) (2015)
- [BayesFlow: Learning complex stochastic models with invertible neural networks](https://mlsanity.com/paper/2003.06281/) (2020)
- [Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks](https://mlsanity.com/paper/1512.07666/) (2015)
- [Particle Transformer for Jet Tagging](https://mlsanity.com/paper/2202.03772/) (2022)
- [ParticleNet: Jet Tagging via Particle Clouds](https://mlsanity.com/paper/1902.08570/) (2019)
- [Machine Learning in the Search for New Fundamental Physics](https://mlsanity.com/paper/2112.03769/) (2021)
- [Langevin diffusions and the Metropolis-adjusted Langevin algorithm](https://mlsanity.com/paper/1309.2983/) (2013)
- [Austerity in MCMC Land: Cutting the Metropolis-Hastings Budget](https://mlsanity.com/paper/1304.5299/) (2013)
- [Bayesian Inference for Large Scale Image Classification](https://mlsanity.com/paper/1908.03491/) (2019)
- [Bridging the Gap between Stochastic Gradient MCMC and Stochastic Optimization](https://mlsanity.com/paper/1512.07962/) (2015)
- [Asymptotically Optimal Exact Minibatch Metropolis-Hastings](https://mlsanity.com/paper/2006.11677/) (2020)
- [An Efficient Minibatch Acceptance Test for Metropolis-Hastings](https://mlsanity.com/paper/1610.06848/) (2016)
- [Laplacian Autoencoders for Learning Stochastic Representations](https://mlsanity.com/paper/2206.15078/) (2022)
- [Hug and Hop: a discrete-time, non-reversible Markov chain Monte-Carlo algorithm](https://mlsanity.com/paper/1907.13570/) (2019)
- [Data Subsampling for Bayesian Neural Networks](https://mlsanity.com/paper/2210.09141/) (2022)

## Built on by (most cited first, 2 of 2)

- [Scalable Bayesian Learning with posteriors](https://mlsanity.com/paper/2406.00104/) (2024)
- [Calibrating Bayesian Generative Machine Learning for Bayesiamplification](https://mlsanity.com/paper/2408.00838/) (2024)

## Abstract (arXiv)

Uncertainty estimation is a key issue when considering the application of deep neural network methods in science and engineering. In this work, we introduce a novel algorithm that quantifies epistemic uncertainty via Monte Carlo sampling from a tempered posterior distribution. It combines the well established Metropolis Adjusted Langevin Algorithm (MALA) with momentum-based optimization using Adam and leverages a prolate proposal distribution, to efficiently draw from the posterior. We prove that the constructed chain admits the Gibbs posterior as invariant distribution and approximates this posterior in total variation distance. Furthermore, we demonstrate the efficiency of the resulting algorithm and the merit of the proposed changes on a state-of-the-art classifier from high-energy particle physics.