# Reinforcement Learning for Adaptive MCMC

arXiv: https://arxiv.org/abs/2405.13574 · HTML: https://mlsanity.com/paper/2405.13574/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 15 of its 55 references that are in the map.
- First posted on arXiv 22 May 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 15 of 15)

- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [An Introduction to Deep Reinforcement Learning](https://mlsanity.com/paper/1811.12560/) (2018)
- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)
- [Parameter Space Noise for Exploration](https://mlsanity.com/paper/1706.01905/) (2017)
- [The Barker proposal: combining robustness and efficiency in gradient-based MCMC](https://mlsanity.com/paper/1908.11812/) (2019)
- [Accelerating Asymptotically Exact MCMC for Computationally Intensive Models via Local Approximations](https://mlsanity.com/paper/1402.1694/) (2014)
- [Gradient-based Adaptive Markov Chain Monte Carlo](https://mlsanity.com/paper/1911.01373/) (2019)
- [Efficient and Generalizable Tuning Strategies for Stochastic Gradient MCMC](https://mlsanity.com/paper/2105.13059/) (2021)
- [Accelerating Markov Chain Monte Carlo sampling with diffusion models](https://mlsanity.com/paper/2309.01454/) (2023)
- [Self-Tuning Hamiltonian Monte Carlo for Accelerated Sampling](https://mlsanity.com/paper/2309.13593/) (2023)
- [Multi-agent Reinforcement Learning Accelerated MCMC on Multiscale Inversion Problem](https://mlsanity.com/paper/2011.08954/) (2020)
- [Adaptive Metropolis-Hastings Sampling using Reversible Dependent Mixture Proposals](https://mlsanity.com/paper/1305.2634/) (2013)
- [Policy Gradient Optimal Correlation Search for Variance Reduction in Monte Carlo simulation and Maximum Optimal Transport](https://mlsanity.com/paper/2307.12703/) (2023)

## Built on by (most cited first, 6 of 6)

- [Harnessing the Power of Reinforcement Learning for Adaptive MCMC](https://mlsanity.com/paper/2507.00671/) (2025)
- [Neural Predictor-Corrector: Solving Homotopy Problems with Reinforcement Learning](https://mlsanity.com/paper/2602.03086/) (2026)
- [Stop the Sampler! Classifier-Based Adaptive Stopping for Sampling Kernels](https://mlsanity.com/paper/2606.16073/) (2026)
- [GRPO-QPS: Target-Preserving Reinforcement Learning for Quantum Posterior Sampling](https://mlsanity.com/paper/2609.14711/) (2026)
- [Amortised Inference through One-Step Implicit Sampling](https://mlsanity.com/paper/2610.12679/) (2026)
- [Learning Graph Energies for Finite-Time Generation](https://mlsanity.com/paper/2610.19946/) (2026)

## Abstract (arXiv)

An informal observation, made by several authors, is that the adaptive design of a Markov transition kernel has the flavour of a reinforcement learning task. Yet, to-date it has remained unclear how to actually exploit modern reinforcement learning technologies for adaptive MCMC. The aim of this paper is to set out a general framework, called Reinforcement Learning Metropolis--Hastings, that is theoretically supported and empirically validated. Our principal focus is on learning fast-mixing Metropolis--Hastings transition kernels, which we cast as deterministic policies and optimise via a policy gradient. Control of the learning rate provably ensures conditions for ergodicity are satisfied. The methodology is used to construct a gradient-free sampler that out-performs a popular gradient-free adaptive Metropolis--Hastings algorithm on ≈ 90 % of tasks in the PosteriorDB benchmark.