# Harnessing the Power of Reinforcement Learning for Adaptive MCMC

arXiv: https://arxiv.org/abs/2507.00671 · HTML: https://mlsanity.com/paper/2507.00671/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 18 of its 52 references that are in the map.
- First posted on arXiv 1 Jul 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 18 of 18)

- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [Annealed Flow Transport Monte Carlo](https://mlsanity.com/paper/2102.07501/) (2021)
- [Underdamped Diffusion Bridges with Applications to Sampling](https://mlsanity.com/paper/2503.01006/) (2025)
- [The Barker proposal: combining robustness and efficiency in gradient-based MCMC](https://mlsanity.com/paper/1908.11812/) (2019)
- [Sampling via Gradient Flows in the Space of Probability Measures](https://mlsanity.com/paper/2310.03597/) (2023)
- [Gradient-based Adaptive Markov Chain Monte Carlo](https://mlsanity.com/paper/1911.01373/) (2019)
- [Efficient and Generalizable Tuning Strategies for Stochastic Gradient MCMC](https://mlsanity.com/paper/2105.13059/) (2021)
- [Accelerating Markov Chain Monte Carlo sampling with diffusion models](https://mlsanity.com/paper/2309.01454/) (2023)
- [Tuning Sequential Monte Carlo Samplers via Greedy Incremental Divergence Minimization](https://mlsanity.com/paper/2503.15704/) (2025)
- [Reinforcement Learning for Adaptive MCMC](https://mlsanity.com/paper/2405.13574/) (2024)
- [Convergence of position-dependent MALA with application to conditional simulation in GLMMs](https://mlsanity.com/paper/2108.12662/) (2021)
- [Self-Tuning Hamiltonian Monte Carlo for Accelerated Sampling](https://mlsanity.com/paper/2309.13593/) (2023)
- [AutoStep: Locally adaptive involutive MCMC](https://mlsanity.com/paper/2410.18929/) (2024)
- [Multi-agent Reinforcement Learning Accelerated MCMC on Multiscale Inversion Problem](https://mlsanity.com/paper/2011.08954/) (2020)

## Built on by (most cited first, 2 of 2)

- [Stop the Sampler! Classifier-Based Adaptive Stopping for Sampling Kernels](https://mlsanity.com/paper/2606.16073/) (2026)
- [GRPO-QPS: Target-Preserving Reinforcement Learning for Quantum Posterior Sampling](https://mlsanity.com/paper/2609.14711/) (2026)

## Abstract (arXiv)

Sampling algorithms drive probabilistic machine learning, and recent years have seen an explosion in the diversity of tools for this task. However, the increasing sophistication of sampling algorithms is correlated with an increase in the tuning burden. There is now a greater need than ever to treat the tuning of samplers as a learning task in its own right. In a conceptual breakthrough, Wang et al (2025) formulated Metropolis-Hastings as a Markov decision process, opening up the possibility for adaptive tuning using Reinforcement Learning (RL). Their emphasis was on theoretical foundations; realising the practical benefit of Reinforcement Learning Metropolis-Hastings (RLMH) was left for subsequent work. The purpose of this paper is twofold: First, we observe the surprising result that natural choices of reward, such as the acceptance rate, or the expected squared jump distance, provide insufficient signal for training RLMH. Instead, we propose a novel reward based on the contrastive divergence, whose superior performance in the context of RLMH is demonstrated. Second, we explore the potential of RLMH and present adaptive gradient-based samplers that balance flexibility of the Markov transition kernel with learnability of the associated RL task. A comprehensive simulation study using the posteriordb benchmark supports the practical effectiveness of RLMH.