# Delayed rejection Hamiltonian Monte Carlo for sampling multiscale distributions

arXiv: https://arxiv.org/abs/2110.00610 · HTML: https://mlsanity.com/paper/2110.00610/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 49 references that are in the map.
- First posted on arXiv 1 Oct 2021.

## Builds on (most cited first, 9 of 9)

- [A Conceptual Introduction to Hamiltonian Monte Carlo](https://mlsanity.com/paper/1701.02434/) (2017)
- [Hamiltonian Monte Carlo for Hierarchical Models](https://mlsanity.com/paper/1312.0906/) (2013)
- [Generalizing Hamiltonian Monte Carlo with Neural Networks](https://mlsanity.com/paper/1711.09268/) (2017)
- [The Barker proposal: combining robustness and efficiency in gradient-based MCMC](https://mlsanity.com/paper/1908.11812/) (2019)
- [Ensemble preconditioning for Markov chain Monte Carlo simulation](https://mlsanity.com/paper/1607.03954/) (2016)
- [Hamiltonian Monte Carlo Without Detailed Balance](https://mlsanity.com/paper/1409.5191/) (2014)
- [A general perspective on the Metropolis-Hastings kernel](https://mlsanity.com/paper/2012.14881/) (2020)
- [A Discrete Bouncy Particle Sampler](https://mlsanity.com/paper/1707.05200/) (2017)
- [Implicit Hamiltonian Monte Carlo for Sampling Multiscale Distributions](https://mlsanity.com/paper/1911.05754/) (2019)

## Built on by (most cited first, 7 of 7)

- [Amortized Bayesian Multilevel Models](https://mlsanity.com/paper/2408.13230/) (2024)
- [GIST: Gibbs self-tuning for locally adaptive Hamiltonian Monte Carlo](https://mlsanity.com/paper/2404.15253/) (2024)
- [AutoStep: Locally adaptive involutive MCMC](https://mlsanity.com/paper/2410.18929/) (2024)
- [The Within-Orbit Adaptive Leapfrog No-U-Turn Sampler](https://mlsanity.com/paper/2506.18746/) (2025)
- [ATLAS: Adapting Trajectory Lengths and Step-Size for Hamiltonian Monte Carlo](https://mlsanity.com/paper/2410.21587/) (2024)
- [Efficiently Vectorized MCMC on Modern Accelerators](https://mlsanity.com/paper/2503.17405/) (2025)
- [Optimal Slice-Adaptive Tuning of Hybrid Slice Sampling](https://mlsanity.com/paper/2609.08172/) (2026)

## Abstract (arXiv)

The efficiency of Hamiltonian Monte Carlo (HMC) can suffer when sampling a distribution with a wide range of length scales, because the small step sizes needed for stability in high-curvature regions are inefficient elsewhere. To address this we present a delayed rejection variant: if an initial HMC trajectory is rejected, we make one or more subsequent proposals each using a step size geometrically smaller than the last. We extend the standard delayed rejection framework by allowing the probability of a retry to depend on the probability of accepting the previous proposal. We test the scheme in several sampling tasks, including multiscale model distributions such as Neal's funnel, and statistical applications. Delayed rejection enables up to five-fold performance gains over optimally-tuned HMC, as measured by effective sample size per gradient evaluation. Even for simpler distributions, delayed rejection provides increased robustness to step size misspecification. Along the way, we provide an accessible but rigorous review of detailed balance for HMC.