# A survey of Monte Carlo methods for noisy and costly densities with application to reinforcement learning and ABC

arXiv: https://arxiv.org/abs/2108.00490 · HTML: https://mlsanity.com/paper/2108.00490/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 101 references that are in the map.
- First posted on arXiv 1 Aug 2021.

## Builds on (most cited first, 30 of 33)

- [Bayesian Optimization for Likelihood-Free Inference of Simulator-Based Statistical Models](https://mlsanity.com/paper/1501.03291/) (2015)
- [Local Gaussian process approximation for large computer experiments](https://mlsanity.com/paper/1303.0383/) (2013)
- [On Markov chain Monte Carlo methods for tall data](https://mlsanity.com/paper/1505.02827/) (2015)
- [A survey on policy search algorithms for learning robot controllers in a handful of trials](https://mlsanity.com/paper/1807.02303/) (2018)
- [Marginal likelihood computation for model selection and hypothesis testing: an extensive review](https://mlsanity.com/paper/2005.08334/) (2020)
- [Probabilistic Integration: A Role in Statistical Computation?](https://mlsanity.com/paper/1512.00933/) (2015)
- [Variational Bayesian Monte Carlo with Noisy Likelihoods](https://mlsanity.com/paper/2006.08655/) (2020)
- [A Survey of Monte Carlo Methods for Parameter Estimation](https://mlsanity.com/paper/2107.11820/) (2021)
- [Convergence Guarantees for Adaptive Bayesian Quadrature Methods](https://mlsanity.com/paper/1905.10271/) (2019)
- [Speeding Up MCMC by Efficient Data Subsampling](https://mlsanity.com/paper/1404.4178/) (2014)
- [Parallel Gaussian process surrogate Bayesian inference with noisy likelihood evaluations](https://mlsanity.com/paper/1905.01252/) (2019)
- [GPS-ABC: Gaussian Process Surrogate Approximate Bayesian Computation](https://mlsanity.com/paper/1401.2838/) (2014)
- [Accelerating Asymptotically Exact MCMC for Computationally Intensive Models via Local Approximations](https://mlsanity.com/paper/1402.1694/) (2014)
- [Efficient acquisition rules for model-based approximate Bayesian computation](https://mlsanity.com/paper/1704.00520/) (2017)
- [Active emulation of computer codes with Gaussian processes -- Application to remote sensing](https://mlsanity.com/paper/1912.06552/) (2019)
- [Noisy Monte Carlo: Convergence of Markov chains with approximate transition kernels](https://mlsanity.com/paper/1403.5496/) (2014)
- [A Bayes-Sard Cubature Method](https://mlsanity.com/paper/1804.03016/) (2018)
- [A Joint introduction to Gaussian Processes and Relevance Vector Machines with Connections to Kalman filtering and other Kernel Smoothers](https://mlsanity.com/paper/2009.09217/) (2020)
- [Adaptive, delayed-acceptance MCMC for targets with expensive likelihoods](https://mlsanity.com/paper/1509.00172/) (2015)
- [Adaptive Gaussian process approximation for Bayesian inference with expensive likelihood functions](https://mlsanity.com/paper/1703.09930/) (2017)
- [Deep Importance Sampling based on Regression for Model Inversion and Emulation](https://mlsanity.com/paper/2010.10346/) (2020)
- [Ensemble Inference Methods for Models With Noisy and Expensive Likelihoods](https://mlsanity.com/paper/2104.03384/) (2021)
- [Accelerating ABC methods using Gaussian processes](https://mlsanity.com/paper/1401.1436/) (2014)
- [Adaptive quadrature schemes for Bayesian inference via active learning](https://mlsanity.com/paper/2006.00535/) (2020)
- [The Recycling Gibbs Sampler for Efficient Learning](https://mlsanity.com/paper/1611.07056/) (2016)
- [Adaptive Independent Sticky MCMC algorithms](https://mlsanity.com/paper/1308.3779/) (2013)
- [Query Efficient Posterior Estimation in Scientific Experiments via Bayesian Active Learning](https://mlsanity.com/paper/1702.01145/) (2017)
- [Speeding Up MCMC by Delayed Acceptance and Data Subsampling](https://mlsanity.com/paper/1507.06110/) (2015)
- [Approximate Bayesian inference from noisy likelihoods with Gaussian process emulated MCMC](https://mlsanity.com/paper/2104.03942/) (2021)
- [Importance sampling squared for Bayesian inference in latent variable models](https://mlsanity.com/paper/1309.3339/) (2013)

## Built on by (most cited first, 8 of 8)

- [ISFL: Federated Learning for Non-i.i.d. Data with Local Importance Sampling](https://mlsanity.com/paper/2210.02119/) (2022)
- [Optimality in Noisy Importance Sampling](https://mlsanity.com/paper/2201.02432/) (2022)
- [The Normalized Maximum Likelihood for Regular Non-Smooth Models: Measure-Theoretic Foundations and Geometric Sampling](https://mlsanity.com/paper/2605.24477/) (2026)
- [CAMEO: Curiosity Augmented Metropolis for Exploratory Optimal Policies](https://mlsanity.com/paper/2205.09433/) (2022)
- [Deep Metric Tensor Regularized Policy Gradient](https://mlsanity.com/paper/2305.11017/) (2023)
- [Binary classification based Monte Carlo simulation](https://mlsanity.com/paper/2307.16035/) (2023)
- [A unifying view of contrastive learning, importance sampling, and bridge sampling for energy-based models](https://mlsanity.com/paper/2604.08116/) (2026)
- [AME: A Multi-Type Contributor Attribution Framework in Generative AI Markets](https://mlsanity.com/paper/2606.16075/) (2026)

## Abstract (arXiv)

This survey gives an overview of Monte Carlo methodologies using surrogate models, for dealing with densities which are intractable, costly, and/or noisy. This type of problem can be found in numerous real-world scenarios, including stochastic optimization and reinforcement learning, where each evaluation of a density function may incur some computationally-expensive or even physical (real-world activity) cost, likely to give different results each time. The surrogate model does not incur this cost, but there are important trade-offs and considerations involved in the choice and design of such methodologies. We classify the different methodologies into three main classes and describe specific instances of algorithms under a unified notation. A modular scheme which encompasses the considered methods is also presented. A range of application scenarios is discussed, with special attention to the likelihood-free setting and reinforcement learning. Several numerical comparisons are also provided.