# Sandwiching the marginal likelihood using bidirectional Monte Carlo

arXiv: https://arxiv.org/abs/1511.02543 · HTML: https://mlsanity.com/paper/1511.02543/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Nov 2015.

## Built on by (most cited first, 20 of 20)

- [Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo](https://mlsanity.com/paper/2404.17546/) (2024)
- [Bayesian Model Selection, the Marginal Likelihood, and Generalization](https://mlsanity.com/paper/2202.11678/) (2022)
- [MCMC Variational Inference via Uncorrected Hamiltonian Annealing](https://mlsanity.com/paper/2107.04150/) (2021)
- [Monte Carlo Variational Auto-Encoders](https://mlsanity.com/paper/2106.15921/) (2021)
- [Differentiable Annealed Importance Sampling and the Perils of Gradient Noise](https://mlsanity.com/paper/2107.10211/) (2021)
- [Surrogate Likelihoods for Variational Annealed Importance Sampling](https://mlsanity.com/paper/2112.12194/) (2021)
- [q-Paths: Generalizing the Geometric Annealing Path using Power Means](https://mlsanity.com/paper/2107.00745/) (2021)
- [Adaptive Annealed Importance Sampling with Constant Rate Progress](https://mlsanity.com/paper/2306.15283/) (2023)
- [Bounding Evidence and Estimating Log-Likelihood in VAE](https://mlsanity.com/paper/2206.09453/) (2022)
- [Estimating the Rate-Distortion Function by Wasserstein Gradient Descent](https://mlsanity.com/paper/2310.18908/) (2023)
- [Variational Inference with Locally Enhanced Bounds for Hierarchical Models](https://mlsanity.com/paper/2203.04432/) (2022)
- [Variational Learning of Gaussian Process Latent Variable Models through Stochastic Gradient Annealed Importance Sampling](https://mlsanity.com/paper/2408.06710/) (2024)
- [An Easy to Interpret Diagnostic for Approximate Inference: Symmetric Divergence Over Simulations](https://mlsanity.com/paper/2103.01030/) (2021)
- [Estimators of Entropy and Information via Inference in Probabilistic Models](https://mlsanity.com/paper/2202.12363/) (2022)
- [BAM: Bayes with Adaptive Memory](https://mlsanity.com/paper/2202.02405/) (2022)
- [Guaranteed Bounds for Posterior Inference in Universal Probabilistic Programming](https://mlsanity.com/paper/2204.02948/) (2022)
- [Optimization of Annealed Importance Sampling Hyperparameters](https://mlsanity.com/paper/2209.13226/) (2022)
- [Multimodal Continuous Reasoning via Asymmetric Mutual Variational Learning](https://mlsanity.com/paper/2607.00461/) (2026)
- [Intriguing Properties of Modern GANs](https://mlsanity.com/paper/2402.14098/) (2024)
- [Tight Bounds for Jensen's Gap with Applications to Variational Inference](https://mlsanity.com/paper/2502.03988/) (2025)

## Abstract (arXiv)

Computing the marginal likelihood (ML) of a model requires marginalizing out all of the parameters and latent variables, a difficult high-dimensional summation or integration problem. To make matters worse, it is often hard to measure the accuracy of one's ML estimates. We present bidirectional Monte Carlo, a technique for obtaining accurate log-ML estimates on data simulated from a model. This method obtains stochastic lower bounds on the log-ML using annealed importance sampling or sequential Monte Carlo, and obtains stochastic upper bounds by running these same algorithms in reverse starting from an exact posterior sample. The true value can be sandwiched between these two stochastic bounds with high probability. Using the ground truth log-ML estimates obtained from our method, we quantitatively evaluate a wide variety of existing ML estimators on several latent variable models: clustering, a low rank approximation, and a binary attributes model. These experiments yield insights into how to accurately estimate marginal likelihoods.