# Parallel MCMC Without Embarrassing Failures

arXiv: https://arxiv.org/abs/2202.11154 · HTML: https://mlsanity.com/paper/2202.11154/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 38 references that are in the map.
- First posted on arXiv 22 Feb 2022.

## Builds on (most cited first, 20 of 20)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [High Dimensional Bayesian Optimisation and Bandits via Additive Models](https://mlsanity.com/paper/1503.01673/) (2015)
- [Gaussian Process Kernels for Pattern Discovery and Extrapolation](https://mlsanity.com/paper/1302.4245/) (2013)
- [Exact Gaussian Processes on a Million Data Points](https://mlsanity.com/paper/1903.08114/) (2019)
- [Sampling for Inference in Probabilistic Models with Fast Bayesian Quadrature](https://mlsanity.com/paper/1411.0439/) (2014)
- [Asymptotically Exact, Embarrassingly Parallel MCMC](https://mlsanity.com/paper/1311.4780/) (2013)
- [Partitioned Variational Inference: A unified framework encompassing federated and continual learning](https://mlsanity.com/paper/1811.11206/) (2018)
- [QLSD: Quantised Langevin stochastic dynamics for Bayesian federated learning](https://mlsanity.com/paper/2106.00797/) (2021)
- [Patterns of Scalable Bayesian Inference](https://mlsanity.com/paper/1602.05221/) (2016)
- [Variational Bayesian Monte Carlo with Noisy Likelihoods](https://mlsanity.com/paper/2006.08655/) (2020)
- [Variational Bayesian Monte Carlo](https://mlsanity.com/paper/1810.05558/) (2018)
- [Parallel Gaussian process surrogate Bayesian inference with noisy likelihood evaluations](https://mlsanity.com/paper/1905.01252/) (2019)
- [Federated Stochastic Gradient Langevin Dynamics](https://mlsanity.com/paper/2004.11231/) (2020)
- [Merging MCMC Subposteriors through Gaussian-Process Approximations](https://mlsanity.com/paper/1605.08576/) (2016)
- [When are Iterative Gaussian Processes Reliably Accurate?](https://mlsanity.com/paper/2112.15246/) (2021)
- [Embarrassingly parallel MCMC using deep invertible transformations](https://mlsanity.com/paper/1903.04556/) (2019)
- [Parallelizing MCMC with Random Partition Trees](https://mlsanity.com/paper/1506.03164/) (2015)
- [Adaptive Gaussian process approximation for Bayesian inference with expensive likelihood functions](https://mlsanity.com/paper/1703.09930/) (2017)

## Built on by (most cited first, 11 of 11)

- [Parallel Simulation for Log-concave Sampling and Score-based Diffusion Models](https://mlsanity.com/paper/2412.07435/) (2024)
- [Enhancing Transfer Learning with Flexible Nonparametric Posterior Sampling](https://mlsanity.com/paper/2403.07282/) (2024)
- [Machine Learning and the Future of Bayesian Computation](https://mlsanity.com/paper/2304.11251/) (2023)
- [Fast post-process Bayesian inference with Variational Sparse Bayesian Quadrature](https://mlsanity.com/paper/2303.05263/) (2023)
- [Embarrassingly Parallel GFlowNets](https://mlsanity.com/paper/2406.03288/) (2024)
- [Federated Averaging Langevin Dynamics: Toward a unified theory and new algorithms](https://mlsanity.com/paper/2211.00100/) (2022)
- [Data Subsampling for Bayesian Neural Networks](https://mlsanity.com/paper/2210.09141/) (2022)
- [Comparison of parallel SMC and MCMC for Bayesian deep learning](https://mlsanity.com/paper/2402.06173/) (2024)
- [One-Shot Federated Learning with Bayesian Pseudocoresets](https://mlsanity.com/paper/2406.02177/) (2024)
- [Normalizing Flow Regression for Bayesian Inference with Offline Likelihood Evaluations](https://mlsanity.com/paper/2504.11554/) (2025)
- [Measuring the right thing: justifying metrics in AI impact assessments](https://mlsanity.com/paper/2504.05007/) (2025)

## Abstract (arXiv)

Embarrassingly parallel Markov Chain Monte Carlo (MCMC) exploits parallel computing to scale Bayesian inference to large datasets by using a two-step approach. First, MCMC is run in parallel on (sub)posteriors defined on data partitions. Then, a server combines local results. While efficient, this framework is very sensitive to the quality of subposterior sampling. Common sampling problems such as missing modes or misrepresentation of low-density regions are amplified -- instead of being corrected -- in the combination phase, leading to catastrophic failures. In this work, we propose a novel combination strategy to mitigate this issue. Our strategy, Parallel Active Inference (PAI), leverages Gaussian Process (GP) surrogate modeling and active learning. After fitting GPs to subposteriors, PAI (i) shares information between GP surrogates to cover missing modes; and (ii) uses active sampling to individually refine subposterior approximations. We validate PAI in challenging benchmarks, including heavy-tailed and multi-modal posteriors and a real-world application to computational neuroscience. Empirical results show that PAI succeeds where previous methods catastrophically fail, with a small communication overhead.