mlsanityData to 7 Oct 2026

Asymptotically Exact, Embarrassingly Parallel MCMC

arXiv 1311.4780 ↗First posted 19 Nov 2013stat.ML cs.DC cs.LGcited by 30

Not on the map. Why?

Abstract (arXiv)

Communication costs, resulting from synchronization requirements during learning, can greatly slow down many parallel machine learning algorithms. In this paper, we present a parallel Markov chain Monte Carlo (MCMC) algorithm in which subsets of data are processed independently, with very little communication. First, we arbitrarily partition data onto multiple machines. Then, on each machine, any classical MCMC method (e.g., Gibbs sampling) may be used to draw samples from a posterior distribution given the data subset. Finally, the samples from each machine are combined to form samples from the full posterior. This embarrassingly parallel algorithm allows each machine to act independently on a subset of the data (without communication) until the final combination stage. We prove that our algorithm generates asymptotically exact samples and empirically demonstrate its ability to parallelize burn-in and sampling in several models.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Nov 2013.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 5; 2023: 4; 2024: 7; 2025: 4; 2026: 4.

30 papers.

Sort:

By area:

  1. 2021QLSD: Quantised Langevin stochastic dynamics for Bayesian federated learningcited by 26
  2. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  3. 2021A Survey of Monte Carlo Methods for Parameter Estimationcited by 23
  4. 2021On Convergence of Federated Averaging Langevin Dynamicscited by 14
  5. 2021DG-LMC: A Turn-key and Scalable Synchronous Distributed MCMC Algorithm via Langevin Monte Carlo within Gibbscited by 11
  6. 2022Parallel MCMC Without Embarrassing Failurescited by 11
  7. 2023Calibrated One Round Federated Learning with Bayesian Inference in the Predictive Spacecited by 10
  8. 2024Parallelized Midpoint Randomization for Langevin Monte Carlocited by 9
  9. 2022SwISS: A Scalable Markov chain Monte Carlo Divide-and-Conquer Strategycited by 6
  10. 2025Capture Global Feature Statistics for One-Shot Federated Learningcited by 6
  11. 2024Enhancing Transfer Learning with Flexible Nonparametric Posterior Samplingcited by 5
  12. 2023Federated Variational Inference Methods for Structured Latent Variable Modelscited by 4
18 more
  1. 2023Machine Learning and the Future of Bayesian Computationcited by 4
  2. 2021MCMC-driven importance samplerscited by 3
  3. 2021Divide-and-Conquer Fusioncited by 3
  4. 2024Embarrassingly Parallel GFlowNetscited by 3
  5. 2022Federated Averaging Langevin Dynamics: Toward a unified theory and new algorithmscited by 2
  6. 2024One-Shot Federated Learning with Bayesian Pseudocoresetscited by 1
  7. 2024Data value estimation on private gradientscited by 1
  8. 2025Adaptive posterior distributions for uncertainty analysis of covariance matrices in Bayesian inversion problems for multioutput signalscited by 1
  9. 2025Federated Learning via Meta-Variational Dropoutcited by 1
  10. 2026Vertical Consensus Inference for High-Dimensional Random Partitioncited by 1
  11. 2022Distributed data analyticscited by 0
  12. 2022Robust One Round Federated Learning with Predictive Space Bayesian Inferencecited by 0
  13. 2024Distributed Markov Chain Monte Carlo Sampling based on the Alternating Direction Method of Multiplierscited by 0
  14. 2024Diffusion Generative Modelling for Divide-and-Conquer MCMCcited by 0
  15. 2025Measuring the right thing: justifying metrics in AI impact assessmentscited by 0
  16. 2026Differential Privacy of Gaussian Process Posterior Samplingcited by 0
  17. 2026Divide-and-Conquer: Towards Generalizable Amortized Bayesian Inference for the Drift Diffusion Modelcited by 0
  18. 2026Every pooling rule has its world: matching probability combination rules to situations and stakescited by 0