# DG-LMC: A Turn-key and Scalable Synchronous Distributed MCMC Algorithm via Langevin Monte Carlo within Gibbs

arXiv: https://arxiv.org/abs/2106.06300 · HTML: https://mlsanity.com/paper/2106.06300/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 49 references that are in the map.
- First posted on arXiv 11 Jun 2021.

## Builds on (most cited first, 22 of 22)

- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Non-asymptotic convergence analysis for the Unadjusted Langevin Algorithm](https://mlsanity.com/paper/1507.05021/) (2015)
- [A Survey on Distributed Machine Learning](https://mlsanity.com/paper/1912.09789/) (2019)
- [High-dimensional Bayesian inference via the Unadjusted Langevin Algorithm](https://mlsanity.com/paper/1605.01559/) (2016)
- [User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradient](https://mlsanity.com/paper/1710.00095/) (2017)
- [The promises and pitfalls of Stochastic Gradient Langevin Dynamics](https://mlsanity.com/paper/1811.10072/) (2018)
- [On Markov chain Monte Carlo methods for tall data](https://mlsanity.com/paper/1505.02827/) (2015)
- [Split-and-augmented Gibbs sampler - Application to large-scale inference problems](https://mlsanity.com/paper/1804.05809/) (2018)
- [Asymptotically Exact, Embarrassingly Parallel MCMC](https://mlsanity.com/paper/1311.4780/) (2013)
- [Partitioned Variational Inference: A unified framework encompassing federated and continual learning](https://mlsanity.com/paper/1811.11206/) (2018)
- [Efficient MCMC Sampling with Dimension-Free Convergence Rate using ADMM-type Splitting](https://mlsanity.com/paper/1905.11937/) (2019)
- [Distributed Bayesian Learning with Stochastic Natural-gradient Expectation Propagation and the Posterior Server](https://mlsanity.com/paper/1512.09327/) (2015)
- [Expectation propagation as a way of life: A framework for Bayesian inference on partitioned data](https://mlsanity.com/paper/1412.4869/) (2014)
- [Merging MCMC Subposteriors through Gaussian-Process Approximations](https://mlsanity.com/paper/1605.08576/) (2016)
- [Parallelizing MCMC via Weierstrass Sampler](https://mlsanity.com/paper/1312.4605/) (2013)
- [Embarrassingly parallel MCMC using deep invertible transformations](https://mlsanity.com/paper/1903.04556/) (2019)
- [Parallelizing MCMC with Random Partition Trees](https://mlsanity.com/paper/1506.03164/) (2015)
- [Stochastic Gradient MCMC with Stale Gradients](https://mlsanity.com/paper/1610.06664/) (2016)
- [Asymptotically exact data augmentation: models, properties and algorithms](https://mlsanity.com/paper/1902.05754/) (2019)
- [Variational consensus Monte Carlo](https://mlsanity.com/paper/1506.03074/) (2015)
- [Monte Carlo Fusion](https://mlsanity.com/paper/1901.00139/) (2019)

## Built on by (most cited first, 11 of 11)

- [FedPop: A Bayesian Approach for Personalised Federated Learning](https://mlsanity.com/paper/2206.03611/) (2022)
- [Regularization by denoising: Bayesian model and Langevin-within-split Gibbs sampling](https://mlsanity.com/paper/2402.12292/) (2024)
- [FedHB: Hierarchical Bayesian Federated Learning](https://mlsanity.com/paper/2305.04979/) (2023)
- [Adaptive Compression in Federated Learning via Side Information](https://mlsanity.com/paper/2306.12625/) (2023)
- [Federated Variational Inference Methods for Structured Latent Variable Models](https://mlsanity.com/paper/2302.03314/) (2023)
- [Machine Learning and the Future of Bayesian Computation](https://mlsanity.com/paper/2304.11251/) (2023)
- [Asynchronous and Distributed Data Augmentation for Massive Data Settings](https://mlsanity.com/paper/2109.08969/) (2021)
- [Membership Inference Attacks via Adversarial Examples](https://mlsanity.com/paper/2207.13572/) (2022)
- [Federated Averaging Langevin Dynamics: Toward a unified theory and new algorithms](https://mlsanity.com/paper/2211.00100/) (2022)
- [Trustworthy Personalized Bayesian Federated Learning via Posterior Fine-Tune](https://mlsanity.com/paper/2402.16911/) (2024)
- [Bregman geometry-aware split Gibbs sampling for Bayesian Poisson inverse problems](https://mlsanity.com/paper/2511.12257/) (2025)

## Abstract (arXiv)

Performing reliable Bayesian inference on a big data scale is becoming a keystone in the modern era of machine learning. A workhorse class of methods to achieve this task are Markov chain Monte Carlo (MCMC) algorithms and their design to handle distributed datasets has been the subject of many works. However, existing methods are not completely either reliable or computationally efficient. In this paper, we propose to fill this gap in the case where the dataset is partitioned and stored on computing nodes within a cluster under a master/slaves architecture. We derive a user-friendly centralised distributed MCMC algorithm with provable scaling in high-dimensional settings. We illustrate the relevance of the proposed methodology on both synthetic and real data experiments.