# Robust Distributed Bayesian Learning with Stragglers via Consensus Monte Carlo

arXiv: https://arxiv.org/abs/2112.09794 · HTML: https://mlsanity.com/paper/2112.09794/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 23 references that are in the map.
- First posted on arXiv 17 Dec 2021.

## Builds on (most cited first, 14 of 14)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [Speeding Up Distributed Machine Learning Using Codes](https://mlsanity.com/paper/1512.02673/) (2015)
- [Federated Generalized Bayesian Learning via Distributed Stein Variational Gradient Descent](https://mlsanity.com/paper/2009.06419/) (2020)
- [Partitioned Variational Inference: A unified framework encompassing federated and continual learning](https://mlsanity.com/paper/1811.11206/) (2018)
- [Patterns of Scalable Bayesian Inference](https://mlsanity.com/paper/1602.05221/) (2016)
- [A Brief Introduction to Machine Learning for Engineers](https://mlsanity.com/paper/1709.02840/) (2017)
- [A Berkeley View of Systems Challenges for AI](https://mlsanity.com/paper/1712.05855/) (2017)
- [Gradient Coding with Clustering and Multi-message Communication](https://mlsanity.com/paper/1903.01974/) (2019)
- [LAGC: Lazily Aggregated Gradient Coding for Straggler-Tolerant and Communication-Efficient Distributed Learning](https://mlsanity.com/paper/1905.09148/) (2019)
- [Wireless Federated Langevin Monte Carlo: Repurposing Channel Noise for Bayesian Sampling and Privacy](https://mlsanity.com/paper/2108.07644/) (2021)
- [Variational consensus Monte Carlo](https://mlsanity.com/paper/1506.03074/) (2015)
- [Channel-Driven Monte Carlo Sampling for Bayesian Distributed Learning in Wireless Data Centers](https://mlsanity.com/paper/2103.01351/) (2021)

## Abstract (arXiv)

This paper studies distributed Bayesian learning in a setting encompassing a central server and multiple workers by focusing on the problem of mitigating the impact of stragglers. The standard one-shot, or embarrassingly parallel, Bayesian learning protocol known as consensus Monte Carlo (CMC) is generalized by proposing two straggler-resilient solutions based on grouping and coding. Two main challenges in designing straggler-resilient algorithms for CMC are the need to estimate the statistics of the workers' outputs across multiple shots, and the joint non-linear post-processing of the outputs of the workers carried out at the server. This is in stark contrast to other distributed settings like gradient coding, which only require the per-shot sum of the workers' outputs. The proposed methods, referred to as Group-based CMC (G-CMC) and Coded CMC (C-CMC), leverage redundant computing at the workers in order to enable the estimation of global posterior samples at the server based on partial outputs from the workers. Simulation results show that C-CMC may outperform G-CMC for a small number of workers, while G-CMC is generally preferable for a larger number of workers.