# Distributed Bayesian Learning with Stochastic Natural-gradient Expectation Propagation and the Posterior Server

arXiv: https://arxiv.org/abs/1512.09327 · HTML: https://mlsanity.com/paper/1512.09327/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2015.

## Built on by (most cited first, 17 of 17)

- [A Field Guide to Federated Optimization](https://mlsanity.com/paper/2107.06917/) (2021)
- [Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling](https://mlsanity.com/paper/2311.08718/) (2023)
- [Time-independent Generalization Bounds for SGLD in Non-convex Settings](https://mlsanity.com/paper/2111.12876/) (2021)
- [QLSD: Quantised Langevin stochastic dynamics for Bayesian federated learning](https://mlsanity.com/paper/2106.00797/) (2021)
- [Partitioned Variational Inference: A Framework for Probabilistic Federated Learning](https://mlsanity.com/paper/2202.12275/) (2022)
- [DG-LMC: A Turn-key and Scalable Synchronous Distributed MCMC Algorithm via Langevin Monte Carlo within Gibbs](https://mlsanity.com/paper/2106.06300/) (2021)
- [Task Agnostic and Post-hoc Unseen Distribution Detection](https://mlsanity.com/paper/2207.13083/) (2022)
- [A Review of Bayesian Uncertainty Quantification in Deep Probabilistic Image Segmentation](https://mlsanity.com/paper/2411.16370/) (2024)
- [Robust uncertainty estimates with out-of-distribution pseudo-inputs training](https://mlsanity.com/paper/2201.05890/) (2022)
- [Can Multiple Responses from an LLM Reveal the Sources of Its Uncertainty?](https://mlsanity.com/paper/2509.04464/) (2025)
- [Optimising Distributions with Natural Gradient Surrogates](https://mlsanity.com/paper/2310.11837/) (2023)
- [One-Shot Federated Learning with Bayesian Pseudocoresets](https://mlsanity.com/paper/2406.02177/) (2024)
- [Federated Generalised Variational Inference: A Robust Probabilistic Federated Learning Framework](https://mlsanity.com/paper/2502.00846/) (2025)
- [Distributed data analytics](https://mlsanity.com/paper/2203.14088/) (2022)
- [Variational Inference on the Final-Layer Output of Neural Networks](https://mlsanity.com/paper/2302.02420/) (2023)
- [Fearless Stochasticity in Expectation Propagation](https://mlsanity.com/paper/2406.01801/) (2024)
- [Probabilistic Machine Learning for Uncertainty-Aware Diagnosis of Industrial Systems](https://mlsanity.com/paper/2509.18810/) (2025)

## Abstract (arXiv)

This paper makes two contributions to Bayesian machine learning algorithms. Firstly, we propose stochastic natural gradient expectation propagation (SNEP), a novel alternative to expectation propagation (EP), a popular variational inference algorithm. SNEP is a black box variational algorithm, in that it does not require any simplifying assumptions on the distribution of interest, beyond the existence of some Monte Carlo sampler for estimating the moments of the EP tilted distributions. Further, as opposed to EP which has no guarantee of convergence, SNEP can be shown to be convergent, even when using Monte Carlo moment estimates. Secondly, we propose a novel architecture for distributed Bayesian learning which we call the posterior server. The posterior server allows scalable and robust Bayesian learning in cases where a data set is stored in a distributed manner across a cluster, with each compute node containing a disjoint subset of data. An independent Monte Carlo sampler is run on each compute node, with direct access only to the local data subset, but which targets an approximation to the global posterior distribution given all data across the whole cluster. This is achieved by using a distributed asynchronous implementation of SNEP to pass messages across the cluster. We demonstrate SNEP and the posterior server on distributed Bayesian learning of logistic regression and neural networks. Keywords: Distributed Learning, Large Scale Learning, Deep Learning, Bayesian Learn- ing, Variational Inference, Expectation Propagation, Stochastic Approximation, Natural Gradient, Markov chain Monte Carlo, Parameter Server, Posterior Server.