# Fearless Stochasticity in Expectation Propagation

arXiv: https://arxiv.org/abs/2406.01801 · HTML: https://mlsanity.com/paper/2406.01801/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 54 references that are in the map.
- First posted on arXiv 3 Jun 2024.

## Builds on (most cited first, 15 of 15)

- [Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks](https://mlsanity.com/paper/1502.05336/) (2015)
- [Efficient and Modular Implicit Differentiation](https://mlsanity.com/paper/2105.15183/) (2021)
- [Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyro](https://mlsanity.com/paper/1912.11554/) (2019)
- [Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models](https://mlsanity.com/paper/1703.04265/) (2017)
- [Deep Gaussian Processes for Regression using Approximate Expectation Propagation](https://mlsanity.com/paper/1602.04133/) (2016)
- [GLASSES: Relieving The Myopia Of Bayesian Optimisation](https://mlsanity.com/paper/1510.06299/) (2015)
- [Partitioned Variational Inference: A unified framework encompassing federated and continual learning](https://mlsanity.com/paper/1811.11206/) (2018)
- [Stochastic Expectation Propagation](https://mlsanity.com/paper/1506.04132/) (2015)
- [Distributed Bayesian Learning with Stochastic Natural-gradient Expectation Propagation and the Posterior Server](https://mlsanity.com/paper/1512.09327/) (2015)
- [Expectation propagation as a way of life: A framework for Bayesian inference on partitioned data](https://mlsanity.com/paper/1412.4869/) (2014)
- [Bayes-Newton Methods for Approximate Bayesian Inference with PSD Guarantees](https://mlsanity.com/paper/2111.01721/) (2021)
- [Scalable Gaussian Process Classification via Expectation Propagation](https://mlsanity.com/paper/1507.04513/) (2015)
- [Kernel-Based Just-In-Time Learning for Passing Expectation Propagation Messages](https://mlsanity.com/paper/1503.02551/) (2015)
- [Estimating the normal-inverse-Wishart distribution](https://mlsanity.com/paper/2405.16088/) (2024)
- [Optimising Distributions with Natural Gradient Surrogates](https://mlsanity.com/paper/2310.11837/) (2023)

## Abstract (arXiv)

Expectation propagation (EP) is a family of algorithms for performing approximate inference in probabilistic models. The updates of EP involve the evaluation of moments -- expectations of certain functions -- which can be estimated from Monte Carlo (MC) samples. However, the updates are not robust to MC noise when performed naively, and various prior works have attempted to address this issue in different ways. In this work, we provide a novel perspective on the moment-matching updates of EP; namely, that they perform natural-gradient-based optimisation of a variational objective. We use this insight to motivate two new EP variants, with updates that are particularly well-suited to MC estimation. They remain stable and are most sample-efficient when estimated with just a single sample. These new variants combine the benefits of their predecessors and address key weaknesses. In particular, they are easier to tune, offer an improved speed-accuracy trade-off, and do not rely on the use of debiasing estimators. We demonstrate their efficacy on a variety of probabilistic inference tasks.