# Distributed Variational Inference for Online Supervised Learning

arXiv: https://arxiv.org/abs/2309.02606 · HTML: https://mlsanity.com/paper/2309.02606/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 57 references that are in the map.
- First posted on arXiv 5 Sep 2023.

## Builds on (most cited first, 23 of 23)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent](https://mlsanity.com/paper/1705.09056/) (2017)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [Stochastic Gradient Descent as Approximate Bayesian Inference](https://mlsanity.com/paper/1704.04289/) (2017)
- [Advances in Variational Inference](https://mlsanity.com/paper/1711.05597/) (2017)
- [Decentralized Federated Averaging](https://mlsanity.com/paper/2104.11375/) (2021)
- [Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithms](https://mlsanity.com/paper/2010.05273/) (2020)
- [Personalized Federated Learning via Variational Bayesian Inference](https://mlsanity.com/paper/2206.07977/) (2022)
- [Learning Robot Skills with Temporal Variational Inference](https://mlsanity.com/paper/2006.16232/) (2020)
- [Partitioned Variational Inference: A unified framework encompassing federated and continual learning](https://mlsanity.com/paper/1811.11206/) (2018)
- [Provable convergence guarantees for black-box variational inference](https://mlsanity.com/paper/2306.03638/) (2023)
- [Fast Convergence Rates for Distributed Non-Bayesian Learning](https://mlsanity.com/paper/1508.05161/) (2015)
- [Gaussian variational approximation with a factor covariance structure](https://mlsanity.com/paper/1701.03208/) (2017)
- [Semi-analytical approximations to statistical moments of sigmoid and softmax mappings of normal variables](https://mlsanity.com/paper/1703.00091/) (2017)
- [DiNNO: Distributed Neural Network Optimization for Multi-Robot Collaborative Learning](https://mlsanity.com/paper/2109.08665/) (2021)
- [FedGiA: An Efficient Hybrid Algorithm for Federated Learning](https://mlsanity.com/paper/2205.01438/) (2022)
- [Exactly Sparse Gaussian Variational Inference with Application to Derivative-Free Batch Nonlinear State Estimation](https://mlsanity.com/paper/1911.08333/) (2019)
- [Nonlinear Kalman Filtering with Divergence Minimization](https://mlsanity.com/paper/1705.00722/) (2017)
- [Autonomous Navigation in Unknown Environments with Sparse Bayesian Kernel-based Occupancy Mapping](https://mlsanity.com/paper/2009.07207/) (2020)
- [Distributed Variational Bayesian Algorithms Over Sensor Networks](https://mlsanity.com/paper/2011.13600/) (2020)
- [Consensus Driven by the Geometric Mean](https://mlsanity.com/paper/1511.02604/) (2015)

## Built on by (most cited first, 1 of 1)

- [Distributed Bayesian Estimation in Sensor Networks: Consensus on Marginal Densities](https://mlsanity.com/paper/2312.01227/) (2023)

## Abstract (arXiv)

Developing efficient solutions for inference problems in intelligent sensor networks is crucial for the next generation of location, tracking, and mapping services. This paper develops a scalable distributed probabilistic inference algorithm that applies to continuous variables, intractable posteriors and large-scale real-time data in sensor networks. In a centralized setting, variational inference is a fundamental technique for performing approximate Bayesian estimation, in which an intractable posterior density is approximated with a parametric density. Our key contribution lies in the derivation of a separable lower bound on the centralized estimation objective, which enables distributed variational inference with one-hop communication in a sensor network. Our distributed evidence lower bound (DELBO) consists of a weighted sum of observation likelihood and divergence to prior densities, and its gap to the measurement evidence is due to consensus and modeling errors. To solve binary classification and regression problems while handling streaming data, we design an online distributed algorithm that maximizes DELBO, and specialize it to Gaussian variational densities with non-linear likelihoods. The resulting distributed Gaussian variational inference (DGVI) efficiently inverts a 1-rank correction to the covariance matrix. Finally, we derive a diagonalized version for online distributed inference in high-dimensional models, and apply it to multi-robot probabilistic mapping using indoor LiDAR data.