# Federated Learning with Uncertainty via Distilled Predictive Distributions

arXiv: https://arxiv.org/abs/2206.07562 · HTML: https://mlsanity.com/paper/2206.07562/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 41 references that are in the map.
- First posted on arXiv 15 Jun 2022.

## Builds on (most cited first, 29 of 29)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021)
- [Three Approaches for Personalization with Applications to Federated Learning](https://mlsanity.com/paper/2002.10619/) (2020)
- [What Are Bayesian Neural Network Posteriors Really Like?](https://mlsanity.com/paper/2104.14421/) (2021)
- [Advances in Variational Inference](https://mlsanity.com/paper/1711.05597/) (2017)
- [Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning](https://mlsanity.com/paper/1902.03932/) (2019)
- [FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning](https://mlsanity.com/paper/2009.01974/) (2020)
- [A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challenges](https://mlsanity.com/paper/2110.14051/) (2021)
- [Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithms](https://mlsanity.com/paper/2010.05273/) (2020)
- [Personalized Federated Learning with Gaussian Processes](https://mlsanity.com/paper/2106.15482/) (2021)
- [HyperGAN: A Generative Model for Diverse, Performant Neural Networks](https://mlsanity.com/paper/1901.11058/) (2019)
- [Bayesian Dark Knowledge](https://mlsanity.com/paper/1506.04416/) (2015)
- [A Bayesian Federated Learning Framework with Online Laplace Approximation](https://mlsanity.com/paper/2102.01936/) (2021)
- [Federated Generalized Bayesian Learning via Distributed Stein Variational Gradient Descent](https://mlsanity.com/paper/2009.06419/) (2020)
- [Adversarial Distillation of Bayesian Neural Network Posteriors](https://mlsanity.com/paper/1806.10317/) (2018)
- [Federated Stochastic Gradient Langevin Dynamics](https://mlsanity.com/paper/2004.11231/) (2020)
- [Generalized Bayesian Posterior Expectation Distillation for Deep Neural Networks](https://mlsanity.com/paper/2005.08110/) (2020)
- [Federated Active Learning (F-AL): an Efficient Annotation Strategy for Federated Learning](https://mlsanity.com/paper/2202.00195/) (2022)
- [An Expectation-Maximization Perspective on Federated Learning](https://mlsanity.com/paper/2111.10192/) (2021)
- [Bayesian Federated Learning over Wireless Networks](https://mlsanity.com/paper/2012.15486/) (2020)

## Built on by (most cited first, 12 of 12)

- [Federated Distillation: A Survey](https://mlsanity.com/paper/2404.08564/) (2024)
- [Privacy Preserving Federated Learning in Medical Imaging with Uncertainty Estimation](https://mlsanity.com/paper/2406.12815/) (2024)
- [Future-Proofing Medical Imaging with Privacy-Preserving Federated Learning and Uncertainty Quantification: A Review](https://mlsanity.com/paper/2409.16340/) (2024)
- [Federated Learning with Uncertainty and Personalization via Efficient Second-order Optimization](https://mlsanity.com/paper/2411.18385/) (2024)
- [Achieving Distributive Justice in Federated Learning via Uncertainty Quantification](https://mlsanity.com/paper/2504.15924/) (2025)
- [Information-Geometric Barycenters for Bayesian Federated Learning](https://mlsanity.com/paper/2412.11646/) (2024)
- [Federated Variational Inference for Bayesian Mixture Models](https://mlsanity.com/paper/2502.12684/) (2025)
- [Personalized Bayesian Federated Learning with Wasserstein Barycenter Aggregation](https://mlsanity.com/paper/2505.14161/) (2025)
- [Bayesian Federated Cause-of-Death Classification and Quantification Under Distribution Shift](https://mlsanity.com/paper/2505.02257/) (2025)
- [Cost-Free Personalization via Information-Geometric Projection in Bayesian Federated Learning](https://mlsanity.com/paper/2509.10132/) (2025)
- [Uncertainty-Aware Federated Learning for Infant Movement Analysis](https://mlsanity.com/paper/2609.31463/) (2026)
- [FIDAL: Diversity-Aware Federated Active Learning Under Real-World Distribution Shifts](https://mlsanity.com/paper/2609.31637/) (2026)

## Abstract (arXiv)

Most existing federated learning methods are unable to estimate model/predictive uncertainty since the client models are trained using the standard loss function minimization approach which ignores such uncertainties. In many situations, however, especially in limited data settings, it is beneficial to take into account the uncertainty in the model parameters at each client as it leads to more accurate predictions and also because reliable estimates of uncertainty can be used for tasks, such as out-of-distribution (OOD) detection, and sequential decision-making tasks, such as active learning. We present a framework for federated learning with uncertainty where, in each round, each client infers the posterior distribution over its parameters as well as the posterior predictive distribution (PPD), distills the PPD into a single deep neural network, and sends this network to the server. Unlike some of the recent Bayesian approaches to federated learning, our approach does not require sending the whole posterior distribution of the parameters from each client to the server but only the PPD in the distilled form as a deep neural network. In addition, when making predictions at test time, it does not require computationally expensive Monte-Carlo averaging over the posterior distribution because our approach always maintains the PPD in the form of a single deep neural network. Moreover, our approach does not make any restrictive assumptions, such as the form of the clients' posterior distributions, or of their PPDs. We evaluate our approach on classification in federated setting, as well as active learning and OOD detection in federated settings, on which our approach outperforms various existing federated learning baselines.