# A Bayesian Federated Learning Framework with Online Laplace Approximation

arXiv: https://arxiv.org/abs/2102.01936 · HTML: https://mlsanity.com/paper/2102.01936/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 48 references that are in the map.
- First posted on arXiv 3 Feb 2021.

## Builds on (most cited first, 27 of 27)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [Local SGD Converges Fast and Communicates Little](https://mlsanity.com/paper/1805.09767/) (2018)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Bayesian Nonparametric Federated Learning of Neural Networks](https://mlsanity.com/paper/1905.12022/) (2019)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)
- [Overcoming Catastrophic Forgetting by Incremental Moment Matching](https://mlsanity.com/paper/1703.08475/) (2017)
- [Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1805.07810/) (2018)
- [Practical Secure Aggregation for Federated Learning on User-Held Data](https://mlsanity.com/paper/1611.04482/) (2016)
- [FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning](https://mlsanity.com/paper/2009.01974/) (2020)
- [Revisiting Natural Gradient for Deep Networks](https://mlsanity.com/paper/1301.3584/) (2013)
- [Communication Efficient Distributed Optimization using an Approximate Newton-type Method](https://mlsanity.com/paper/1312.7853/) (2013)
- [Overcoming Forgetting in Federated Learning on Non-IID Data](https://mlsanity.com/paper/1910.07796/) (2019)
- [On the convergence properties of a K-step averaging stochastic gradient descent algorithm for nonconvex optimization](https://mlsanity.com/paper/1708.01012/) (2017)
- [Deep learning with Elastic Averaging SGD](https://mlsanity.com/paper/1412.6651/) (2014)
- [Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithms](https://mlsanity.com/paper/2010.05273/) (2020)
- [Variational Federated Multi-Task Learning](https://mlsanity.com/paper/1906.06268/) (2019)
- [AIDE: Fast and Communication Efficient Distributed Optimization](https://mlsanity.com/paper/1608.06879/) (2016)
- [Convolution-Weight-Distribution Assumption: Rethinking the Criteria of Channel Pruning](https://mlsanity.com/paper/2004.11627/) (2020)

## Built on by (most cited first, 30 of 41)

- [Personalized Federated Learning via Variational Bayesian Inference](https://mlsanity.com/paper/2206.07977/) (2022)
- [Bayesian Federated Learning: A Survey](https://mlsanity.com/paper/2304.13267/) (2023)
- [FedLPA: One-shot Federated Learning with Layer-Wise Posterior Aggregation](https://mlsanity.com/paper/2310.00339/) (2023)
- [Federated Learning with Uncertainty via Distilled Predictive Distributions](https://mlsanity.com/paper/2206.07562/) (2022)
- [Modeling Inter-Intra Heterogeneity for Graph Federated Learning](https://mlsanity.com/paper/2412.11402/) (2024)
- [FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusion](https://mlsanity.com/paper/2410.20380/) (2024)
- [Privacy Preserving Federated Learning in Medical Imaging with Uncertainty Estimation](https://mlsanity.com/paper/2406.12815/) (2024)
- [PRIOR: Personalized Prior for Reactivating the Information Overlooked in Federated Learning](https://mlsanity.com/paper/2310.09183/) (2023)
- [Federated Bayesian Neural Regression: A Scalable Global Federated Gaussian Process](https://mlsanity.com/paper/2206.06357/) (2022)
- [How to Combine Variational Bayesian Networks in Federated Learning](https://mlsanity.com/paper/2206.10897/) (2022)
- [Bayesian Federated Inference for estimating Statistical Models based on Non-shared Multicenter Data sets](https://mlsanity.com/paper/2302.07677/) (2023)
- [Colorectal Polyp Segmentation in the Deep Learning Era: A Comprehensive Survey](https://mlsanity.com/paper/2401.11734/) (2024)
- [FedSI: Federated Subnetwork Inference for Efficient Uncertainty Quantification](https://mlsanity.com/paper/2404.15657/) (2024)
- [Bayesian data fusion with shared priors](https://mlsanity.com/paper/2212.07311/) (2022)
- [Federated Bayesian Deep Learning: The Application of Statistical Aggregation Methods to Bayesian Models](https://mlsanity.com/paper/2403.15263/) (2024)
- [A Bayesian Framework for Clustered Federated Learning](https://mlsanity.com/paper/2410.15473/) (2024)
- [Privacy Preserving Bayesian Federated Learning in Heterogeneous Settings](https://mlsanity.com/paper/2306.07959/) (2023)
- [Federated Neuro-Symbolic Learning](https://mlsanity.com/paper/2308.15324/) (2023)
- [FedBayes: A Zero-Trust Federated Learning Aggregation to Defend Against Adversarial Attacks](https://mlsanity.com/paper/2312.04587/) (2023)
- [Bayesian Deep Learning Via Expectation Maximization and Turbo Deep Approximate Message Passing](https://mlsanity.com/paper/2402.07366/) (2024)
- [Bayesian Federated Inference for Survival Models](https://mlsanity.com/paper/2404.17464/) (2024)
- [Future-Proofing Medical Imaging with Privacy-Preserving Federated Learning and Uncertainty Quantification: A Review](https://mlsanity.com/paper/2409.16340/) (2024)
- [Federated Learning with Uncertainty and Personalization via Efficient Second-order Optimization](https://mlsanity.com/paper/2411.18385/) (2024)
- [Personalized Federated Learning with Hidden Information on Personalized Prior](https://mlsanity.com/paper/2211.10684/) (2022)
- [Personalized Federated Learning via Amortized Bayesian Meta-Learning](https://mlsanity.com/paper/2307.02222/) (2023)
- [Trustworthy Personalized Bayesian Federated Learning via Posterior Fine-Tune](https://mlsanity.com/paper/2402.16911/) (2024)
- [Bayesian Federated Model Compression for Communication and Computation Efficiency](https://mlsanity.com/paper/2404.07532/) (2024)
- [Variational Bayes for Federated Continual Learning](https://mlsanity.com/paper/2405.14291/) (2024)
- [Task Diversity in Bayesian Federated Learning: Simultaneous Processing of Classification and Regression](https://mlsanity.com/paper/2412.10897/) (2024)
- [Information-Geometric Barycenters for Bayesian Federated Learning](https://mlsanity.com/paper/2412.11646/) (2024)

## Abstract (arXiv)

Federated learning (FL) allows multiple clients to collaboratively learn a globally shared model through cycles of model aggregation and local model training, without the need to share data. Most existing FL methods train local models separately on different clients, and then simply average their parameters to obtain a centralized model on the server side. However, these approaches generally suffer from large aggregation errors and severe local forgetting, which are particularly bad in heterogeneous data settings. To tackle these issues, in this paper, we propose a novel FL framework that uses online Laplace approximation to approximate posteriors on both the client and server side. On the server side, a multivariate Gaussian product mechanism is employed to construct and maximize a global posterior, largely reducing the aggregation errors induced by large discrepancies between local models. On the client side, a prior loss that uses the global posterior probabilistic parameters delivered from the server is designed to guide the local training. Binding such learning constraints from other clients enables our method to mitigate local forgetting. Finally, we achieve state-of-the-art results on several benchmarks, clearly demonstrating the advantages of the proposed method.