# Personalized Federated Learning via Variational Bayesian Inference

arXiv: https://arxiv.org/abs/2206.07977 · HTML: https://mlsanity.com/paper/2206.07977/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 56 references that are in the map.
- First posted on arXiv 16 Jun 2022.

## Builds on (most cited first, 30 of 37)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [Personalized Federated Learning with Moreau Envelopes](https://mlsanity.com/paper/2006.08848/) (2020)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Federated Learning with Personalization Layers](https://mlsanity.com/paper/1912.00818/) (2019)
- [Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints](https://mlsanity.com/paper/1910.01991/) (2019)
- [Robust and Communication-Efficient Federated Learning from Non-IID Data](https://mlsanity.com/paper/1903.02891/) (2019)
- [Bayesian Nonparametric Federated Learning of Neural Networks](https://mlsanity.com/paper/1905.12022/) (2019)
- [FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization](https://mlsanity.com/paper/1909.13014/) (2019)
- [Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users](https://mlsanity.com/paper/2007.06823/) (2020)
- [Personalized Cross-Silo Federated Learning on Non-IID Data](https://mlsanity.com/paper/2007.03797/) (2020)
- [Federated Learning of a Mixture of Global and Local Models](https://mlsanity.com/paper/2002.05516/) (2020)
- [FetchSGD: Communication-Efficient Federated Learning with Sketching](https://mlsanity.com/paper/2007.07682/) (2020)
- [FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning](https://mlsanity.com/paper/2009.01974/) (2020)
- [Practical Deep Learning with Bayesian Principles](https://mlsanity.com/paper/1906.02506/) (2019)
- [Federated Meta-Learning with Fast Convergence and Efficient Communication](https://mlsanity.com/paper/1802.07876/) (2018)
- [One-Shot Federated Learning](https://mlsanity.com/paper/1902.11175/) (2019)
- [Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithms](https://mlsanity.com/paper/2010.05273/) (2020)
- [FedDANE: A Federated Newton-Type Method](https://mlsanity.com/paper/2001.01920/) (2020)
- [Personalized Federated Learning with Gaussian Processes](https://mlsanity.com/paper/2106.15482/) (2021)
- [Concentration of tempered posteriors and of their variational approximations](https://mlsanity.com/paper/1706.09293/) (2017)
- [Posterior Concentration for Sparse Deep Learning](https://mlsanity.com/paper/1803.09138/) (2018)
- [A Bayesian Federated Learning Framework with Online Laplace Approximation](https://mlsanity.com/paper/2102.01936/) (2021)

## Built on by (most cited first, 30 of 68)

- [Heterogeneous Federated Learning: State-of-the-art and Research Challenges](https://mlsanity.com/paper/2307.10616/) (2023)
- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [Towards Personalized Federated Learning via Heterogeneous Model Reassembly](https://mlsanity.com/paper/2308.08643/) (2023)
- [Bayesian Federated Learning: A Survey](https://mlsanity.com/paper/2304.13267/) (2023)
- [Confidence-aware Personalized Federated Learning via Variational Expectation Maximization](https://mlsanity.com/paper/2305.12557/) (2023)
- [FedBone: Towards Large-Scale Federated Multi-Task Learning](https://mlsanity.com/paper/2306.17465/) (2023)
- [Modeling Inter-Intra Heterogeneity for Graph Federated Learning](https://mlsanity.com/paper/2412.11402/) (2024)
- [FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Data](https://mlsanity.com/paper/2401.08977/) (2024)
- [Privacy Preserving Federated Learning in Medical Imaging with Uncertainty Estimation](https://mlsanity.com/paper/2406.12815/) (2024)
- [FedHB: Hierarchical Bayesian Federated Learning](https://mlsanity.com/paper/2305.04979/) (2023)
- [PRIOR: Personalized Prior for Reactivating the Information Overlooked in Federated Learning](https://mlsanity.com/paper/2310.09183/) (2023)
- [A Comprehensive Survey of Federated Transfer Learning: Challenges, Methods and Applications](https://mlsanity.com/paper/2403.01387/) (2024)
- [Towards Efficient and Certified Recovery from Poisoning Attacks in Federated Learning](https://mlsanity.com/paper/2401.08216/) (2024)
- [MAP: Model Aggregation and Personalization in Federated Learning with Incomplete Classes](https://mlsanity.com/paper/2404.09232/) (2024)
- [How to Combine Variational Bayesian Networks in Federated Learning](https://mlsanity.com/paper/2206.10897/) (2022)
- [Federated Learning via Variational Bayesian Inference: Personalization, Sparsity and Clustering](https://mlsanity.com/paper/2303.04345/) (2023)
- [Personalized Federated Learning of Probabilistic Models: A PAC-Bayesian Approach](https://mlsanity.com/paper/2401.08351/) (2024)
- [Rethinking Personalized Federated Learning with Clustering-based Dynamic Graph Propagation](https://mlsanity.com/paper/2401.15874/) (2024)
- [Avoid Overfitting User Specific Information in Federated Keyword Spotting](https://mlsanity.com/paper/2206.08864/) (2022)
- [Differentiated Federated Reinforcement Learning Based Traffic Offloading on Space-Air-Ground Integrated Networks](https://mlsanity.com/paper/2212.02075/) (2022)
- [Federated Variational Inference Methods for Structured Latent Variable Models](https://mlsanity.com/paper/2302.03314/) (2023)
- [FedBIAD: Communication-Efficient and Accuracy-Guaranteed Federated Learning with Bayesian Inference-Based Adaptive Dropout](https://mlsanity.com/paper/2307.07172/) (2023)
- [FedSI: Federated Subnetwork Inference for Efficient Uncertainty Quantification](https://mlsanity.com/paper/2404.15657/) (2024)
- [Federated Bayesian Deep Learning: The Application of Statistical Aggregation Methods to Bayesian Models](https://mlsanity.com/paper/2403.15263/) (2024)
- [Federated Neural Nonparametric Point Processes](https://mlsanity.com/paper/2410.05637/) (2024)
- [A Bayesian Framework for Clustered Federated Learning](https://mlsanity.com/paper/2410.15473/) (2024)
- [Personalized Language Model Learning on Text Data Without User Identifiers](https://mlsanity.com/paper/2501.06062/) (2025)
- [Federated Variational Inference: Towards Improved Personalization and Generalization](https://mlsanity.com/paper/2305.13672/) (2023)
- [Privacy Preserving Bayesian Federated Learning in Heterogeneous Settings](https://mlsanity.com/paper/2306.07959/) (2023)
- [Federated Neuro-Symbolic Learning](https://mlsanity.com/paper/2308.15324/) (2023)

## Abstract (arXiv)

Federated learning faces huge challenges from model overfitting due to the lack of data and statistical diversity among clients. To address these challenges, this paper proposes a novel personalized federated learning method via Bayesian variational inference named pFedBayes. To alleviate the overfitting, weight uncertainty is introduced to neural networks for clients and the server. To achieve personalization, each client updates its local distribution parameters by balancing its construction error over private data and its KL divergence with global distribution from the server. Theoretical analysis gives an upper bound of averaged generalization error and illustrates that the convergence rate of the generalization error is minimax optimal up to a logarithmic factor. Experiments show that the proposed method outperforms other advanced personalized methods on personalized models, e.g., pFedBayes respectively outperforms other SOTA algorithms by 1.25%, 0.42% and 11.71% on MNIST, FMNIST and CIFAR-10 under non-i.i.d. limited data.