# Personalized Federated Learning under Mixture of Distributions

arXiv: https://arxiv.org/abs/2305.01068 · HTML: https://mlsanity.com/paper/2305.01068/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 65 references that are in the map.
- First posted on arXiv 1 May 2023.

## Builds on (most cited first, 30 of 46)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [Personalized Federated Learning with Moreau Envelopes](https://mlsanity.com/paper/2006.08848/) (2020)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [An Efficient Framework for Clustered Federated Learning](https://mlsanity.com/paper/2006.04088/) (2020)
- [Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints](https://mlsanity.com/paper/1910.01991/) (2019)
- [Towards Personalized Federated Learning](https://mlsanity.com/paper/2103.00710/) (2021)
- [Robust and Communication-Efficient Federated Learning from Non-IID Data](https://mlsanity.com/paper/1903.02891/) (2019)
- [Federated Learning in Mobile Edge Networks: A Comprehensive Survey](https://mlsanity.com/paper/1909.11875/) (2019)
- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021)
- [Adaptive Personalized Federated Learning](https://mlsanity.com/paper/2003.13461/) (2020)
- [Open-Set Recognition: a Good Closed-Set Classifier is All You Need?](https://mlsanity.com/paper/2110.06207/) (2021)
- [Personalized Federated Learning: A Meta-Learning Approach](https://mlsanity.com/paper/2002.07948/) (2020)
- [Improving Federated Learning Personalization via Model Agnostic Meta Learning](https://mlsanity.com/paper/1909.12488/) (2019)
- [Personalized Federated Learning using Hypernetworks](https://mlsanity.com/paper/2103.04628/) (2021)
- [Federated Multi-Task Learning under a Mixture of Distributions](https://mlsanity.com/paper/2108.10252/) (2021)
- [FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning](https://mlsanity.com/paper/2009.01974/) (2020)
- [Federated Evaluation of On-device Personalization](https://mlsanity.com/paper/1910.10252/) (2019)
- [FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2108.06098/) (2021)
- [Interpolation-Prediction Networks for Irregularly Sampled Time Series](https://mlsanity.com/paper/1909.07782/) (2019)
- [Decentralized Collaborative Learning of Personalized Models over Networks](https://mlsanity.com/paper/1610.05202/) (2016)
- [Personalized Federated Learning with Gaussian Processes](https://mlsanity.com/paper/2106.15482/) (2021)

## Built on by (most cited first, 30 of 37)

- [Towards Personalized Federated Learning via Heterogeneous Model Reassembly](https://mlsanity.com/paper/2308.08643/) (2023)
- [PeFLL: Personalized Federated Learning by Learning to Learn](https://mlsanity.com/paper/2306.05515/) (2023)
- [A Survey for Federated Learning Evaluations: Goals and Measures](https://mlsanity.com/paper/2308.11841/) (2023)
- [FOOGD: Federated Collaboration for Both Out-of-distribution Generalization and Detection](https://mlsanity.com/paper/2410.11397/) (2024)
- [Provably Personalized and Robust Federated Learning](https://mlsanity.com/paper/2306.08393/) (2023)
- [ZooPFL: Exploring Black-box Foundation Models for Personalized Federated Learning](https://mlsanity.com/paper/2310.05143/) (2023)
- [One-Shot Sequential Federated Learning for Non-IID Data by Enhancing Local Model Diversity](https://mlsanity.com/paper/2404.12130/) (2024)
- [Personalized Federated Learning with Mixture of Models for Adaptive Prediction and Model Fine-Tuning](https://mlsanity.com/paper/2410.21547/) (2024)
- [Towards the Theory of Unsupervised Federated Learning: Non-asymptotic Analysis of Federated EM Algorithms](https://mlsanity.com/paper/2310.15330/) (2023)
- [Explicit Personalization and Local Training: Double Communication Acceleration in Federated Learning](https://mlsanity.com/paper/2305.13170/) (2023)
- [FedSPD: A Soft-clustering Approach for Personalized Decentralized Federated Learning](https://mlsanity.com/paper/2410.18862/) (2024)
- [FedSI: Federated Subnetwork Inference for Efficient Uncertainty Quantification](https://mlsanity.com/paper/2404.15657/) (2024)
- [Differentially Private Clustered Federated Learning](https://mlsanity.com/paper/2405.19272/) (2024)
- [FedMerge: Federated Personalization via Model Merging](https://mlsanity.com/paper/2504.06768/) (2025)
- [Enhancing Clustered Federated Learning: Integration of Strategies and Improved Methodologies](https://mlsanity.com/paper/2310.05397/) (2023)
- [Federated Gaussian Mixture Models](https://mlsanity.com/paper/2506.01780/) (2025)
- [Fair Federated Data Clustering through Personalization: Bridging the Gap between Diverse Data Distributions](https://mlsanity.com/paper/2407.04302/) (2024)
- [Federated Variational Inference for Bayesian Mixture Models](https://mlsanity.com/paper/2502.12684/) (2025)
- [Federated Instrumental Variable Analysis via Federated Generalized Method of Moments](https://mlsanity.com/paper/2505.21012/) (2025)
- [Beyond Aggregation: Guiding Clients in Heterogeneous Federated Learning](https://mlsanity.com/paper/2509.23049/) (2025)
- [A Federated Generalized Expectation-Maximization Algorithm for Mixture Models with an Unknown Number of Components](https://mlsanity.com/paper/2601.21160/) (2026)
- [Auditable Homomorphic-based Decentralized Collaborative AI with Attribute-based Differential Privacy](https://mlsanity.com/paper/2403.00023/) (2024)
- [Personalized Additive Modeling for Multi-level Federated Learning](https://mlsanity.com/paper/2405.16472/) (2024)
- [Fast convergence of a Federated Expectation-Maximization Algorithm](https://mlsanity.com/paper/2408.05819/) (2024)
- [Your Data, My Model: Learning Who Really Helps in Federated Learning](https://mlsanity.com/paper/2409.02064/) (2024)
- [Deep Domain Isolation and Sample Clustered Federated Learning for Semantic Segmentation](https://mlsanity.com/paper/2410.14693/) (2024)
- [Optimize Incompatible Parameters through Compatibility-aware Knowledge Integration](https://mlsanity.com/paper/2501.07596/) (2025)
- [WeiDetect: Weibull Distribution-Based Defense against Poisoning Attacks in Federated Learning for Network Intrusion Detection Systems](https://mlsanity.com/paper/2504.04367/) (2025)
- [Federated Learning with Unlabeled Clients: Personalization Can Happen in Low Dimensions](https://mlsanity.com/paper/2505.15579/) (2025)
- [Shift Happens: Mixture of Experts based Continual Adaptation in Federated Learning](https://mlsanity.com/paper/2506.18789/) (2025)

## Abstract (arXiv)

The recent trend towards Personalized Federated Learning (PFL) has garnered significant attention as it allows for the training of models that are tailored to each client while maintaining data privacy. However, current PFL techniques primarily focus on modeling the conditional distribution heterogeneity (i.e. concept shift), which can result in suboptimal performance when the distribution of input data across clients diverges (i.e. covariate shift). Additionally, these techniques often lack the ability to adapt to unseen data, further limiting their effectiveness in real-world scenarios. To address these limitations, we propose a novel approach, FedGMM, which utilizes Gaussian mixture models (GMM) to effectively fit the input data distributions across diverse clients. The model parameters are estimated by maximum likelihood estimation utilizing a federated Expectation-Maximization algorithm, which is solved in closed form and does not assume gradient similarity. Furthermore, FedGMM possesses an additional advantage of adapting to new clients with minimal overhead, and it also enables uncertainty quantification. Empirical evaluations on synthetic and benchmark datasets demonstrate the superior performance of our method in both PFL classification and novel sample detection.