# Personalized Federated Learning with Gaussian Processes

arXiv: https://arxiv.org/abs/2106.15482 · HTML: https://mlsanity.com/paper/2106.15482/

## Key facts

- Cited by 76 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 89 references that are in the map.
- First posted on arXiv 29 Jun 2021.

## Builds on (most cited first, 30 of 58)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [Personalized Federated Learning with Moreau Envelopes](https://mlsanity.com/paper/2006.08848/) (2020)
- [Local SGD Converges Fast and Communicates Little](https://mlsanity.com/paper/1805.09767/) (2018)
- [Federated Optimization: Distributed Machine Learning for On-Device Intelligence](https://mlsanity.com/paper/1610.02527/) (2016)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Federated Learning with Personalization Layers](https://mlsanity.com/paper/1912.00818/) (2019)
- [Towards Personalized Federated Learning](https://mlsanity.com/paper/2103.00710/) (2021)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Bayesian Nonparametric Federated Learning of Neural Networks](https://mlsanity.com/paper/1905.12022/) (2019)
- [Meta-SGD: Learning to Learn Quickly for Few-Shot Learning](https://mlsanity.com/paper/1707.09835/) (2017)
- [Adaptive Personalized Federated Learning](https://mlsanity.com/paper/2003.13461/) (2020)
- [FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization](https://mlsanity.com/paper/1909.13014/) (2019)
- [Think Locally, Act Globally: Federated Learning with Local and Global Representations](https://mlsanity.com/paper/2001.01523/) (2020)
- [Three Approaches for Personalization with Applications to Federated Learning](https://mlsanity.com/paper/2002.10619/) (2020)
- [Personalized Federated Learning: A Meta-Learning Approach](https://mlsanity.com/paper/2002.07948/) (2020)
- [Personalized Cross-Silo Federated Learning on Non-IID Data](https://mlsanity.com/paper/2007.03797/) (2020)
- [Improving Federated Learning Personalization via Model Agnostic Meta Learning](https://mlsanity.com/paper/1909.12488/) (2019)
- [Scalable Variational Gaussian Process Classification](https://mlsanity.com/paper/1411.2005/) (2014)
- [Personalized Federated Learning using Hypernetworks](https://mlsanity.com/paper/2103.04628/) (2021)
- [Federated Learning of a Mixture of Global and Local Models](https://mlsanity.com/paper/2002.05516/) (2020)

## Built on by (most cited first, 30 of 76)

- [Personalized Federated Learning with Feature Alignment and Classifier Collaboration](https://mlsanity.com/paper/2306.11867/) (2023)
- [Personalized Federated Learning via Variational Bayesian Inference](https://mlsanity.com/paper/2206.07977/) (2022)
- [Personalized Federated Learning through Local Memorization](https://mlsanity.com/paper/2111.09360/) (2021)
- [pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learning](https://mlsanity.com/paper/2206.03655/) (2022)
- [Test-Time Robust Personalization for Federated Learning](https://mlsanity.com/paper/2205.10920/) (2022)
- [Personalized Federated Learning under Mixture of Distributions](https://mlsanity.com/paper/2305.01068/) (2023)
- [FedTP: Federated Learning by Transformer Personalization](https://mlsanity.com/paper/2211.01572/) (2022)
- [Acceleration of Federated Learning with Alleviated Forgetting in Local Training](https://mlsanity.com/paper/2203.02645/) (2022)
- [Bayesian Federated Learning: A Survey](https://mlsanity.com/paper/2304.13267/) (2023)
- [Federated Learning as Variational Inference: A Scalable Expectation Propagation Approach](https://mlsanity.com/paper/2302.04228/) (2023)
- [Confidence-aware Personalized Federated Learning via Variational Expectation Maximization](https://mlsanity.com/paper/2305.12557/) (2023)
- [Federated Learning from Small Datasets](https://mlsanity.com/paper/2110.03469/) (2021)
- [Personalized Federated Learning via Feature Distribution Adaptation](https://mlsanity.com/paper/2411.00329/) (2024)
- [Federated Learning with Uncertainty via Distilled Predictive Distributions](https://mlsanity.com/paper/2206.07562/) (2022)
- [Personalized Online Federated Learning with Multiple Kernels](https://mlsanity.com/paper/2311.05108/) (2023)
- [Bayesian Uncertainty for Gradient Aggregation in Multi-Task Learning](https://mlsanity.com/paper/2402.04005/) (2024)
- [FedDWA: Personalized Federated Learning with Dynamic Weight Adjustment](https://mlsanity.com/paper/2305.06124/) (2023)
- [FedHB: Hierarchical Bayesian Federated Learning](https://mlsanity.com/paper/2305.04979/) (2023)
- [FedSampling: A Better Sampling Strategy for Federated Learning](https://mlsanity.com/paper/2306.14245/) (2023)
- [PRIOR: Personalized Prior for Reactivating the Information Overlooked in Federated Learning](https://mlsanity.com/paper/2310.09183/) (2023)
- [Spectral Co-Distillation for Personalized Federated Learning](https://mlsanity.com/paper/2401.17124/) (2024)
- [SemiPFL: Personalized Semi-Supervised Federated Learning Framework for Edge Intelligence](https://mlsanity.com/paper/2203.08176/) (2022)
- [Fed-CO2: Cooperation of Online and Offline Models for Severe Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2312.13923/) (2023)
- [DKL-KAN: Scalable Deep Kernel Learning using Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2407.21176/) (2024)
- [Personalized Federated Learning with Server-Side Information](https://mlsanity.com/paper/2205.11044/) (2022)
- [Guided Deep Kernel Learning](https://mlsanity.com/paper/2302.09574/) (2023)
- [Federated Learning of Shareable Bases for Personalization-Friendly Image Classification](https://mlsanity.com/paper/2304.07882/) (2023)
- [FedCal: Achieving Local and Global Calibration in Federated Learning via Aggregated Parameterized Scaler](https://mlsanity.com/paper/2405.15458/) (2024)
- [Federated Bayesian Neural Regression: A Scalable Global Federated Gaussian Process](https://mlsanity.com/paper/2206.06357/) (2022)
- [Semi-Synchronous Personalized Federated Learning over Mobile Edge Networks](https://mlsanity.com/paper/2209.13115/) (2022)

## Abstract (arXiv)

Federated learning aims to learn a global model that performs well on client devices with limited cross-client communication. Personalized federated learning (PFL) further extends this setup to handle data heterogeneity between clients by learning personalized models. A key challenge in this setting is to learn effectively across clients even though each client has unique data that is often limited in size. Here we present pFedGP, a solution to PFL that is based on Gaussian processes (GPs) with deep kernel learning. GPs are highly expressive models that work well in the low data regime due to their Bayesian nature. However, applying GPs to PFL raises multiple challenges. Mainly, GPs performance depends heavily on access to a good kernel function, and learning a kernel requires a large training set. Therefore, we propose learning a shared kernel function across all clients, parameterized by a neural network, with a personal GP classifier for each client. We further extend pFedGP to include inducing points using two novel methods, the first helps to improve generalization in the low data regime and the second reduces the computational cost. We derive a PAC-Bayes generalization bound on novel clients and empirically show that it gives non-vacuous guarantees. Extensive experiments on standard PFL benchmarks with CIFAR-10, CIFAR-100, and CINIC-10, and on a new setup of learning under input noise show that pFedGP achieves well-calibrated predictions while significantly outperforming baseline methods, reaching up to 21% in accuracy gain.