# A Survey on Heterogeneous Federated Learning

arXiv: https://arxiv.org/abs/2210.04505 · HTML: https://mlsanity.com/paper/2210.04505/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 118 of its 223 references that are in the map.
- First posted on arXiv 10 Oct 2022.

## Builds on (most cited first, 30 of 118)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Federated Optimization: Distributed Machine Learning for On-Device Intelligence](https://mlsanity.com/paper/1610.02527/) (2016)
- [Federated Learning with Personalization Layers](https://mlsanity.com/paper/1912.00818/) (2019)
- [An Efficient Framework for Clustered Federated Learning](https://mlsanity.com/paper/2006.04088/) (2020)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [Towards Personalized Federated Learning](https://mlsanity.com/paper/2103.00710/) (2021)
- [Differentially Private Federated Learning: A Client Level Perspective](https://mlsanity.com/paper/1712.07557/) (2017)
- [FedMD: Heterogenous Federated Learning via Model Distillation](https://mlsanity.com/paper/1910.03581/) (2019)
- [Robust and Communication-Efficient Federated Learning from Non-IID Data](https://mlsanity.com/paper/1903.02891/) (2019)
- [Federated Machine Learning: Concept and Applications](https://mlsanity.com/paper/1902.04885/) (2019)
- [Split learning for health: Distributed deep learning without sharing raw patient data](https://mlsanity.com/paper/1812.00564/) (2018)
- [Return of Frustratingly Easy Domain Adaptation](https://mlsanity.com/paper/1511.05547/) (2015)
- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021)
- [A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection](https://mlsanity.com/paper/1907.09693/) (2019)
- [Adaptive Personalized Federated Learning](https://mlsanity.com/paper/2003.13461/) (2020)
- [FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization](https://mlsanity.com/paper/1909.13014/) (2019)
- [Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge](https://mlsanity.com/paper/1804.08333/) (2018)
- [HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients](https://mlsanity.com/paper/2010.01264/) (2020)

## Built on by (most cited first, 30 of 42)

- [Heterogeneous Federated Learning: State-of-the-art and Research Challenges](https://mlsanity.com/paper/2307.10616/) (2023)
- [DiLoCo: Distributed Low-Communication Training of Language Models](https://mlsanity.com/paper/2311.08105/) (2023)
- [When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directions](https://mlsanity.com/paper/2306.15546/) (2023)
- [Cross-Silo Prototypical Calibration for Federated Learning with Non-IID Data](https://mlsanity.com/paper/2308.03457/) (2023)
- [Federated Large Language Models: Current Progress and Future Directions](https://mlsanity.com/paper/2409.15723/) (2024)
- [Overcoming Data and Model Heterogeneities in Decentralized Federated Learning via Synthetic Anchors](https://mlsanity.com/paper/2405.11525/) (2024)
- [Multimodal Federated Learning in Healthcare: a Review](https://mlsanity.com/paper/2310.09650/) (2023)
- [FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneity](https://mlsanity.com/paper/2404.09816/) (2024)
- [Open Challenges and Opportunities in Federated Foundation Models Towards Biomedical Healthcare](https://mlsanity.com/paper/2405.06784/) (2024)
- [Advances in Robust Federated Learning: A Survey with Heterogeneity Considerations](https://mlsanity.com/paper/2405.09839/) (2024)
- [FedYolo: Augmenting Federated Learning with Pretrained Transformers](https://mlsanity.com/paper/2307.04905/) (2023)
- [DapperFL: Domain Adaptive Federated Learning with Model Fusion Pruning for Edge Devices](https://mlsanity.com/paper/2412.05823/) (2024)
- [FedLPS: Heterogeneous Federated Learning for Multiple Tasks with Local Parameter Sharing](https://mlsanity.com/paper/2402.08578/) (2024)
- [Federated Graph Condensation with Information Bottleneck Principles](https://mlsanity.com/paper/2405.03911/) (2024)
- [An effective theory of collective deep learning](https://mlsanity.com/paper/2310.12802/) (2023)
- [Aggregating Intrinsic Information to Enhance BCI Performance through Federated Learning](https://mlsanity.com/paper/2308.11636/) (2023)
- [Internal Cross-layer Gradients for Extending Homogeneity to Heterogeneity in Federated Learning](https://mlsanity.com/paper/2308.11464/) (2023)
- [Federated Out-of-Distribution Generalization: A Causal Augmentation View](https://mlsanity.com/paper/2504.19882/) (2025)
- [UPFL: Unsupervised Personalized Federated Learning towards New Clients](https://mlsanity.com/paper/2307.15994/) (2023)
- [Entangled Mean Estimation in High-Dimensions](https://mlsanity.com/paper/2501.05425/) (2025)
- [Global Intervention and Distillation for Federated Out-of-Distribution Generalization](https://mlsanity.com/paper/2504.00850/) (2025)
- [Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environments](https://mlsanity.com/paper/2505.19699/) (2025)
- [Personalized Federated Learning via Amortized Bayesian Meta-Learning](https://mlsanity.com/paper/2307.02222/) (2023)
- [Dataset Dictionary Learning in a Wasserstein Space for Federated Domain Adaptation](https://mlsanity.com/paper/2407.11647/) (2024)
- [Conditioning on Local Statistics for Scalable Heterogeneous Federated Learning](https://mlsanity.com/paper/2503.00378/) (2025)
- [Data-Free Black-Box Federated Learning via Zeroth-Order Gradient Estimation](https://mlsanity.com/paper/2503.06028/) (2025)
- [DP2FL: Dual Prompt Personalized Federated Learning in Foundation Models](https://mlsanity.com/paper/2504.16357/) (2025)
- [Decision-Focused Federated Learning Under Heterogeneous Objectives and Constraints](https://mlsanity.com/paper/2604.20031/) (2026)
- [Heterogeneous Federated Learning via Personalized Generative Networks](https://mlsanity.com/paper/2308.13265/) (2023)
- [Evaluating Multi-Global Server Architecture for Federated Learning](https://mlsanity.com/paper/2311.15382/) (2023)

## Abstract (arXiv)

Federated learning (FL) has been proposed to protect data privacy and virtually assemble the isolated data silos by cooperatively training models among organizations without breaching privacy and security. However, FL faces heterogeneity from various aspects, including data space, statistical, and system heterogeneity. For example, collaborative organizations without conflict of interest often come from different areas and have heterogeneous data from different feature spaces. Participants may also want to train heterogeneous personalized local models due to non-IID and imbalanced data distribution and various resource-constrained devices. Therefore, heterogeneous FL is proposed to address the problem of heterogeneity in FL. In this survey, we comprehensively investigate the domain of heterogeneous FL in terms of data space, statistical, system, and model heterogeneity. We first give an overview of FL, including its definition and categorization. Then, We propose a precise taxonomy of heterogeneous FL settings for each type of heterogeneity according to the problem setting and learning objective. We also investigate the transfer learning methodologies to tackle the heterogeneity in FL. We further present the applications of heterogeneous FL. Finally, we highlight the challenges and opportunities and envision promising future research directions toward new framework design and trustworthy approaches.