# Federated Learning on Non-IID Data: A Survey

arXiv: https://arxiv.org/abs/2106.06843 · HTML: https://mlsanity.com/paper/2106.06843/

## Key facts

- Cited by 346 papers in the map (data through 14 Oct 2026).
- Builds on 110 of its 190 references that are in the map.
- First posted on arXiv 12 Jun 2021.

## Builds on (most cited first, 30 of 110)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Personalized Federated Learning with Moreau Envelopes](https://mlsanity.com/paper/2006.08848/) (2020)
- [Federated Learning on Non-IID Data Silos: An Experimental Study](https://mlsanity.com/paper/2102.02079/) (2021)
- [Multi-view Convolutional Neural Networks for 3D Shape Recognition](https://mlsanity.com/paper/1505.00880/) (2015)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Federated Learning with Personalization Layers](https://mlsanity.com/paper/1912.00818/) (2019)
- [Federated Learning with Differential Privacy: Algorithms and Performance Analysis](https://mlsanity.com/paper/1911.00222/) (2019)
- [An Efficient Framework for Clustered Federated Learning](https://mlsanity.com/paper/2006.04088/) (2020)

## Built on by (most cited first, 30 of 346)

- [Heterogeneous Federated Learning: State-of-the-art and Research Challenges](https://mlsanity.com/paper/2307.10616/) (2023)
- [NOTE: Robust Continual Test-time Adaptation Against Temporal Correlation](https://mlsanity.com/paper/2208.05117/) (2022)
- [Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmark](https://mlsanity.com/paper/2311.06750/) (2023)
- [Personalized Federated Learning with Feature Alignment and Classifier Collaboration](https://mlsanity.com/paper/2306.11867/) (2023)
- [Selective Aggregation for Low-Rank Adaptation in Federated Learning](https://mlsanity.com/paper/2410.01463/) (2024)
- [Universal Test-time Adaptation through Weight Ensembling, Diversity Weighting, and Prior Correction](https://mlsanity.com/paper/2306.00650/) (2023)
- [Client Selection in Federated Learning: Principles, Challenges, and Opportunities](https://mlsanity.com/paper/2211.01549/) (2022)
- [Better Generative Replay for Continual Federated Learning](https://mlsanity.com/paper/2302.13001/) (2023)
- [Federated Conformal Predictors for Distributed Uncertainty Quantification](https://mlsanity.com/paper/2305.17564/) (2023)
- [Cross-Silo Federated Learning: Challenges and Opportunities](https://mlsanity.com/paper/2206.12949/) (2022)
- [Federated Learning for Medical Applications: A Taxonomy, Current Trends, Challenges, and Future Research Directions](https://mlsanity.com/paper/2208.03392/) (2022)
- [An Aggregation-Free Federated Learning for Tackling Data Heterogeneity](https://mlsanity.com/paper/2404.18962/) (2024)
- [Towards Federated Clustering: A Federated Fuzzy c-Means Algorithm (FFCM)](https://mlsanity.com/paper/2201.07316/) (2022)
- [FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergence](https://mlsanity.com/paper/2204.08125/) (2022)
- [A Critical Review on the Use (and Misuse) of Differential Privacy in Machine Learning](https://mlsanity.com/paper/2206.04621/) (2022)
- [CrowdGuard: Federated Backdoor Detection in Federated Learning](https://mlsanity.com/paper/2210.07714/) (2022)
- [Personalized Federated Learning with Adaptive Batchnorm for Healthcare](https://mlsanity.com/paper/2112.00734/) (2021)
- [Privacy and Fairness in Federated Learning: on the Perspective of Trade-off](https://mlsanity.com/paper/2306.14123/) (2023)
- [FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decomposition](https://mlsanity.com/paper/2404.18848/) (2024)
- [Federated Learning with GAN-based Data Synthesis for Non-IID Clients](https://mlsanity.com/paper/2206.05507/) (2022)
- [Selective Knowledge Sharing for Privacy-Preserving Federated Distillation without A Good Teacher](https://mlsanity.com/paper/2304.01731/) (2023)
- [Towards Effective Clustered Federated Learning: A Peer-to-peer Framework with Adaptive Neighbor Matching](https://mlsanity.com/paper/2203.12285/) (2022)
- [FRAug: Tackling Federated Learning with Non-IID Features via Representation Augmentation](https://mlsanity.com/paper/2205.14900/) (2022)
- [A Survey of Trustworthy Federated Learning with Perspectives on Security, Robustness, and Privacy](https://mlsanity.com/paper/2302.10637/) (2023)
- [FS-Real: Towards Real-World Cross-Device Federated Learning](https://mlsanity.com/paper/2303.13363/) (2023)
- [Phoenix: A Federated Generative Diffusion Model](https://mlsanity.com/paper/2306.04098/) (2023)
- [Federated Learning for Computer Vision](https://mlsanity.com/paper/2308.13558/) (2023)
- [A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiency](https://mlsanity.com/paper/2307.10655/) (2023)
- [CalFAT: Calibrated Federated Adversarial Training with Label Skewness](https://mlsanity.com/paper/2205.14926/) (2022)
- [MP-FedCL: Multiprototype Federated Contrastive Learning for Edge Intelligence](https://mlsanity.com/paper/2304.01950/) (2023)

## Abstract (arXiv)

Federated learning is an emerging distributed machine learning framework for privacy preservation. However, models trained in federated learning usually have worse performance than those trained in the standard centralized learning mode, especially when the training data are not independent and identically distributed (Non-IID) on the local devices. In this survey, we pro-vide a detailed analysis of the influence of Non-IID data on both parametric and non-parametric machine learning models in both horizontal and vertical federated learning. In addition, cur-rent research work on handling challenges of Non-IID data in federated learning are reviewed, and both advantages and disadvantages of these approaches are discussed. Finally, we suggest several future research directions before concluding the paper.