Data to 7 Oct 2026

Federated Learning on Non-IID Data Silos: An Experimental Study

arXiv 2102.02079 ↗First posted 3 Feb 2021cs.LG cs.DCcited by 566

Not on the map. Why?

Abstract (arXiv)

Due to the increasing privacy concerns and data regulations, training data have been increasingly fragmented, forming distributed databases of multiple "data silos" (e.g., within different organizations and countries). To develop effective machine learning services, there is a must to exploit data from such distributed databases without exchanging the raw data. Recently, federated learning (FL) has been a solution with growing interests, which enables multiple parties to collaboratively train a machine learning model without exchanging their local data. A key and common challenge on distributed databases is the heterogeneity of the data distribution among the parties. The data of different parties are usually non-independently and identically distributed (i.e., non-IID). There have been many FL algorithms to address the learning effectiveness under non-IID data settings. However, there lacks an experimental study on systematically understanding their advantages and disadvantages, as previous studies have very rigid data partitioning strategies among parties, which are hardly representative and thorough. In this paper, to help researchers better understand and study the non-IID data setting in federated learning, we propose comprehensive data partitioning strategies to cover the typical non-IID data cases. Moreover, we conduct extensive experiments to evaluate state-of-the-art FL algorithms. We find that non-IID does bring significant challenges in learning accuracy of FL algorithms, and none of the existing state-of-the-art FL algorithms outperforms others in all cases. Our experiments provide insights for future studies of addressing the challenges in "data silos".

Key facts. Cited by 566 papers in the map (data through 7 Oct 2026). Builds on 47 of its 85 references that are in the map. First posted on arXiv 3 Feb 2021.

Builds on

47 of 85 references.

References · 47

2015Deep Residual Learning for Image Recognitioncited by 51,809
2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
2019Advances and Open Problems in Federated Learningcited by 3,021
2018Federated Optimization in Heterogeneous Networkscited by 2,971
2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
2016Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoisingcited by 1,352
2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
2020Adaptive Federated Optimizationcited by 1,036
2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836

35 more

2019Towards Federated Learning at Scale: System Designcited by 826
2018LEAF: A Benchmark for Federated Settingscited by 780
2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
2020Personalized Federated Learning with Moreau Envelopescited by 612
2018Federated Learning for Mobile Keyboard Predictioncited by 565
2020Federated Learning with Matched Averagingcited by 564
2017Differentially Private Federated Learning: A Client Level Perspectivecited by 451
2019Agnostic Federated Learningcited by 451
2019Federated Machine Learning: Concept and Applicationscited by 409
2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 371
2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 298
2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 174
2013Minimizing Finite Sums with the Stochastic Average Gradientcited by 164
2019Federated Adversarial Domain Adaptationcited by 140
2019One-Shot Federated Learningcited by 124
2020Lower Bounds and Optimal Algorithms for Personalized Federated Learningcited by 122
2020Robust Federated Learning: The Case of Affine Distribution Shiftscited by 102
2020Privacy Preserving Vertical Federated Learning for Tree-based Modelscited by 89
2019Neo: A Learned Query Optimizercited by 80
2020Federated Bayesian Optimization via Thompson Samplingcited by 77
2020Practical One-Shot Federated Learning for Cross-Silo Settingcited by 76
2020Siloed Federated Learning for Multi-Centric Histopathology Datasetscited by 61
2020Federated Learning with Only Positive Labelscited by 55
2019Practical Federated Gradient Boosting Decision Treescited by 53
2019ALEX: An Updatable Adaptive Learned Indexcited by 43
2018Rafiki: Machine Learning as an Analytics Service Systemcited by 33
2020The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systemscited by 31
2020Benchmarking Learned Indexescited by 27
2017Attribution Modeling Increases Efficiency of Bidding in Display Advertisingcited by 19
2017Ease.ml: Towards Multi-tenant Resource Sharing for Machine Learning Workloadscited by 16
2019The Machine Learning Bazaar: Harnessing the ML Ecosystem for Effective System Developmentcited by 13
2020Evaluation Framework For Large-scale Federated Learningcited by 4
2019Understanding and Benchmarking the Impact of GDPR on Database Systemscited by 3
2020FLBench: A Benchmark Suite for Federated Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 31; 2022: 79; 2023: 119; 2024: 158; 2025: 131; 2026: 48.

566 papers.

Sort:

By area:

2021Model-Contrastive Federated Learningcited by 762
2021Federated Learning on Non-IID Data: A Surveycited by 345
2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
2021Federated Learning for Smart Healthcare: A Surveycited by 182
2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 160
2022NOTE: Robust Continual Test-time Adaptation Against Temporal Correlationcited by 150
2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 133
2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114
2022Federated Learning with Label Distribution Skew via Logits Calibrationcited by 108
2021DENSE: Data-Free One-Shot Federated Learningcited by 100
2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 100
2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98

38 more

2023Personalized Federated Learning with Feature Alignment and Classifier Collaborationcited by 97
2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
2021FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Taskscited by 84
2021FedProc: Prototypical Contrastive Federated Learning on Non-IID datacited by 80
2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
2022FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learningcited by 65
2023Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytescited by 55
2021FedLab: A Flexible Federated Learning Frameworkcited by 54
2023FedFed: Feature Distillation against Data Heterogeneity in Federated Learningcited by 54
2022Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learningcited by 52
2022Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environmentscited by 49
2022FedFA: Federated Learning with Feature Anchors to Align Features and Classifiers for Heterogeneous Datacited by 49
2021Flexible Clustered Federated Learning for Client-Level Data Distribution Shiftcited by 47
2024FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learningcited by 47
2022FedBalancer: Data and Pace Control for Efficient Federated Learning on Heterogeneous Clientscited by 46
2023Federated Conformal Predictors for Distributed Uncertainty Quantificationcited by 46
2022Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learningcited by 45
2021AutoFL: Enabling Heterogeneity-Aware Energy Efficient Federated Learningcited by 39
2021FedCG: Leverage Conditional GAN for Protecting Privacy and Maintaining Competitive Performance in Federated Learningcited by 39
2023Generalizable Heterogeneous Federated Cross-Correlation and Instance Similarity Learningcited by 39
2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 37
2021Non-IID data and Continual Learning processes in Federated Learning: A long road aheadcited by 36
2022Efficient Distribution Similarity Identification in Clustered Federated Learning via Principal Angles Between Client Data Subspacescited by 36
2022Learning Across Domains and Devices: Style-Driven Source-Free Domain Adaptation in Clustered Federated Learningcited by 35
2021FedIPR: Ownership Verification for Federated Deep Neural Network Modelscited by 33
2024An Aggregation-Free Federated Learning for Tackling Data Heterogeneitycited by 33
2021Differentially Private Federated Bayesian Optimization with Distributed Explorationcited by 32
2022RSCFed: Random Sampling Consensus Federated Semi-supervised Learningcited by 32
2021FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Modelscited by 31
2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
2022FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergencecited by 29
2024Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimizationcited by 28
2021Resource-Efficient Federated Learningcited by 26
2022TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernelscited by 26
2022Federated Learning with GAN-based Data Synthesis for Non-IID Clientscited by 25
2024Federated Unlearning: A Survey on Methods, Design Guidelines, and Evaluation Metricscited by 25
2022FedGCN: Convergence-Communication Tradeoffs in Federated Training of Graph Convolutional Networkscited by 24

50 most cited of 566 All 566 (JSON)