mlsanityData to 7 Oct 2026

The Non-IID Data Quagmire of Decentralized Machine Learning

arXiv 1910.00189 ↗First posted 1 Oct 2019cs.LG stat.MLcited by 303

Not on the map. Why?

Abstract (arXiv)

Many large-scale machine learning (ML) applications need to perform decentralized learning over datasets generated at different devices and locations. Such datasets pose a significant challenge to decentralized learning because their different contexts result in significant data distribution skew across devices/locations. In this paper, we take a step toward better understanding this challenge by presenting a detailed experimental study of decentralized DNN training on a common type of data skew: skewed distribution of data labels across devices/locations. Our study shows that: (i) skewed data labels are a fundamental and pervasive problem for decentralized learning, causing significant accuracy loss across many ML applications, DNN models, training datasets, and decentralized learning algorithms; (ii) the problem is particularly challenging for DNN models with batch normalization; and (iii) the degree of data skew is a key determinant of the difficulty of the problem. Based on these findings, we present SkewScout, a system-level approach that adapts the communication frequency of decentralized learning algorithms to the (skew-induced) accuracy loss between data partitions. We also show that group normalization can recover much of the accuracy loss of batch normalization.

Key facts. Cited by 303 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 64; 2022: 71; 2023: 65; 2024: 55; 2025: 33; 2026: 15.

303 papers.

Sort:

By area:

  1. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  2. 2021A Field Guide to Federated Optimizationcited by 312
  3. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  4. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  5. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 188
  6. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 140
  7. 2022Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challengescited by 137
  8. 2022FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settingscited by 121
  9. 2022Federated Learning with Partial Model Personalizationcited by 114
  10. 2021Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learningcited by 111
  11. 2021FedMix: Approximation of Mixup under Mean Augmented Federated Learningcited by 108
  12. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
38 more
  1. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
  2. 2023Deep Model Fusion: A Surveycited by 89
  3. 2021FedCM: Federated Learning with Client-level Momentumcited by 77
  4. 2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
  5. 2021On Large-Cohort Training for Federated Learningcited by 73
  6. 2022Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learningcited by 71
  7. 2021Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Datacited by 69
  8. 2021What Do We Mean by Generalization in Federated Learning?cited by 62
  9. 2021HarmoFL: Harmonizing Local and Global Drifts in Federated Learning on Heterogeneous Medical Imagescited by 62
  10. 2022FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learningcited by 62
  11. 2021Federated Learning with Fair Averagingcited by 59
  12. 2021FedLab: A Flexible Federated Learning Frameworkcited by 54
  13. 2021Adapt to Adaptation: Learning Personalization for Cross-Silo Federated Learningcited by 53
  14. 2021Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learningcited by 52
  15. 2021Consensus Control for Decentralized Deep Learningcited by 49
  16. 2022FedFA: Federated Learning with Feature Anchors to Align Features and Classifiers for Heterogeneous Datacited by 49
  17. 2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
  18. 2023Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscapecited by 46
  19. 2023Federated Conformal Predictors for Distributed Uncertainty Quantificationcited by 46
  20. 2021SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Trainingcited by 45
  21. 2022Orchestra: Unsupervised Federated Learning via Globally Consistent Clusteringcited by 45
  22. 2021Federated Learning Meets Natural Language Processing: A Surveycited by 42
  23. 2022Closing the Generalization Gap of Cross-silo Federated Medical Image Segmentationcited by 41
  24. 2022Label-Efficient Self-Supervised Federated Learning for Tackling Data Heterogeneity in Medical Imagingcited by 39
  25. 2022Test-Time Robust Personalization for Federated Learningcited by 39
  26. 2022Gradient Obfuscation Gives a False Sense of Security in Federated Learningcited by 37
  27. 2022Multi-Level Branched Regularization for Federated Learningcited by 34
  28. 2021Cross-Gradient Aggregation for Decentralized Learning from Non-IID datacited by 32
  29. 2023Decentralized Gradient Tracking with Local Stepscited by 32
  30. 2024Relaxed Contrastive Learning for Federated Learningcited by 32
  31. 2021Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Datacited by 31
  32. 2022Refined Convergence and Topology Learning for Decentralized SGD with Heterogeneous Datacited by 31
  33. 2022Achieving Personalized Federated Learning with Sparse Local Modelscited by 30
  34. 2022Federated Learning Under Intermittent Client Availability and Time-Varying Communication Constraintscited by 30
  35. 2021SemiFed: Semi-supervised Federated Learning with Consistency and Pseudo-Labelingcited by 29
  36. 2022Cocktail Party Attack: Breaking Aggregation-Based Privacy in Federated Learning using Independent Component Analysiscited by 29
  37. 2023Why Batch Normalization Damage Federated Learning on Non-IID Data?cited by 29
  38. 2022FedSpace: An Efficient Federated Learning Framework at Satellites and Ground Stationscited by 28

50 most cited of 303 All 303 (JSON)