mlsanityData to 14 Oct 2026

Rethinking Data Heterogeneity in Federated Learning: Introducing a New Notion and Standard Benchmarks

arXiv 2209.15595 ↗First posted 30 Sep 2022cs.LG cs.AIcited by 24

Not on the map. Why?

Abstract (arXiv)

Though successful, federated learning presents new challenges for machine learning, especially when the issue of data heterogeneity, also known as Non-IID data, arises. To cope with the statistical heterogeneity, previous works incorporated a proximal term in local optimization or modified the model aggregation scheme at the server side or advocated clustered federated learning approaches where the central server groups agent population into clusters with jointly trainable data distributions to take the advantage of a certain level of personalization. While effective, they lack a deep elaboration on what kind of data heterogeneity and how the data heterogeneity impacts the accuracy performance of the participating clients. In contrast to many of the prior federated learning approaches, we demonstrate not only the issue of data heterogeneity in current setups is not necessarily a problem but also in fact it can be beneficial for the FL participants. Our observations are intuitive: (1) Dissimilar labels of clients (label skew) are not necessarily considered data heterogeneity, and (2) the principal angle between the agents' data subspaces spanned by their corresponding principal vectors of data is a better estimate of the data heterogeneity. Our code is available at https://github.com/MMorafah/FL-SC-NIID.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 28 of its 45 references that are in the map. First posted on arXiv 30 Sep 2022.

Builds on

28 of 45 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  4. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  5. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  6. 2018Federated Learning with Non-IID Datacited by 1,108
  7. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  8. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  9. 2021Model-Contrastive Federated Learningcited by 766
  10. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  11. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  12. 2020Federated Learning with Matched Averagingcited by 564
16 more
  1. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  2. 2020An Efficient Framework for Clustered Federated Learningcited by 511
  3. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 503
  4. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 387
  5. 2020Adaptive Personalized Federated Learningcited by 358
  6. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 325
  7. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 325
  8. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 243
  9. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  10. 2021Personalized Federated Learning using Hypernetworkscited by 212
  11. 2021Heterogeneity for the Win: One-Shot Federated Clusteringcited by 87
  12. 2021Towards Fair Federated Learning with Zero-Shot Data Augmentationcited by 64
  13. 2020Federated Learning via Synthetic Datacited by 54
  14. 2022Efficient Distribution Similarity Identification in Clustered Federated Learning via Principal Angles Between Client Data Subspacescited by 36
  15. 2021Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneitycited by 27
  16. 2022FLIS: Clustered Federated Learning via Inference Similarity for Non-IID Data Distributioncited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 4; 2024: 8; 2025: 8; 2026: 2.

24 papers.

Sort:

By area:

  1. 2024FedMM: Federated Multi-Modal Learning with Modality Heterogeneity in Computational Pathologycited by 17
  2. 2025Federated Continual Learning: Concepts, Challenges, and Solutionscited by 13
  3. 2022When Do Curricula Work in Federated Learning?cited by 9
  4. 2022FedCC: Robust Federated Learning against Model Poisoning Attackscited by 5
  5. 2025A Thorough Assessment of the Non-IID Data Impact in Federated Learningcited by 5
  6. 2023PS-FedGAN: An Efficient Federated Learning Framework Based on Partially Shared Generative Adversarial Networks For Data Privacycited by 4
  7. 2025Homophily Heterogeneity Matters in Graph Federated Learning: A Spectrum Sharing and Complementing Perspectivecited by 4
  8. 2023FedBug: A Bottom-Up Gradual Unfreezing Framework for Federated Learningcited by 3
  9. 2023PFL-GAN: When Client Heterogeneity Meets Generative Models in Personalized Federated Learningcited by 3
  10. 2025Impact of Labeling Inaccuracy and Image Noise on Tooth Segmentation in Panoramic Radiographs using Federated, Centralized and Local Learningcited by 3
  11. 2024Federated Learning with Heterogeneous Data Handling for Robust Vehicular Object Detectioncited by 2
  12. 2024Geometrically Inspired Kernel Machines for Collaborative Learning Beyond Gradient Descentcited by 2
12 more
  1. 2025Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environmentscited by 2
  2. 2024Rethinking the Starting Point: Collaborative Pre-Training for Federated Downstream Taskscited by 1
  3. 2024Adaptive Coded Federated Learning: Privacy Preservation and Straggler Mitigationcited by 1
  4. 2025FedTeddi: Temporal Drift and Divergence Aware Scheduling for Timely Federated Edge Learningcited by 1
  5. 2023FedSym: Unleashing the Power of Entropy for Benchmarking the Algorithms for Federated Learningcited by 0
  6. 2024Empowering HWNs with Efficient Data Labeling: A Clustered Federated Semi-Supervised Learning Approachcited by 0
  7. 2024Optimisation of federated learning settings under statistical heterogeneity variationscited by 0
  8. 2024BACSA: A Bias-Aware Client Selection Algorithm for Privacy-Preserving Federated Learning in Wireless Healthcare Networkscited by 0
  9. 2025PA-CFL: Privacy-Adaptive Clustered Federated Learning for Transformer-Based Sales Forecasting on Heterogeneous Retail Datacited by 0
  10. 2025Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learningcited by 0
  11. 2026An Efficient Unsupervised Federated Learning Approach for Anomaly Detection in Heterogeneous IoT Networkscited by 0
  12. 2026Enhancing Federated Quadruplet Learning: Stochastic Client Selection and Embedding Stability Analysiscited by 0