# Rethinking Data Heterogeneity in Federated Learning: Introducing a New Notion and Standard Benchmarks

arXiv: https://arxiv.org/abs/2209.15595 · HTML: https://mlsanity.com/paper/2209.15595/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 30 Sep 2022.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Federated Learning on Non-IID Data Silos: An Experimental Study](https://mlsanity.com/paper/2102.02079/) (2021)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Federated Learning Based on Dynamic Regularization](https://mlsanity.com/paper/2111.04263/) (2021)
- [An Efficient Framework for Clustered Federated Learning](https://mlsanity.com/paper/2006.04088/) (2020)
- [Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints](https://mlsanity.com/paper/1910.01991/) (2019)
- [Bayesian Nonparametric Federated Learning of Neural Networks](https://mlsanity.com/paper/1905.12022/) (2019)
- [Adaptive Personalized Federated Learning](https://mlsanity.com/paper/2003.13461/) (2020)
- [Think Locally, Act Globally: Federated Learning with Local and Global Representations](https://mlsanity.com/paper/2001.01523/) (2020)
- [Three Approaches for Personalization with Applications to Federated Learning](https://mlsanity.com/paper/2002.10619/) (2020)
- [Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data](https://mlsanity.com/paper/1811.11479/) (2018)
- [No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data](https://mlsanity.com/paper/2106.05001/) (2021)
- [Personalized Federated Learning using Hypernetworks](https://mlsanity.com/paper/2103.04628/) (2021)
- [Heterogeneity for the Win: One-Shot Federated Clustering](https://mlsanity.com/paper/2103.00697/) (2021)
- [Towards Fair Federated Learning with Zero-Shot Data Augmentation](https://mlsanity.com/paper/2104.13417/) (2021)
- [Federated Learning via Synthetic Data](https://mlsanity.com/paper/2008.04489/) (2020)
- [Efficient Distribution Similarity Identification in Clustered Federated Learning via Principal Angles Between Client Data Subspaces](https://mlsanity.com/paper/2209.10526/) (2022)
- [Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneity](https://mlsanity.com/paper/2105.00562/) (2021)
- [FLIS: Clustered Federated Learning via Inference Similarity for Non-IID Data Distribution](https://mlsanity.com/paper/2208.09754/) (2022)

## Built on by (most cited first, 24 of 24)

- [FedMM: Federated Multi-Modal Learning with Modality Heterogeneity in Computational Pathology](https://mlsanity.com/paper/2402.15858/) (2024)
- [Federated Continual Learning: Concepts, Challenges, and Solutions](https://mlsanity.com/paper/2502.07059/) (2025)
- [When Do Curricula Work in Federated Learning?](https://mlsanity.com/paper/2212.12712/) (2022)
- [FedCC: Robust Federated Learning against Model Poisoning Attacks](https://mlsanity.com/paper/2212.01976/) (2022)
- [A Thorough Assessment of the Non-IID Data Impact in Federated Learning](https://mlsanity.com/paper/2503.17070/) (2025)
- [PS-FedGAN: An Efficient Federated Learning Framework Based on Partially Shared Generative Adversarial Networks For Data Privacy](https://mlsanity.com/paper/2305.11437/) (2023)
- [Homophily Heterogeneity Matters in Graph Federated Learning: A Spectrum Sharing and Complementing Perspective](https://mlsanity.com/paper/2502.13732/) (2025)
- [FedBug: A Bottom-Up Gradual Unfreezing Framework for Federated Learning](https://mlsanity.com/paper/2307.10317/) (2023)
- [PFL-GAN: When Client Heterogeneity Meets Generative Models in Personalized Federated Learning](https://mlsanity.com/paper/2308.12454/) (2023)
- [Impact of Labeling Inaccuracy and Image Noise on Tooth Segmentation in Panoramic Radiographs using Federated, Centralized and Local Learning](https://mlsanity.com/paper/2509.06553/) (2025)
- [Federated Learning with Heterogeneous Data Handling for Robust Vehicular Object Detection](https://mlsanity.com/paper/2405.01108/) (2024)
- [Geometrically Inspired Kernel Machines for Collaborative Learning Beyond Gradient Descent](https://mlsanity.com/paper/2407.04335/) (2024)
- [Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environments](https://mlsanity.com/paper/2505.19699/) (2025)
- [Rethinking the Starting Point: Collaborative Pre-Training for Federated Downstream Tasks](https://mlsanity.com/paper/2402.02225/) (2024)
- [Adaptive Coded Federated Learning: Privacy Preservation and Straggler Mitigation](https://mlsanity.com/paper/2403.14905/) (2024)
- [FedTeddi: Temporal Drift and Divergence Aware Scheduling for Timely Federated Edge Learning](https://mlsanity.com/paper/2509.07342/) (2025)
- [FedSym: Unleashing the Power of Entropy for Benchmarking the Algorithms for Federated Learning](https://mlsanity.com/paper/2310.07807/) (2023)
- [Empowering HWNs with Efficient Data Labeling: A Clustered Federated Semi-Supervised Learning Approach](https://mlsanity.com/paper/2401.10646/) (2024)
- [Optimisation of federated learning settings under statistical heterogeneity variations](https://mlsanity.com/paper/2406.06340/) (2024)
- [BACSA: A Bias-Aware Client Selection Algorithm for Privacy-Preserving Federated Learning in Wireless Healthcare Networks](https://mlsanity.com/paper/2411.01050/) (2024)
- [PA-CFL: Privacy-Adaptive Clustered Federated Learning for Transformer-Based Sales Forecasting on Heterogeneous Retail Data](https://mlsanity.com/paper/2503.12220/) (2025)
- [Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning](https://mlsanity.com/paper/2505.21877/) (2025)
- [An Efficient Unsupervised Federated Learning Approach for Anomaly Detection in Heterogeneous IoT Networks](https://mlsanity.com/paper/2602.24209/) (2026)
- [Enhancing Federated Quadruplet Learning: Stochastic Client Selection and Embedding Stability Analysis](https://mlsanity.com/paper/2605.07888/) (2026)

## Abstract (arXiv)

Though successful, federated learning presents new challenges for machine learning, especially when the issue of data heterogeneity, also known as Non-IID data, arises. To cope with the statistical heterogeneity, previous works incorporated a proximal term in local optimization or modified the model aggregation scheme at the server side or advocated clustered federated learning approaches where the central server groups agent population into clusters with jointly trainable data distributions to take the advantage of a certain level of personalization. While effective, they lack a deep elaboration on what kind of data heterogeneity and how the data heterogeneity impacts the accuracy performance of the participating clients. In contrast to many of the prior federated learning approaches, we demonstrate not only the issue of data heterogeneity in current setups is not necessarily a problem but also in fact it can be beneficial for the FL participants. Our observations are intuitive: (1) Dissimilar labels of clients (label skew) are not necessarily considered data heterogeneity, and (2) the principal angle between the agents' data subspaces spanned by their corresponding principal vectors of data is a better estimate of the data heterogeneity. Our code is available at https://github.com/MMorafah/FL-SC-NIID.