# Benchmarking Data Heterogeneity Evaluation Approaches for Personalized Federated Learning

arXiv: https://arxiv.org/abs/2410.07286 · HTML: https://mlsanity.com/paper/2410.07286/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 66 references that are in the map.
- First posted on arXiv 9 Oct 2024.

## Builds on (most cited first, 30 of 33)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [Data Shapley: Equitable Valuation of Data for Machine Learning](https://mlsanity.com/paper/1904.02868/) (2019)
- [FedBN: Federated Learning on Non-IID Features via Local Batch Normalization](https://mlsanity.com/paper/2102.07623/) (2021)
- [Federated Learning on Non-IID Data Silos: An Experimental Study](https://mlsanity.com/paper/2102.02079/) (2021)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Towards Personalized Federated Learning](https://mlsanity.com/paper/2103.00710/) (2021)
- [Differentially Private Federated Learning: A Client Level Perspective](https://mlsanity.com/paper/1712.07557/) (2017)
- [Bayesian Nonparametric Federated Learning of Neural Networks](https://mlsanity.com/paper/1905.12022/) (2019)
- [Federated Learning on Non-IID Data: A Survey](https://mlsanity.com/paper/2106.06843/) (2021)
- [No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data](https://mlsanity.com/paper/2106.05001/) (2021)
- [Personalized Federated Learning using Hypernetworks](https://mlsanity.com/paper/2103.04628/) (2021)
- [Federated Learning for Internet of Things: Recent Advances, Taxonomy, and Open Challenges](https://mlsanity.com/paper/2009.13012/) (2020)
- [Learning the Pareto Front with Hypernetworks](https://mlsanity.com/paper/2010.04104/) (2020)
- [Multi-Center Federated Learning: Clients Clustering for Better Personalization](https://mlsanity.com/paper/2108.08647/) (2021)
- [Collaborative Fairness in Federated Learning](https://mlsanity.com/paper/2008.12161/) (2020)
- [GTG-Shapley: Efficient and Accurate Participant Contribution Evaluation in Federated Learning](https://mlsanity.com/paper/2109.02053/) (2021)
- [Model-sharing Games: Analyzing Federated Learning Under Voluntary Participation](https://mlsanity.com/paper/2010.00753/) (2020)
- [Federated Learning with Only Positive Labels](https://mlsanity.com/paper/2004.10342/) (2020)
- [Incentive Mechanisms for Federated Learning: From Economic and Game Theoretic Perspective](https://mlsanity.com/paper/2111.11850/) (2021)
- [Optimizing the Collaboration Structure in Cross-Silo Federated Learning](https://mlsanity.com/paper/2306.06508/) (2023)
- [Collaboration Equilibrium in Federated Learning](https://mlsanity.com/paper/2108.07926/) (2021)
- [MarS-FL: Enabling Competitors to Collaborate in Federated Learning](https://mlsanity.com/paper/2110.13464/) (2021)
- [Continuous Transfer Learning with Label-informed Distribution Alignment](https://mlsanity.com/paper/2006.03230/) (2020)
- [FedCompetitors: Harmonious Collaboration in Federated Learning with Competing Participants](https://mlsanity.com/paper/2312.11391/) (2023)

## Built on by (most cited first, 3 of 3)

- [Free-Rider and Conflict Aware Collaboration Formation for Cross-Silo Federated Learning](https://mlsanity.com/paper/2410.19321/) (2024)
- [Voronoi-grid-based Pareto Front Learning and Its Application to Collaborative Federated Learning](https://mlsanity.com/paper/2505.20648/) (2025)
- [FedDAG: Clustered Federated Learning via Global Data and Gradient Integration for Heterogeneous Environments](https://mlsanity.com/paper/2602.23504/) (2026)

## Abstract (arXiv)

There is growing research interest in measuring the statistical heterogeneity of clients' local datasets. Such measurements are used to estimate the suitability for collaborative training of personalized federated learning (PFL) models. Currently, these research endeavors are taking place in silos and there is a lack of a unified benchmark to provide a fair and convenient comparison among various approaches in common settings. We aim to bridge this important gap in this paper. The proposed benchmarking framework currently includes six representative approaches. Extensive experiments have been conducted to compare these approaches under five standard non-IID FL settings, providing much needed insights into which approaches are advantageous under which settings. The proposed framework offers useful guidance on the suitability of various data divergence measures in FL systems. It is beneficial for keeping related research activities on the right track in terms of: (1) designing PFL schemes, (2) selecting appropriate data heterogeneity evaluation approaches for specific FL application scenarios, and (3) addressing fairness issues in collaborative model training. The code is available at https://github.com/Xiaoni-61/DH-Benchmark.