mlsanityData to 14 Oct 2026

Tackling Data Heterogeneity in Federated Learning through Knowledge Distillation with Inequitable Aggregation

arXiv 2506.20431 ↗First posted 25 Jun 2025cs.LGcited by 1

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

Federated learning aims to train a global model in a distributed environment that is close to the performance of centralized training. However, issues such as client label skew, data quantity skew, and other heterogeneity problems severely degrade the model's performance. Most existing methods overlook the scenario where only a small portion of clients participate in training within a large-scale client setting, whereas our experiments show that this scenario presents a more challenging federated learning task. Therefore, we propose a Knowledge Distillation with teacher-student Inequitable Aggregation (KDIA) strategy tailored to address the federated learning setting mentioned above, which can effectively leverage knowledge from all clients. In KDIA, the student model is the average aggregation of the participating clients, while the teacher model is formed by a weighted aggregation of all clients based on three frequencies: participation intervals, participation counts, and data volume proportions. During local training, self-knowledge distillation is performed. Additionally, we utilize a generator trained on the server to generate approximately independent and identically distributed (IID) data features locally for auxiliary training. We conduct extensive experiments on the CIFAR-10/100/CINIC-10 datasets and various heterogeneous settings to evaluate KDIA. The results show that KDIA can achieve better accuracy with fewer rounds of training, and the improvement is more significant under severe heterogeneity.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 29 of its 42 references that are in the map. First posted on arXiv 25 Jun 2025.

Builds on

29 of 42 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions16 Jun
  2. 2023FedFed: Feature Distillation against Data Heterogeneity in Federated Learning8 Oct
  3. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning17 Mar
  4. 2021Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data30 Jun
  5. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learning20 May

Other references · 24

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2019Advances and Open Problems in Federated Learningcited by 3,046
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  5. 2014Conditional Generative Adversarial Netscited by 2,162
  6. 2020Knowledge Distillation: A Surveycited by 1,603
  7. 2018Federated Learning with Non-IID Datacited by 1,108
  8. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  9. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  10. 2021Model-Contrastive Federated Learningcited by 766
  11. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  12. 2020Federated Learning with Matched Averagingcited by 564
12 more
  1. 2018Born Again Neural Networkscited by 546
  2. 2021Federated Learning on Non-IID Data: A Surveycited by 346
  3. 2022FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correctioncited by 181
  4. 2018Snapshot Distillation: Teacher-Student Optimization in One Generationcited by 99
  5. 2023FedDisco: Federated Learning with Discrepancy-Aware Collaborationcited by 67
  6. 2022FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learningcited by 66
  7. 2022Client Selection in Federated Learning: Principles, Challenges, and Opportunitiescited by 62
  8. 2020Federated Knowledge Distillationcited by 51
  9. 2022Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learningcited by 46
  10. 2020Communication-Efficient Federated Learning via Optimal Client Samplingcited by 39
  11. 2021FedCG: Leverage Conditional GAN for Protecting Privacy and Maintaining Competitive Performance in Federated Learningcited by 39
  12. 2023Fake It Till Make It: Federated Learning with Consensus-Oriented Generationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Enhancing Federated Quadruplet Learning: Stochastic Client Selection and Embedding Stability Analysiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023FedSDD: Scalable and Diversity-enhanced Distillation for Model Aggregation in Federated Learning
  2. 2022Closing the Gap between Client and Global Model Performance in Heterogeneous Federated Learning
  3. 2022Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusion
  4. 2026CRAD: Class-wise Reliability-Aware Distillation for Decentralized Heterogeneous Federated Learning
  5. 2021Towards Model Agnostic Federated Learning Using Knowledge Distillation
  6. 2024Training Heterogeneous Client Models using Knowledge Distillation in Serverless Federated Learning
  7. 2025Enabling Weak Client Participation via On-device Knowledge Distillation in Heterogeneous Federated Learning
  8. 2025SFedKD: Sequential Federated Learning with Discrepancy-Aware Multi-Teacher Knowledge Distillation