mlsanityData to 14 Oct 2026

FedDistill: Global Model Distillation for Local Model De-Biasing in Non-IID Federated Learning

arXiv 2404.09210 ↗First posted 14 Apr 2024cs.LG cs.AI cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) is a novel approach that allows for collaborative machine learning while preserving data privacy by leveraging models trained on decentralized devices. However, FL faces challenges due to non-uniformly distributed (non-iid) data across clients, which impacts model performance and its generalization capabilities. To tackle the non-iid issue, recent efforts have utilized the global model as a teaching mechanism for local models. However, our pilot study shows that their effectiveness is constrained by imbalanced data distribution, which induces biases in local models and leads to a 'local forgetting' phenomenon, where the ability of models to generalize degrades over time, particularly for underrepresented classes. This paper introduces FedDistill, a framework enhancing the knowledge transfer from the global model to local models, focusing on the issue of imbalanced class distribution. Specifically, FedDistill employs group distillation, segmenting classes based on their frequency in local datasets to facilitate a focused distillation process to classes with fewer samples. Additionally, FedDistill dissects the global model into a feature extractor and a classifier. This separation empowers local models with more generalized data representation capabilities and ensures more accurate classification across all classes. FedDistill mitigates the adverse effects of data imbalance, ensuring that local models do not forget underrepresented classes but instead become more adept at recognizing and classifying them accurately. Our comprehensive experiments demonstrate FedDistill's effectiveness, surpassing existing baselines in accuracy and convergence speed across several benchmark datasets.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 28 of its 31 references that are in the map. First posted on arXiv 14 Apr 2024.

Builds on

28 of 31 references.

References · 28

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  4. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,245
  5. 2018Federated Learning with Non-IID Datacited by 1,108
  6. 2021Model-Contrastive Federated Learningcited by 766
  7. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 742
  8. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  9. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  10. 2020Federated Learning with Matched Averagingcited by 564
  11. 2021Deep Long-Tailed Learning: A Surveycited by 417
  12. 2022Decoupled Knowledge Distillationcited by 388
16 more
  1. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  2. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 222
  3. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  4. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  5. 2019Overcoming Forgetting in Federated Learning on Non-IID Datacited by 120
  6. 2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114
  7. 2019Data-Free Adversarial Distillationcited by 90
  8. 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
  9. 2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 69
  10. 2022Acceleration of Federated Learning with Alleviated Forgetting in Local Trainingcited by 29
  11. 2022Federated Learning with GAN-based Data Synthesis for Non-IID Clientscited by 25
  12. 2022A Survey on Federated Recommendation Systemscited by 25
  13. 2022Revisiting Self-Distillationcited by 24
  14. 2022FEDIC: Federated Learning on Non-IID and Long-Tailed Data via Calibrated Distillationcited by 19
  15. 2023A Survey on Class Imbalance in Federated Learningcited by 13
  16. 2023Re-Weighted Softmax Cross-Entropy to Control Forgetting in Federated Learningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 3.

4 papers.

  1. 2024Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directionscited by 13
  2. 2026FedSDR: Federated Self-Distillation with Rectificationcited by 0
  3. 2026Optimized Federated Knowledge Distillation with Distributed Neural Architecture Searchcited by 0
  4. 2026FedADB: Class Anchor-Driven Dual-Branch Federated Learning for Mitigating Forgettingcited by 0