mlsanityData to 14 Oct 2026

Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data

arXiv 2107.00051 ↗First posted 30 Jun 2021cs.LG cs.AI cs.DCcited by 31

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

Federated learning enables multiple clients to collaboratively learn a global model by periodically aggregating the clients' models without transferring the local data. However, due to the heterogeneity of the system and data, many approaches suffer from the "client-drift" issue that could significantly slow down the convergence of the global model training. As clients perform local updates on heterogeneous data through heterogeneous systems, their local models drift apart. To tackle this issue, one intuitive idea is to guide the local model training by the global teachers, i.e., past global models, where each client learns the global knowledge from past global models via adaptive knowledge distillation techniques. Coming from these insights, we propose a novel approach for heterogeneous federated learning, namely FedGKD, which fuses the knowledge from historical global models for local training to alleviate the "client-drift" issue. In this paper, we evaluate FedGKD with extensive experiments on various CV/NLP datasets (i.e., CIFAR-10/100, Tiny-ImageNet, AG News, SST5) and different heterogeneous settings. The proposed method is guaranteed to converge under common assumptions, and achieves superior empirical accuracy in fewer communication runs than five state-of-the-art methods.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 24 of its 37 references that are in the map. First posted on arXiv 30 Jun 2021.

Builds on

24 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learning20 May

Other references · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2019Advances and Open Problems in Federated Learningcited by 3,046
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  8. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  9. 2018Federated Learning with Non-IID Datacited by 1,108
  10. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  11. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  12. 2021Model-Contrastive Federated Learningcited by 766
11 more
  1. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  2. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 564
  3. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  4. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 415
  5. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 305
  6. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 175
  7. 2020Federated Continual Learning with Weighted Inter-client Transfercited by 161
  8. 2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 69
  9. 2020Federated Knowledge Distillationcited by 51
  10. 2020Communication-Efficient Federated Distillationcited by 28
  11. 2020Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillationcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 6; 2023: 10; 2024: 8; 2025: 5; 2026: 1.

31 papers.

Sort:

By area:

  1. 2022Multi-Level Branched Regularization for Federated Learningcited by 34
  2. 2024Relaxed Contrastive Learning for Federated Learningcited by 32
  3. 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
  4. 2023GradMA: A Gradient-Memory-based Accelerated Federated Learning with Alleviated Catastrophic Forgettingcited by 27
  5. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  6. 2022Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillationcited by 22
  7. 2023FedICT: Federated Multi-task Distillation for Multi-access Edge Computingcited by 17
  8. 2024Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directionscited by 13
  9. 2024Advances in Robust Federated Learning: A Survey with Heterogeneity Considerationscited by 12
  10. 2022FedDKD: Federated Learning with Decentralized Knowledge Distillationcited by 9
  11. 2023Re-Weighted Softmax Cross-Entropy to Control Forgetting in Federated Learningcited by 9
  12. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutionscited by 9
19 more
  1. 2024Federated Learning with Label-Masking Distillationcited by 7
  2. 2023FedAL: Black-Box Federated Knowledge Distillation Enabled by Adversarial Learningcited by 5
  3. 2023Federated Learning on Non-iid Data via Local and Global Distillationcited by 4
  4. 2023Rethinking Client Drift in Federated Learning: A Logit Perspectivecited by 4
  5. 2023UNIDEAL: Curriculum Knowledge Distillation Federated Learningcited by 4
  6. 2025Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Datacited by 4
  7. 2026Federated Modality-specific Encoders and Partially Personalized Fusion Decoder for Multimodal Brain Tumor Segmentationcited by 3
  8. 2021Aggregation Delayed Federated Learningcited by 2
  9. 2024FedQUIT: On-Device Federated Unlearning via a Quasi-Competent Virtual Teachercited by 2
  10. 2025FedPromo: Federated Lightweight Proxy Models at the Edge Bring New Domains to Foundation Modelscited by 2
  11. 2022Domain Discrepancy Aware Distillation for Model Aggregation in Federated Learningcited by 1
  12. 2023FedSDD: Scalable and Diversity-enhanced Distillation for Model Aggregation in Federated Learningcited by 1
  13. 2025Federated learning in low-resource settings: A chest imaging study in Africa -- Challenges and lessons learnedcited by 1
  14. 2025Tackling Data Heterogeneity in Federated Learning through Knowledge Distillation with Inequitable Aggregationcited by 1
  15. 2022Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusioncited by 0
  16. 2023Heterogeneous Federated Learning via Personalized Generative Networkscited by 0
  17. 2024FedSiKD: Clients Similarity and Knowledge Distillation: Addressing Non-i.i.d. and Constraints in Federated Learningcited by 0
  18. 2024Tailored Federated Learning: Leveraging Direction Regulation & Knowledge Distillationcited by 0
  19. 2025Provably Near-Optimal Federated Ensemble Distillation with Negligible Overheadcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Closing the Gap between Client and Global Model Performance in Heterogeneous Federated Learning
  2. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning
  3. 2023Heterogeneous Federated Learning Using Knowledge Codistillation
  4. 2023The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillation
  5. 2024Federated Distillation: A Survey
  6. 2021Towards Model Agnostic Federated Learning Using Knowledge Distillation
  7. 2025SFedKD: Sequential Federated Learning with Discrepancy-Aware Multi-Teacher Knowledge Distillation
  8. 2024Federated Learning with a Single Shared Image