mlsanityData to 14 Oct 2026

Federated Learning on Non-iid Data via Local and Global Distillation

arXiv 2306.14443 ↗First posted 26 Jun 2023cs.LG cs.CRcited by 4

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

Most existing federated learning algorithms are based on the vanilla FedAvg scheme. However, with the increase of data complexity and the number of model parameters, the amount of communication traffic and the number of iteration rounds for training such algorithms increases significantly, especially in non-independently and homogeneously distributed scenarios, where they do not achieve satisfactory performance. In this work, we propose FedND: federated learning with noise distillation. The main idea is to use knowledge distillation to optimize the model training process. In the client, we propose a self-distillation method to train the local model. In the server, we generate noisy samples for each client and use them to distill other clients. Finally, the global model is obtained by the aggregation of local models. Experimental results show that the algorithm achieves the best performance and is more communication-efficient than state-of-the-art methods.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 43 of its 56 references that are in the map. First posted on arXiv 26 Jun 2023.

Builds on

43 of 56 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022FedDTG:Federated Data-Free Knowledge Distillation via Three-Player Generative Adversarial Networks10 Jan
  2. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillation30 Aug
  3. 2021Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data30 Jun
  4. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learning20 May

Other references · 39

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  3. 2019Advances and Open Problems in Federated Learningcited by 3,046
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  6. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  7. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  8. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  9. 2018Federated Learning with Non-IID Datacited by 1,108
  10. 2020Adaptive Federated Optimizationcited by 1,046
  11. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  12. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
27 more
  1. 2021Model-Contrastive Federated Learningcited by 766
  2. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  3. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
  4. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  5. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 564
  6. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  7. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  8. 2021Federated Learning on Non-IID Data: A Surveycited by 346
  9. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 243
  10. 2020Federated learning with hierarchical clustering of local updates to improve training on non-IID datacited by 223
  11. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 175
  12. 2020Federated Learning for Internet of Things: Recent Advances, Taxonomy, and Open Challengescited by 139
  13. 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
  14. 2020FedVision: An Online Visual Object Detection Platform Powered by Federated Learningcited by 124
  15. 2020TiFL: A Tier-based Federated Learning Systemcited by 96
  16. 2019Federated Learning over Wireless Networks: Convergence Analysis and Resource Allocationcited by 94
  17. 2021Clustered Sampling: Low-Variance and Improved Representativity for Clients Selection in Federated Learningcited by 93
  18. 2020Federated Accelerated Stochastic Gradient Descentcited by 85
  19. 2021Multi-institutional Collaborations for Improving Deep Learning-based Magnetic Resonance Image Reconstruction Using Federated Learningcited by 78
  20. 2021FedCM: Federated Learning with Client-level Momentumcited by 77
  21. 2020Vertically Federated Graph Neural Network for Privacy-Preserving Node Classificationcited by 61
  22. 2021FedFace: Collaborative Learning of Face Recognition Modelcited by 26
  23. 2021Stronger Privacy for Federated Collaborative Filtering with Implicit Feedbackcited by 26
  24. 2022Differential Private Knowledge Transfer for Privacy-Preserving Cross-Domain Recommendationcited by 22
  25. 2022Exploiting Data Sparsity in Secure Cross-Platform Social Recommendationcited by 17
  26. 2021FedHe: Heterogeneous Models and Communication-Efficient Federated Learningcited by 13
  27. 2022FedQAS: Privacy-aware machine reading comprehension with federated learningcited by 3

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 1.

4 papers.

  1. 2024Federated Distillation: A Surveycited by 15
  2. 2024Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directionscited by 13
  3. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutionscited by 9
  4. 2025FedSKC: Federated Learning with Non-IID Data via Structural Knowledge Collaborationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Data
  2. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning
  3. 2022Knowledge Distillation for Federated Learning: a Practical Guide
  4. 2022FedDKD: Federated Learning with Decentralized Knowledge Distillation
  5. 2023FedSDD: Scalable and Diversity-enhanced Distillation for Model Aggregation in Federated Learning
  6. 2024Training Heterogeneous Client Models using Knowledge Distillation in Serverless Federated Learning
  7. 2023The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillation
  8. 2022Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusion