mlsanityData to 7 Oct 2026

Label driven Knowledge Distillation for Federated Learning with non-IID Data

arXiv 2209.14520 ↗First posted 29 Sep 2022cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

In real-world applications, Federated Learning (FL) meets two challenges: (1) scalability, especially when applied to massive IoT networks; and (2) how to be robust against an environment with heterogeneous data. Realizing the first problem, we aim to design a novel FL framework named Full-stack FL (F2L). More specifically, F2L utilizes a hierarchical network architecture, making extending the FL network accessible without reconstructing the whole network system. Moreover, leveraging the advantages of hierarchical network design, we propose a new label-driven knowledge distillation (LKD) technique at the global server to address the second problem. As opposed to current knowledge distillation techniques, LKD is capable of training a student model, which consists of good knowledge from all teachers' models. Therefore, our proposed algorithm can effectively extract the knowledge of the regions' data distribution (i.e., the regional aggregated models) to reduce the divergence between clients' models when operating under the FL system with non-independent identically distributed data. Extensive experiment results reveal that: (i) our F2L method can significantly improve the overall FL efficiency in all global distillations, and (ii) F2L rapidly achieves convergence as global distillation stages occur instead of increasing on each communication cycle.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 43 of its 69 references that are in the map. First posted on arXiv 29 Sep 2022.

Builds on

43 of 69 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  8. 2018Federated Learning with Non-IID Datacited by 1,099
  9. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  10. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  11. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  12. 2021Federated Learning Based on Dynamic Regularizationcited by 555
31 more
  1. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  2. 2017Differentially Private Federated Learning: A Client Level Perspectivecited by 451
  3. 2019Federated Learning in Mobile Edge Networks: A Comprehensive Surveycited by 407
  4. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
  5. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  6. 2020Federated learning with hierarchical clustering of local updates to improve training on non-IID datacited by 222
  7. 2017EMNIST: an extension of MNIST to handwritten letterscited by 212
  8. 2019Client-Edge-Cloud Hierarchical Federated Learningcited by 202
  9. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  10. 2019Energy Efficient Federated Learning Over Wireless Communication Networkscited by 181
  11. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  12. 2020Is Local SGD Better than Minibatch SGD?cited by 151
  13. 2015Adding Gradient Noise Improves Learning for Very Deep Networkscited by 144
  14. 2019Distributed Learning with Compressed Gradient Differencescited by 110
  15. 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 91
  16. 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
  17. 2020Densely Guided Knowledge Distillation using Multiple Teacher Assistantscited by 82
  18. 2020On the Generalization Benefit of Noise in Stochastic Gradient Descentcited by 81
  19. 2021Adaptive Multi-Teacher Multi-level Knowledge Distillationcited by 69
  20. 2019Bad Global Minima Exist and SGD Can Reach Themcited by 49
  21. 2019Ensemble Knowledge Distillation for Learning Improved and Efficient Networkscited by 46
  22. 2021Confidence-Aware Multi-Teacher Knowledge Distillationcited by 39
  23. 2020Hydra: Preserving Ensemble Diversity for Model Distillationcited by 31
  24. 2019Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamationcited by 23
  25. 2019A Survey of Multi-Access Edge Computing in 5G and Beyond: Fundamentals, Technology Integration, and State-of-the-Artcited by 18
  26. 2016Data Collection and Wireless Communication in Internet of Things (IoT) Using Economic Analysis and Pricing Models: A Surveycited by 12
  27. 2022HCFL: A High Compression Approach for Communication-Efficient Federated Learning in Very Large Scale IoT Networkscited by 12
  28. 2022On the Generalization of Wasserstein Robust Federated Learningcited by 9
  29. 2021Enabling Large-Scale Federated Learning over Wireless Edge Networkscited by 3
  30. 2021Federated Learning Framework with Straggling Mitigation and Privacy-Awareness for AI-based Mobile Application Servicescited by 2
  31. 2022Resource Allocation for Compression-aided Federated Learning with High Distortion Ratecited by 2

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2.

3 papers.

  1. 2024Applications of Generative AI (GAI) for Mobile and Wireless Networking: A Surveycited by 9
  2. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutionscited by 8
  3. 2023Energy-Efficient and Real-Time Sensing for Federated Continual Learning via Sample-Driven Controlcited by 1