mlsanityData to 7 Oct 2026

FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Models

arXiv 2109.03775 ↗First posted 8 Sep 2021cs.LG stat.MLcited by 31

Not on the map. Why?

Abstract (arXiv)

Federated learning enables multiple distributed devices to collaboratively learn a shared prediction model without centralizing their on-device data. Most of the current algorithms require comparable individual efforts for local training with the same structure and size of on-device models, which, however, impedes participation from resource-constrained devices. Given the widespread yet heterogeneous devices nowadays, in this paper, we propose an innovative federated learning framework with heterogeneous on-device models through Zero-shot Knowledge Transfer, named by FedZKT. Specifically, FedZKT allows devices to independently determine the on-device models upon their local resources. To achieve knowledge transfer across these heterogeneous on-device models, a zero-shot distillation approach is designed without any prerequisites for private on-device data, which is contrary to certain prior research based on a public dataset or a pre-trained data generator. Moreover, this compute-intensive distillation task is assigned to the server to allow the participation of resource-constrained devices, where a generator is adversarially learned with the ensemble of collected on-device models. The distilled central knowledge is then sent back in the form of the corresponding on-device model parameters, which can be easily absorbed on the device side. Extensive experimental studies demonstrate the effectiveness and robustness of FedZKT towards on-device knowledge agnostic, on-device model heterogeneity, and other challenging federated learning scenarios, such as heterogeneous on-device data and straggler effects.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 39 of its 47 references that are in the map. First posted on arXiv 8 Sep 2021.

Builds on

39 of 47 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  7. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  8. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  9. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  10. 2018Federated Learning with Non-IID Datacited by 1,099
  11. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  12. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
27 more
  1. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  2. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  3. 2013Do Deep Nets Really Need to be Deep?cited by 466
  4. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  5. 2018Deep Learning for Classical Japanese Literaturecited by 415
  6. 2019Federated Machine Learning: Concept and Applicationscited by 409
  7. 2018Split learning for health: Distributed deep learning without sharing raw patient datacited by 391
  8. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  9. 2019On the Efficacy of Knowledge Distillationcited by 377
  10. 2018Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edgecited by 347
  11. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
  12. 2020MCUNet: Tiny Deep Learning on IoT Devicescited by 269
  13. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 242
  14. 2019Zero-shot Knowledge Transfer via Adversarial Belief Matchingcited by 129
  15. 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
  16. 2019One-Shot Federated Learningcited by 124
  17. 2020Data-Free Model Extractioncited by 121
  18. 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 91
  19. 2019Data-Free Adversarial Distillationcited by 90
  20. 2020Data-Free Network Quantization With Adversarial Knowledge Distillationcited by 79
  21. 2019Accelerating Federated Learning via Momentum Gradient Descentcited by 75
  22. 2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 67
  23. 2020Federated Knowledge Distillationcited by 51
  24. 2020Federated Model Distillation with Noise-Free Differential Privacycited by 40
  25. 2020Communication-Efficient Federated Distillationcited by 28
  26. 2021FedH2L: Federated Learning with Model and Statistical Heterogeneitycited by 17
  27. 2021Age-Stratified COVID-19 Spread Analysis and Vaccination: A Multitype Random Network Approachcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 10; 2024: 7; 2025: 7; 2026: 3.

31 papers.

Sort:

By area:

  1. 2023FedGH: Heterogeneous Federated Learning with Generalized Global Headercited by 56
  2. 2023pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuningcited by 38
  3. 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
  4. 2023Federated Domain Generalization: A Surveycited by 25
  5. 2024pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learningcited by 21
  6. 2024Federated Model Heterogeneous Matryoshka Representation Learningcited by 21
  7. 2024Federated Distillation: A Surveycited by 14
  8. 2023FedSSA: Semantic Similarity-based Aggregation for Efficient Model-Heterogeneous Personalized Federated Learningcited by 10
  9. 2024Breaking the Memory Wall for Heterogeneous Federated Learning via Progressive Trainingcited by 10
  10. 2022Completely Heterogeneous Federated Learningcited by 9
  11. 2023pFedES: Model Heterogeneous Personalized Federated Learning with Feature Extractor Sharingcited by 9
  12. 2024A Comprehensive Survey of Federated Transfer Learning: Challenges, Methods and Applicationscited by 8
19 more
  1. 2023Federated Learning via Input-Output Collaborative Distillationcited by 6
  2. 2023Knowledge Distillation in Federated Edge Learning: A Surveycited by 5
  3. 2023FedAL: Black-Box Federated Knowledge Distillation Enabled by Adversarial Learningcited by 5
  4. 2023Towards Open Federated Learning Platforms: Survey and Vision from Technical and Legal Perspectivescited by 3
  5. 2024FedD2S: Personalized Data-Free Federated Knowledge Distillationcited by 3
  6. 2024pFedAFM: Adaptive Feature Mixture for Batch-Level Personalization in Heterogeneous Federated Learningcited by 3
  7. 2022Enhancing Efficiency in Multidevice Federated Learning through Data Selectioncited by 2
  8. 2025pMixFed: Efficient Personalized Federated Learning through Adaptive Layer-Wise Mixupcited by 2
  9. 2025Corrected with the Latest Version: Make Robust Asynchronous Federated Learning Possiblecited by 2
  10. 2025FedPromo: Federated Lightweight Proxy Models at the Edge Bring New Domains to Foundation Modelscited by 2
  11. 2023Federated Learning in IoT: a Survey from a Resource-Constrained Perspectivecited by 1
  12. 2025Data-Free Black-Box Federated Learning via Zeroth-Order Gradient Estimationcited by 1
  13. 2025Hypernetworks for Model-Heterogeneous Personalized Federated Learningcited by 1
  14. 2025Bridging Generalization Gap of Heterogeneous Federated Clients Using Generative Modelscited by 1
  15. 2022Exploring Semantic Attributes from A Foundation Model for Federated Learning of Disjoint Label Spacescited by 0
  16. 2025Provably Near-Optimal Federated Ensemble Distillation with Negligible Overheadcited by 0
  17. 2026Optimized Federated Knowledge Distillation with Distributed Neural Architecture Searchcited by 0
  18. 2026EFFEKT: Efficient Federated Knowledge Transfer to Foundation Modelscited by 0
  19. 2026Out-of-Distribution Federated Distillation with Domain-Aware Proxycited by 0