mlsanityData to 14 Oct 2026

CDKT-FL: Cross-Device Knowledge Transfer using Proxy Dataset in Federated Learning

arXiv 2204.01542 ↗First posted 4 Apr 2022cs.LG cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

In a practical setting, how to enable robust Federated Learning (FL) systems, both in terms of generalization and personalization abilities, is one important research question. It is a challenging issue due to the consequences of non-i.i.d. properties of client's data, often referred to as statistical heterogeneity, and small local data samples from the various data distributions. Therefore, to develop robust generalized global and personalized models, conventional FL methods need to redesign the knowledge aggregation from biased local models while considering huge divergence of learning parameters due to skewed client data. In this work, we demonstrate that the knowledge transfer mechanism achieves these objectives and develop a novel knowledge distillation-based approach to study the extent of knowledge transfer between the global model and local models. Henceforth, our method considers the suitability of transferring the outcome distribution and (or) the embedding vector of representation from trained models during cross-device knowledge transfer using a small proxy dataset in heterogeneous FL. In doing so, we alternatively perform cross-device knowledge transfer following general formulations as 1) global knowledge transfer and 2) on-device knowledge transfer. Through simulations on three federated datasets, we show the proposed method achieves significant speedups and high personalized performance of local models. Furthermore, the proposed approach offers a more stable algorithm than other baselines during the training, with minimal communication data load when exchanging the trained model's outcomes and representation.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 33 of its 56 references that are in the map. First posted on arXiv 4 Apr 2022.

Builds on

33 of 56 references.

References · 33

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  5. 2019Advances and Open Problems in Federated Learningcited by 3,046
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  7. 2020Adaptive Federated Optimizationcited by 1,046
  8. 2021Model-Contrastive Federated Learningcited by 766
  9. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  10. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
  11. 2017Deep Mutual Learningcited by 628
  12. 2020Federated Learning with Matched Averagingcited by 564
21 more
  1. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  2. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  3. 2013Do Deep Nets Really Need to be Deep?cited by 474
  4. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  5. 2019Federated Learning in Mobile Edge Networks: A Comprehensive Surveycited by 408
  6. 2018Split learning for health: Distributed deep learning without sharing raw patient datacited by 395
  7. 2020SplitFed: When Federated Learning Meets Split Learningcited by 310
  8. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 243
  9. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 182
  10. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  11. 2021Parameterized Knowledge Transfer for Personalized Federated Learningcited by 109
  12. 2019Detailed comparison of communication efficiency of split learning and federated learningcited by 77
  13. 2017Artificial Intelligence and Statisticscited by 43
  14. 2021Federated Learning with Domain Generalizationcited by 41
  15. 2017CM-GANs: Cross-modal Generative Adversarial Networks for Common Representation Learningcited by 27
  16. 2020Heterogeneous Federated Learningcited by 19
  17. 2022Feature Distribution Matching for Federated Domain Generalizationcited by 18
  18. 2019Robust Federated Learning Through Representation Matching and Adaptive Hyper-parameterscited by 16
  19. 2020Self-organizing Democratized Learning: Towards Large-scale Distributed Learning Systemscited by 14
  20. 2020Distributed and Democratized Learning: Philosophy and Research Challengescited by 6
  21. 2017Cross-modal Common Representation Learning by Hybrid Transfer Networkcited by 5

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 3.

6 papers.

  1. 2023FedMEKT: Distillation-based Embedding Knowledge Transfer for Multimodal Federated Learningcited by 17
  2. 2024Federated Distillation: A Surveycited by 15
  3. 2024Applications of Knowledge Distillation in Remote Sensing: A Surveycited by 8
  4. 2022Decentralized Learning with Multi-Headed Distillationcited by 5
  5. 2023Edge-cloud Collaborative Learning with Federated and Centralized Featurescited by 5
  6. 2024CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Communitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Federated Cross-Training Learners for Robust Generalization under Data Heterogeneity
  2. 2022Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusion
  3. 2022Closing the Gap between Client and Global Model Performance in Heterogeneous Federated Learning
  4. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learning
  5. 2026FedeKD: Energy-Based Gating for Robust Federated Knowledge Distillation under Heterogeneous Settings
  6. 2023The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillation
  7. 2025Enabling Weak Client Participation via On-device Knowledge Distillation in Heterogeneous Federated Learning
  8. 2024Towards Diverse Device Heterogeneous Federated Learning via Task Arithmetic Knowledge Integration