mlsanityData to 7 Oct 2026

An Upload-Efficient Scheme for Transferring Knowledge From a Server-Side Pre-trained Generator to Clients in Heterogeneous Federated Learning

arXiv 2403.15760 ↗First posted 23 Mar 2024cs.AI cs.DCcited by 17

Not on the map. Why?

Abstract (arXiv)

Heterogeneous Federated Learning (HtFL) enables task-specific knowledge sharing among clients with different model architectures while preserving privacy. Despite recent research progress, transferring knowledge in HtFL is still difficult due to data and model heterogeneity. To tackle this, we introduce a public pre-trained generator (e.g., StyleGAN or Stable Diffusion) as the bridge and propose a new upload-efficient knowledge transfer scheme called Federated Knowledge-Transfer-Loop (FedKTL). It can produce task-related prototypical image-vector pairs via the generator's inference on the server. With these pairs, each client can transfer common knowledge from the generator to its local model through an additional supervised local task. We conduct extensive experiments on four datasets under two types of data heterogeneity with 14 heterogeneous models, including CNNs and ViTs. Results show that our FedKTL surpasses seven state-of-the-art methods by up to 7.31%. Moreover, our knowledge transfer scheme is applicable in cloud-edge scenarios with only one edge client. Code: https://github.com/TsingZ0/FedKTL

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 55 of its 81 references that are in the map. First posted on arXiv 23 Mar 2024.

Builds on

55 of 81 references.

References · 55

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  8. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  9. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  10. 2014Going Deeper with Convolutionscited by 4,231
  11. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  12. 2019Advances and Open Problems in Federated Learningcited by 3,021
38 more
  1. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  2. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  3. 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,323
  4. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  5. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  6. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  7. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  8. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  9. 2017Deep Mutual Learningcited by 621
  10. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  11. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  12. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  13. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
  14. 2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 371
  15. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 349
  16. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
  17. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 322
  18. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 242
  19. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  20. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  21. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
  22. 2015Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Featurecited by 181
  23. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  24. 2022Federated Learning from Pre-Trained Models: A Contrastive Learning Approachcited by 141
  25. 2022FedALA: Adaptive Local Aggregation for Personalized Federated Learningcited by 121
  26. 2023Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learningcited by 112
  27. 2021Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learningcited by 111
  28. 2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Expertscited by 110
  29. 2021Parameterized Knowledge Transfer for Personalized Federated Learningcited by 108
  30. 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
  31. 2014Deep learning with Elastic Averaging SGDcited by 90
  32. 2023Multimodal Federated Learning via Contrastive Representation Ensemblecited by 85
  33. 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 83
  34. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  35. 2020Federated Mutual Learningcited by 70
  36. 2020FedHome: Cloud-Edge based Personalized Federated Learning for In-Home Health Monitoringcited by 61
  37. 2021Federated Dropout -- A Simple Approach for Enabling Federated Learning on Resource Constrained Devicescited by 56
  38. 2023FedGH: Heterogeneous Federated Learning with Generalized Global Headercited by 56

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 8; 2026: 6.

17 papers.

Sort:

  1. 2024FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Modelscited by 26
  2. 2024One-shot Federated Learning via Synthetic Distiller-Distillate Communicationcited by 12
  3. 2025AugFL: Augmenting Federated Learning with Pretrained Modelscited by 6
  4. 2025Robust Asymmetric Heterogeneous Federated Learning with Corrupted Clientscited by 5
  5. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
  6. 2025FedBM: Stealing Knowledge from Pre-trained Language Models for Heterogeneous Federated Learningcited by 2
  7. 2025Enhancing Visual Representation with Textual Semantics: Textual Semantics-Powered Prototypes for Heterogeneous Federated Learningcited by 2
  8. 2025Efficient Federated Learning with Heterogeneous Data and Adaptive Dropoutcited by 2
  9. 2024Adaptive Guidance for Local Training in Heterogeneous Federated Learningcited by 0
  10. 2025A Scenario-Oriented Survey of Federated Recommender Systems: Techniques, Challenges, and Future Directionscited by 0
  11. 2025FedMMKT:Co-Enhancing a Server Text-to-Image Model and Client Task Models in Multi-Modal Federated Learningcited by 0
  12. 2026GFPL: Generative Federated Prototype Learning for Resource-Constrained and Data-Imbalanced Vision Taskcited by 0
5 more
  1. 2026Geometric Knowledge-Assisted Federated Dual Knowledge Distillation Approach Towards Remote Sensing Satellite Imagerycited by 0
  2. 2026FedDBP: Enhancing Federated Prototype Learning with Dual-Branch Features and Personalized Global Fusioncited by 0
  3. 2026From Coordinate Matching to Structural Alignment: Rethinking Prototype Alignment in Heterogeneous Federated Learningcited by 0
  4. 2026FedTopo: Relation-Level Topology Sharing for Model-Heterogeneous Federated Learningcited by 0
  5. 2026vFedProtoQNAS: Prototype-Guided Personalized Quantum Neural Architecture Search for Virtual Federated Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Data-Free Continual Learning of Server Models in Model-Heterogeneous Cloud-Device Collaboration
  2. 2023One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models
  3. 2022FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformers
  4. 2025Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environments
  5. 2026Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learning
  6. 2024FedTSA: A Cluster-based Two-Stage Aggregation Method for Model-heterogeneous Federated Learning
  7. 2024Stable Diffusion-based Data Augmentation for Federated Learning with Non-IID Data
  8. 2023CLIP-guided Federated Learning on Heterogeneous and Long-Tailed Data