Data to 7 Oct 2026

FedMD: Heterogenous Federated Learning via Model Distillation

arXiv 1910.03581 ↗First posted 8 Oct 2019cs.LG stat.MLcited by 454

Not on the map. Why?

Abstract (arXiv)

Federated learning enables the creation of a powerful centralized model without compromising data privacy of multiple participants. While successful, it does not incorporate the case where each participant independently designs its own model. Due to intellectual property concerns and heterogeneous nature of tasks and data, this is a widespread requirement in applications of federated learning to areas such as health care and AI as a service. In this work, we use transfer learning and knowledge distillation to develop a universal framework that enables federated learning when each agent owns not only their private data, but also uniquely designed models. We test our framework on the MNIST/FEMNIST dataset and the CIFAR10/CIFAR100 dataset and observe fast improvement across all participating models. With 10 distinct participants, the final test accuracy of each model on average receives a 20% gain on top of what's possible without collaboration and is only a few percent lower than the performance each model would have obtained if all private datasets were pooled and made directly available for all participants.

Key facts. Cited by 454 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 54; 2022: 74; 2023: 95; 2024: 92; 2025: 88; 2026: 50.

454 papers.

Sort:

By area:

2021Towards Personalized Federated Learningcited by 462
2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 349
2021Federated Learning on Non-IID Data: A Surveycited by 345
2021A Field Guide to Federated Optimizationcited by 312
2021Membership Inference Attacks on Machine Learning: A Surveycited by 306
2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 160
2021Federated Learning for Open Bankingcited by 138
2021Distributed Learning in Wireless Networks: Recent Progress and Future Challengescited by 125
2021Multi-Center Federated Learning: Clients Clustering for Better Personalizationcited by 112

38 more

2021Parameterized Knowledge Transfer for Personalized Federated Learningcited by 108
2021DENSE: Data-Free One-Shot Federated Learningcited by 100
2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
2023Deep Model Fusion: A Surveycited by 89
2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
2023Multimodal Federated Learning via Contrastive Representation Ensemblecited by 85
2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 83
2021On Large-Cohort Training for Federated Learningcited by 73
2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 67
2021Personalized Federated Learning through Local Memorizationcited by 57
2023FedGH: Heterogeneous Federated Learning with Generalized Global Headercited by 56
2022Communication-Efficient Adaptive Federated Learningcited by 55
2023FedFed: Feature Distillation against Data Heterogeneity in Federated Learningcited by 54
2021Adapt to Adaptation: Learning Personalization for Cross-Silo Federated Learningcited by 53
2022Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learningcited by 52
2021FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusioncited by 49
2024FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learningcited by 47
2022Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillationcited by 45
2023Fusing Models with Complementary Expertisecited by 45
2021Cluster-driven Graph Federated Learning over Multiple Domainscited by 43
2022A Survey on Heterogeneous Federated Learningcited by 42
2021Device Sampling for Heterogeneous Federated Learning: Theory, Algorithms, and Implementationcited by 40
2023Recent Advances on Federated Learning: A Systematic Surveycited by 40
2023Federated Learning for Computationally-Constrained Heterogeneous Devices: A Surveycited by 40
2023Heterogeneous Federated Knowledge Graph Embedding Learning and Unlearningcited by 39
2023Generalizable Heterogeneous Federated Cross-Correlation and Instance Similarity Learningcited by 39
2023Efficient Personalized Federated Learning via Sparse Model-Adaptationcited by 38
2023pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuningcited by 38
2021Non-IID data and Continual Learning processes in Federated Learning: A long road aheadcited by 36
2022No One Left Behind: Inclusive Federated Learning over Heterogeneous Devicescited by 36
2022FedTP: Federated Learning by Transformer Personalizationcited by 36
2021Emerging Trends in Federated Learning: From Model Fusion to Federated X Learningcited by 35
2022FedBR: Improving Federated Learning on Heterogeneous Data via Local Learning Bias Reductioncited by 35
2023Cross-Silo Prototypical Calibration for Federated Learning with Non-IID Datacited by 35
2022CD2-pFed: Cyclic Distillation-guided Channel Decoupling for Model Personalization in Federated Learningcited by 34
2021QuPeD: Quantized Personalization via Distillation with Applications to Federated Learningcited by 32
2021FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Modelscited by 31
2021Source Inference Attacks in Federated Learningcited by 31

50 most cited of 454 All 454 (JSON)