mlsanityData to 14 Oct 2026

Ensemble Distillation for Robust Model Fusion in Federated Learning

arXiv 2006.07242 ↗First posted 12 Jun 2020cs.LG stat.MLcited by 682

Not on the map. Why?

Cited by 67 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Federated Learning (FL) is a machine learning setting where many devices collaboratively train a machine learning model while keeping the training data decentralized. In most of the current training schemes the central model is refined by averaging the parameters of the server model and the updated parameters from the client side. However, directly averaging model parameters is only possible if all models have the same structure and size, which could be a restrictive constraint in many scenarios. In this work we investigate more powerful and more flexible aggregation schemes for FL. Specifically, we propose ensemble distillation for model fusion, i.e. training the central classifier through unlabeled data on the outputs of the models from the clients. This knowledge distillation technique mitigates privacy risk and cost to the same extent as the baseline FL algorithms, but allows flexible aggregation over heterogeneous client models that can differ e.g. in size, numerical precision or structure. We show in extensive empirical experiments on various CV/NLP datasets (CIFAR-10/100, ImageNet, AG News, SST2) and settings (heterogeneous models/data) that the server model can be trained much faster, requiring fewer communication rounds than any existing FL technique so far.

Key facts. Cited by 682 papers in the map (data through 14 Oct 2026). First posted on arXiv 12 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 73; 2022: 118; 2023: 148; 2024: 144; 2025: 117; 2026: 81.

682 papers.

Sort:

By area:

  1. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  2. 2021Towards Personalized Federated Learningcited by 464
  3. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 383
  4. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  5. 2021Federated Learning on Non-IID Data: A Surveycited by 346
  6. 2021A Field Guide to Federated Optimizationcited by 315
  7. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
  8. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  9. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 187
  10. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  11. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 161
  12. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
38 more
  1. 2022Federated Learning from Pre-Trained Models: A Contrastive Learning Approachcited by 142
  2. 2021Distributed Learning in Wireless Networks: Recent Progress and Future Challengescited by 129
  3. 2022FedALA: Adaptive Local Aggregation for Personalized Federated Learningcited by 122
  4. 2021Back to the Drawing Board: A Critical Evaluation of Poisoning Attacks on Production Federated Learningcited by 118
  5. 2021Multi-Center Federated Learning: Clients Clustering for Better Personalizationcited by 113
  6. 2021Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learningcited by 111
  7. 2021Parameterized Knowledge Transfer for Personalized Federated Learningcited by 109
  8. 2022Federated Learning with Label Distribution Skew via Logits Calibrationcited by 109
  9. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  10. 2021DENSE: Data-Free One-Shot Federated Learningcited by 100
  11. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 99
  12. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  13. 2023Personalized Federated Learning with Feature Alignment and Classifier Collaborationcited by 97
  14. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  15. 2023Deep Model Fusion: A Surveycited by 91
  16. 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
  17. 2023Multimodal Federated Learning via Contrastive Representation Ensemblecited by 86
  18. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
  19. 2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
  20. 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 74
  21. 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
  22. 2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 69
  23. 2021Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Datacited by 69
  24. 2022Tackling Data Heterogeneity in Federated Learning with Class Prototypescited by 67
  25. 2023FedDisco: Federated Learning with Discrepancy-Aware Collaborationcited by 67
  26. 2022Federated Learning on Heterogeneous and Long-Tailed Data via Classifier Re-Training with Federated Featurescited by 63
  27. 2021What Do We Mean by Generalization in Federated Learning?cited by 62
  28. 2023Revisiting Weighted Aggregation in Federated Learning with Neural Networkscited by 58
  29. 2021Personalized Federated Learning through Local Memorizationcited by 57
  30. 2021Decentralized Federated Learning through Proxy Model Sharingcited by 57
  31. 2023FedGH: Heterogeneous Federated Learning with Generalized Global Headercited by 56
  32. 2023No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifiercited by 55
  33. 2022Personalized Subgraph Federated Learningcited by 54
  34. 2023FedFed: Feature Distillation against Data Heterogeneity in Federated Learningcited by 54
  35. 2022Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learningcited by 52
  36. 2023FedMultimodal: A Benchmark For Multimodal Federated Learningcited by 51
  37. 2023Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Modelscited by 50
  38. 2021FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusioncited by 49

50 most cited of 682 All 682 (JSON)