mlsanityData to 14 Oct 2026

Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning

arXiv 2204.12703 ↗First posted 27 Apr 2022cs.LGcited by 81

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) enables edge-devices to collaboratively learn a model without disclosing their private data to a central aggregating server. Most existing FL algorithms require models of identical architecture to be deployed across the clients and server, making it infeasible to train large models due to clients' limited system resources. In this work, we propose a novel ensemble knowledge transfer method named Fed-ET in which small models (different in architecture) are trained on clients, and used to train a larger model at the server. Unlike in conventional ensemble learning, in FL the ensemble can be trained on clients' highly heterogeneous data. Cognizant of this property, Fed-ET uses a weighted consensus distillation scheme with diversity regularization that efficiently extracts reliable consensus from the ensemble while improving generalization by exploiting the diversity within the ensemble. We show the generalization bound for the ensemble of weighted models trained on heterogeneous datasets that supports the intuition of Fed-ET. Our experiments on image and language tasks show that Fed-ET significantly outperforms other state-of-the-art FL algorithms with fewer communicated parameters, and is also robust against high data-heterogeneity.

Key facts. Cited by 81 papers in the map (data through 14 Oct 2026). Builds on 21 of its 26 references that are in the map. First posted on arXiv 27 Apr 2022.

Builds on

21 of 26 references.

References · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2019Advances and Open Problems in Federated Learningcited by 3,046
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  7. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  8. 2019Towards Federated Learning at Scale: System Designcited by 832
  9. 2021Model-Contrastive Federated Learningcited by 766
  10. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  11. 2021A Field Guide to Federated Optimizationcited by 315
  12. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
9 more
  1. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  2. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 175
  3. 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
  4. 2021Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristicscited by 67
  5. 2021FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusioncited by 49
  6. 2021QuPeD: Quantized Personalization via Distillation with Applications to Federated Learningcited by 32
  7. 2020Hydra: Preserving Ensemble Diversity for Model Distillationcited by 31
  8. 2021Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfercited by 23
  9. 2020Periodic Intra-Ensemble Knowledge Distillation for Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 25; 2024: 27; 2025: 15; 2026: 8.

81 papers.

Sort:

By area:

  1. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  2. 2023Multimodal Federated Learning via Contrastive Representation Ensemblecited by 86
  3. 2023FedGH: Heterogeneous Federated Learning with Generalized Global Headercited by 56
  4. 2023FedMultimodal: A Benchmark For Multimodal Federated Learningcited by 51
  5. 2023pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuningcited by 40
  6. 2023GPT-FL: Generative Pre-trained Model-Assisted Federated Learningcited by 24
  7. 2023DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learningcited by 24
  8. 2024pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learningcited by 24
  9. 2022DYNAFED: Tackling Client Data Heterogeneity with Global Dynamicscited by 22
  10. 2024Federated Model Heterogeneous Matryoshka Representation Learningcited by 21
  11. 2024FIARSE: Model-Heterogeneous Federated Learning via Importance-Aware Submodel Extractioncited by 21
  12. 2024FedMKT: Federated Mutual Knowledge Transfer for Large and Small Language Modelscited by 19
38 more
  1. 2023FedICT: Federated Multi-task Distillation for Multi-access Edge Computingcited by 17
  2. 2024FedMAC: Tackling Partial-Modality Missing in Federated Learning with Cross-Modal Aggregation and Contrastive Regularizationcited by 16
  3. 2024Federated Distillation: A Surveycited by 15
  4. 2023Practical Edge Detection via Robust Collaborative Learningcited by 13
  5. 2024The Federation Strikes Back: A Survey of Federated Learning Privacy Attacks, Defenses, Applications, and Policy Landscapecited by 13
  6. 2024Revisiting Ensembling in One-Shot Federated Learningcited by 13
  7. 2023Efficient Federated Prompt Tuning for Black-box Large Pre-trained Modelscited by 12
  8. 2024FedFa: A Fully Asynchronous Training Paradigm for Federated Learningcited by 12
  9. 2024Towards Multi-modal Transformers in Federated Learningcited by 12
  10. 2024Open Challenges and Opportunities in Federated Foundation Models Towards Biomedical Healthcarecited by 12
  11. 2024Green Federated Learning: A new era of Green Aware AIcited by 12
  12. 2023FedSSA: Semantic Similarity-based Aggregation for Efficient Model-Heterogeneous Personalized Federated Learningcited by 10
  13. 2023Agglomerative Federated Learning: Empowering Larger Model Training via End-Edge-Cloud Collaborationcited by 10
  14. 2024Breaking the Memory Wall for Heterogeneous Federated Learning via Progressive Trainingcited by 10
  15. 2022Completely Heterogeneous Federated Learningcited by 9
  16. 2023pFedES: Model Heterogeneous Personalized Federated Learning with Feature Extractor Sharingcited by 9
  17. 2023AdaptiveFL: Adaptive Heterogeneous Federated Learning for Resource-Constrained AIoT Systemscited by 9
  18. 2023FedAIoT: A Federated Learning Benchmark for Artificial Intelligence of Thingscited by 8
  19. 2025Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Datacited by 8
  20. 2022To Store or Not? Online Data Selection for Federated Learning with Limited Storagecited by 7
  21. 2024Federated Learning with Label-Masking Distillationcited by 7
  22. 2023Knowledge Distillation in Federated Edge Learning: A Surveycited by 5
  23. 2023Little is Enough: Boosting Privacy by Sharing Only Hard Labels in Federated Semi-Supervised Learningcited by 5
  24. 2024Towards Diverse Device Heterogeneous Federated Learning via Task Arithmetic Knowledge Integrationcited by 5
  25. 2022FEL: High Capacity Learning for Recommendation and Ranking via Federated Ensemble Learningcited by 4
  26. 2023Local or Global: Selective Knowledge Assimilation for Federated Learning with Limited Labelscited by 4
  27. 2023AdapterFL: Adaptive Heterogeneous Federated Learning for Resource-constrained Mobile Computing Systemscited by 4
  28. 2022Robust Federated Learning against both Data Heterogeneity and Poisoning Attack via Aggregation Optimizationcited by 3
  29. 2023Confidence-based federated distillation for vision-based lane-centeringcited by 3
  30. 2023Towards Open Federated Learning Platforms: Survey and Vision from Technical and Legal Perspectivescited by 3
  31. 2024pFedAFM: Adaptive Feature Mixture for Batch-Level Personalization in Heterogeneous Federated Learningcited by 3
  32. 2025FedBEns: One-Shot Federated Learning based on Bayesian Ensemblecited by 3
  33. 2026Federated Prompt-Tuning with Heterogeneous and Incomplete Multimodal Client Datacited by 3
  34. 2024FedBRB: An Effective Solution to the Small-to-Large Scenario in Device-Heterogeneity Federated Learningcited by 2
  35. 2024FedDr+: Stabilizing Dot-regression with Global Feature Distillation for Federated Learningcited by 2
  36. 2023Distributed Learning in Heterogeneous Environment: federated learning with adaptive aggregation and computation reductioncited by 1
  37. 2023ECLM: Efficient Edge-Cloud Collaborative Learning with Continuous Environment Adaptationcited by 1
  38. 2023FedSDD: Scalable and Diversity-enhanced Distillation for Model Aggregation in Federated Learningcited by 1

50 most cited of 81 All 81 (JSON)