mlsanityData to 14 Oct 2026

FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusion

arXiv 2110.11027 ↗First posted 21 Oct 2021cs.LG cs.AIcited by 49

Not on the map. Why?

Abstract (arXiv)

Today data is often scattered among billions of resource-constrained edge devices with security and privacy constraints. Federated Learning (FL) has emerged as a viable solution to learn a global model while keeping data private, but the model complexity of FL is impeded by the computation resources of edge nodes. In this work, we investigate a novel paradigm to take advantage of a powerful server model to break through model capacity in FL. By selectively learning from multiple teacher clients and itself, a server model develops in-depth knowledge and transfers its knowledge back to clients in return to boost their respective performance. Our proposed framework achieves superior performance on both server and client models and provides several advantages in a unified framework, including flexibility for heterogeneous client architectures, robustness to poisoning attacks, and communication efficiency between clients and server on various image classification tasks.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Builds on 35 of its 46 references that are in the map. First posted on arXiv 21 Oct 2021.

Builds on

35 of 46 references.

References · 35

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  8. 2019Advances and Open Problems in Federated Learningcited by 3,046
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  10. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  11. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,060
  12. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
23 more
  1. 2019Towards Federated Learning at Scale: System Designcited by 832
  2. 2018LEAF: A Benchmark for Federated Settingscited by 792
  3. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  4. 2017Federated Multi-Task Learningcited by 672
  5. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 485
  6. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 464
  7. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  8. 2019Agnostic Federated Learningcited by 456
  9. 2019Fair Resource Allocation in Federated Learningcited by 394
  10. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 387
  11. 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 378
  12. 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 300
  13. 2019A Little Is Enough: Circumventing Defenses For Distributed Learningcited by 288
  14. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 202
  15. 2018NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applicationscited by 147
  16. 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
  17. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  18. 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 93
  19. 2020Reinforced Multi-Teacher Selection for Knowledge Distillationcited by 63
  20. 2021Selective Knowledge Distillation for Neural Machine Translationcited by 43
  21. 2019Abnormal Client Behavior Detection in Federated Learningcited by 40
  22. 2021Knowledge Inheritance for Pre-trained Language Modelscited by 38
  23. 2021Not All Knowledge Is Created Equal: Mutual Distillation of Confident Knowledgecited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 16; 2024: 12; 2025: 13; 2026: 3.

49 papers.

Sort:

By area:

  1. 2023Multimodal Federated Learning via Contrastive Representation Ensemblecited by 86
  2. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
  3. 2023FedGH: Heterogeneous Federated Learning with Generalized Global Headercited by 56
  4. 2023pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuningcited by 40
  5. 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
  6. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  7. 2024pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learningcited by 24
  8. 2022Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillationcited by 22
  9. 2022FLIS: Clustered Federated Learning via Inference Similarity for Non-IID Data Distributioncited by 22
  10. 2024Federated Model Heterogeneous Matryoshka Representation Learningcited by 21
  11. 2023FedICT: Federated Multi-task Distillation for Multi-access Edge Computingcited by 17
  12. 2024Federated Distillation: A Surveycited by 15
37 more
  1. 2024Towards Multi-modal Transformers in Federated Learningcited by 12
  2. 2024Advances in Robust Federated Learning: A Survey with Heterogeneity Considerationscited by 12
  3. 2023Tunable Soft Prompts are Messengers in Federated Learningcited by 10
  4. 2023FedSSA: Semantic Similarity-based Aggregation for Efficient Model-Heterogeneous Personalized Federated Learningcited by 10
  5. 2023Agglomerative Federated Learning: Empowering Larger Model Training via End-Edge-Cloud Collaborationcited by 10
  6. 2023pFedES: Model Heterogeneous Personalized Federated Learning with Feature Extractor Sharingcited by 9
  7. 2025Privacy-Preserving Federated Embedding Learning for Localized Retrieval-Augmented Generationcited by 8
  8. 2023Breaching FedMD: Image Recovery via Paired-Logits Inversion Attackcited by 7
  9. 2023Knowledge Distillation in Federated Edge Learning: A Surveycited by 5
  10. 2023FilFL: Client Filtering for Optimized Client Participation in Federated Learningcited by 5
  11. 2023Edge-cloud Collaborative Learning with Federated and Centralized Featurescited by 5
  12. 2024On the Byzantine-Resilience of Distillation-Based Federated Learningcited by 5
  13. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
  14. 2022Robust Federated Learning against both Data Heterogeneity and Poisoning Attack via Aggregation Optimizationcited by 3
  15. 2024pFedAFM: Adaptive Feature Mixture for Batch-Level Personalization in Heterogeneous Federated Learningcited by 3
  16. 2024FedHPL: Efficient Heterogeneous Federated Learning with Prompt Tuning and Logit Distillationcited by 3
  17. 2024Logits Poisoning Attack in Federated Distillationcited by 2
  18. 2024Practical Insights into Knowledge Distillation for Pre-Trained Modelscited by 2
  19. 2023Masked Autoencoders are Efficient Continual Federated Learnerscited by 1
  20. 2023ECLM: Efficient Edge-Cloud Collaborative Learning with Continuous Environment Adaptationcited by 1
  21. 2024Peak-Controlled Logits Poisoning Attack in Federated Distillationcited by 1
  22. 2025Beyond Model Scale Limits: End-Edge-Cloud Federated Learning with Self-Rectified Knowledge Agglomerationcited by 1
  23. 2025Data-Free Black-Box Federated Learning via Zeroth-Order Gradient Estimationcited by 1
  24. 2025Enabling Weak Client Participation via On-device Knowledge Distillation in Heterogeneous Federated Learningcited by 1
  25. 2025FedSKD: Aggregation-free Model-heterogeneous Federated Learning via Multi-dimensional Similarity Knowledge Distillation for Medical Image Classificationcited by 1
  26. 2025Bridging Generalization Gap of Heterogeneous Federated Clients Using Generative Modelscited by 1
  27. 2025Choice Outweighs Effort: Facilitating Complementary Knowledge Fusion in Federated Learning via Re-calibration and Merit-discriminationcited by 1
  28. 2023AIJack: Let's Hijack AI! Security and Privacy Risk Simulator for Machine Learningcited by 0
  29. 2024FedSiKD: Clients Similarity and Knowledge Distillation: Addressing Non-i.i.d. and Constraints in Federated Learningcited by 0
  30. 2025Moss: Proxy Model-based Full-Weight Aggregation in Federated Learning with Heterogeneous Modelscited by 0
  31. 2025Avoid Forgetting by Preserving Global Knowledge Gradients in Federated Learning with Non-IID Datacited by 0
  32. 2025Towards a Larger Model via One-Shot Federated Learning on Heterogeneous Client Modelscited by 0
  33. 2025FedMMKT:Co-Enhancing a Server Text-to-Image Model and Client Task Models in Multi-Modal Federated Learningcited by 0
  34. 2025FedeCouple: Fine-Grained Balancing of Global-Generalization and Local-Adaptability in Federated Learningcited by 0
  35. 2026Optimized Federated Knowledge Distillation with Distributed Neural Architecture Searchcited by 0
  36. 2026Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clientscited by 0
  37. 2026FedSocket: Recipient-Executable Knowledge Exchange for Heterogeneous Multimodal Federated Learningcited by 0