mlsanityData to 7 Oct 2026

Towards Model Agnostic Federated Learning Using Knowledge Distillation

arXiv 2110.15210 ↗First posted 28 Oct 2021cs.LG cs.DC math.OCcited by 31

Not on the map. Why?

Abstract (arXiv)

Is it possible to design an universal API for federated learning using which an ad-hoc group of data-holders (agents) collaborate with each other and perform federated learning? Such an API would necessarily need to be model-agnostic i.e. make no assumption about the model architecture being used by the agents, and also cannot rely on having representative public data at hand. Knowledge distillation (KD) is the obvious tool of choice to design such protocols. However, surprisingly, we show that most natural KD-based federated learning protocols have poor performance. To investigate this, we propose a new theoretical framework, Federated Kernel ridge regression, which can capture both model heterogeneity as well as data heterogeneity. Our analysis shows that the degradation is largely due to a fundamental limitation of knowledge distillation under data heterogeneity. We further validate our framework by analyzing and designing new protocols based on KD. Their performance on real world experiments using neural networks, though still unsatisfactory, closely matches our theoretical predictions.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 31 of its 43 references that are in the map. First posted on arXiv 28 Oct 2021.

Builds on

31 of 43 references.

References · 31

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2019Advances and Open Problems in Federated Learningcited by 3,021
  5. 2020Knowledge Distillation: A Surveycited by 1,578
  6. 2020Adaptive Federated Optimizationcited by 1,036
  7. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  8. 2017Deep Mutual Learningcited by 621
  9. 2018Adaptive Federated Learning in Resource Constrained Edge Computing Systemscited by 408
  10. 2020Adaptive Personalized Federated Learningcited by 353
  11. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 321
  12. 2021A Field Guide to Federated Optimizationcited by 312
19 more
  1. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  2. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  3. 2019Model Fusion via Optimal Transportcited by 217
  4. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
  5. 2016Practical Secure Aggregation for Federated Learning on User-Held Datacited by 178
  6. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 174
  7. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 171
  8. 2021Towards Understanding Knowledge Distillationcited by 158
  9. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 152
  10. 2019Federated Evaluation of On-device Personalizationcited by 147
  11. 2018Large scale distributed neural network training through online distillationcited by 144
  12. 2020Understanding and Improving Knowledge Distillationcited by 83
  13. 2020Practical One-Shot Federated Learning for Cross-Silo Settingcited by 76
  14. 2020Federated Knowledge Distillationcited by 51
  15. 2021QuPeD: Quantized Personalization via Distillation with Applications to Federated Learningcited by 32
  16. 2020Communication-Efficient Federated Distillationcited by 28
  17. 2020Why distillation helps: a statistical perspectivecited by 21
  18. 2020A Closer Look at Codistillation for Distributed Trainingcited by 8
  19. 2021Optimal Model Averaging: Towards Personalized Collaborative Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 9; 2024: 8; 2025: 5; 2026: 3.

31 papers.

Sort:

By area:

  1. 2022A Survey on Heterogeneous Federated Learningcited by 42
  2. 2022MetaFed: Federated Learning among Federations with Cyclic Knowledge Distillation for Personalized Healthcarecited by 36
  3. 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
  4. 2022TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernelscited by 26
  5. 2022Accelerated Federated Learning with Decoupled Adaptive Optimizationcited by 23
  6. 2023DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learningcited by 23
  7. 2023A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiencycited by 21
  8. 2023FedICT: Federated Multi-task Distillation for Multi-access Edge Computingcited by 17
  9. 2024Federated Distillation: A Surveycited by 14
  10. 2023Agglomerative Federated Learning: Empowering Larger Model Training via End-Edge-Cloud Collaborationcited by 10
  11. 2024Accelerating Heterogeneous Federated Learning with Closed-form Classifierscited by 10
  12. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutionscited by 8
19 more
  1. 2024Real World Federated Learning with a Knowledge Distilled Transformer for Cardiac CT Imagingcited by 8
  2. 2023NeFL: Nested Model Scaling for Federated Learning with System Heterogeneous Clientscited by 7
  3. 2025FedConv: A Learning-on-Model Paradigm for Heterogeneous Federated Clientscited by 7
  4. 2022Supernet Training for Federated Image Classification under System Heterogeneitycited by 4
  5. 2023MrTF: Model Refinery for Transductive Federated Learningcited by 2
  6. 2024Addressing Skewed Heterogeneity via Federated Prototype Rectification with Personalizationcited by 2
  7. 2023Plug In and Learn: Federated Intelligence over a Smart Grid of Modelscited by 1
  8. 2023Structured Cooperative Learning with Graphical Model Priorscited by 1
  9. 2025Beyond Model Scale Limits: End-Edge-Cloud Federated Learning with Self-Rectified Knowledge Agglomerationcited by 1
  10. 2025Secure Federated Data Distillationcited by 1
  11. 2025FedSKD: Aggregation-free Model-heterogeneous Federated Learning via Multi-dimensional Similarity Knowledge Distillation for Medical Image Classificationcited by 1
  12. 2026Training-Free Dynamic Upcycling of Expert Language Modelscited by 1
  13. 2023Heterogeneous Federated Learning Using Knowledge Codistillationcited by 0
  14. 2024CoDream: Exchanging dreams instead of models for federated aggregation with heterogeneous modelscited by 0
  15. 2024InFiConD: Interactive No-code Fine-tuning with Concept-based Knowledge Distillationcited by 0
  16. 2024GeFL: Model-Agnostic Federated Learning with Generative Modelscited by 0
  17. 2025Asynchronous Federated Continual Segmentation with Evolving Clients and Label Spacescited by 0
  18. 2026COSMOS: Model-Agnostic Personalized Federated Learning with Clustered Server Models and Pseudo-Label-Only Communicationcited by 0
  19. 2026IR3DE: A Linear Router for Large Language Modelscited by 0