mlsanityData to 7 Oct 2026

Heterogeneous Federated Learning Using Knowledge Codistillation

arXiv 2310.02549 ↗First posted 4 Oct 2023cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated Averaging, and many federated learning algorithm variants which build upon it, have a limitation: all clients must share the same model architecture. This results in unused modeling capacity on many clients, which limits model performance. To address this issue, we propose a method that involves training a small model on the entire pool and a larger model on a subset of clients with higher capacity. The models exchange information bidirectionally via knowledge distillation, utilizing an unlabeled dataset on a server without sharing parameters. We present two variants of our method, which improve upon federated averaging on image classification and language modeling tasks. We show this technique can be useful even if only out-of-domain or limited in-domain distillation data is available. Additionally, the bi-directional knowledge distillation allows for domain transfer between the models when different pool populations introduce domain shift.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 43 references that are in the map. First posted on arXiv 4 Oct 2023.

Builds on

28 of 43 references.

References · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  7. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  8. 2020Adaptive Federated Optimizationcited by 1,036
  9. 2019Contrastive Representation Distillationcited by 734
  10. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  11. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
  12. 2021A Field Guide to Federated Optimizationcited by 312
16 more
  1. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 242
  2. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  3. 2018Large scale distributed neural network training through online distillationcited by 144
  4. 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
  5. 2020Federated Knowledge Distillationcited by 51
  6. 2020Federated Composite Optimizationcited by 49
  7. 2019Wireless Federated Distillation for Distributed Edge Learning with Heterogeneous Datacited by 39
  8. 2022Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customizationcited by 34
  9. 2021QuPeD: Quantized Personalization via Distillation with Applications to Federated Learningcited by 32
  10. 2021FedJAX: Federated learning simulation with JAXcited by 32
  11. 2021Towards Model Agnostic Federated Learning Using Knowledge Distillationcited by 31
  12. 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
  13. 2020Mix2FLD: Downlink Federated Learning After Uplink Federated Distillation With Two-Way Mixupcited by 29
  14. 2020Subclass Distillationcited by 23
  15. 2020Cooperative Learning via Federated Distillation over Fading Channelscited by 16
  16. 2022Layerwise Bregman Representation Learning with Applications to Knowledge Distillationcited by 1

Built on by

Not cited yet.