mlsanityData to 7 Oct 2026

Federated Knowledge Transfer Fine-tuning Large Server Model with Resource-Constrained IoT Clients

arXiv 2407.05268 ↗First posted 7 Jul 2024cs.LG cs.AI cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

The training of large models, involving fine-tuning, faces the scarcity of high-quality data. Compared to the solutions based on centralized data centers, updating large models in the Internet of Things (IoT) faces challenges in coordinating knowledge from distributed clients by using their private and heterogeneous data. To tackle such a challenge, we propose KOALA (Federated Knowledge Transfer Fine-tuning Large Server Model with Resource-Constrained IoT Clients) to impel the training of large models in IoT. Since the resources obtained by IoT clients are limited and restricted, it is infeasible to locally execute large models and also update them in a privacy-preserving manner. Therefore, we leverage federated learning and knowledge distillation to update large models through collaboration with their small models, which can run locally at IoT clients to process their private data separately and enable large-small model knowledge transfer through iterative learning between the server and clients. Moreover, to support clients with similar or different computing capacities, KOALA is designed with two kinds of large-small model joint learning modes, namely to be homogeneous or heterogeneous. Experimental results demonstrate that compared to the conventional approach, our method can not only achieve similar training performance but also significantly reduce the need for local storage and computing power resources.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 39 references that are in the map. First posted on arXiv 7 Jul 2024.

Builds on

29 of 39 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  8. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  9. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  12. 2014FitNets: Hints for Thin Deep Netscited by 1,619
17 more
  1. 2021Model-Contrastive Federated Learningcited by 762
  2. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  3. 2017Deep Mutual Learningcited by 621
  4. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  5. 2021Towards Personalized Federated Learningcited by 462
  6. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  7. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 309
  8. 2018Distributed learning of deep neural network over multiple agentscited by 277
  9. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 242
  10. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
  11. 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 83
  12. 2018In-Edge AI: Intelligentizing Mobile Edge Computing, Caching and Communication by Federated Learningcited by 80
  13. 2023Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Modelscited by 49
  14. 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
  15. 2023Integration of Large Language Models and Federated Learningcited by 28
  16. 2022FedClassAvg: Local Representation Learning for Personalized Federated Learning on Heterogeneous Neural Networkscited by 23
  17. 2022Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillationcited by 22

Built on by

Not cited yet.