mlsanityData to 14 Oct 2026

Efficient Federated Finetuning of Tiny Transformers with Resource-Constrained Devices

arXiv 2411.07826 ↗First posted 12 Nov 2024cs.LG cs.AI cs.DCcited by 1

Not on the map. Why?

Abstract (arXiv)

In recent years, Large Language Models (LLMs) through Transformer structures have dominated many machine learning tasks, especially text processing. However, these models require massive amounts of data for training and induce high resource requirements, particularly in terms of the large number of Floating Point Operations (FLOPs) and the high amounts of memory needed. To fine-tune such a model in a parameter-efficient way, techniques like Adapter or LoRA have been developed. However, we observe that the application of LoRA, when used in federated learning (FL), while still being parameter-efficient, is memory and FLOP inefficient. Based on that observation, we develop a novel layer finetuning scheme that allows devices in cross-device FL to make use of pretrained neural networks (NNs) while adhering to given resource constraints. We show that our presented scheme outperforms the current state of the art when dealing with homogeneous or heterogeneous computation and memory constraints and is on par with LoRA regarding limited communication, thereby achieving significantly higher accuracies in FL training.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 12 Nov 2024.

Builds on

21 of 33 references.

References · 21

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  4. 2018LEAF: A Benchmark for Federated Settingscited by 792
  5. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  6. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  7. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  8. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  9. 2019FedHealth: A Federated Transfer Learning Framework for Wearable Healthcarecited by 182
  10. 2018Expanding the Reach of Federated Learning by Reducing Client Resource Requirementscited by 163
  11. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
  12. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
9 more
  1. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 93
  2. 2023Federated Learning for Computationally-Constrained Heterogeneous Devices: A Surveycited by 40
  3. 2021FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorizationcited by 31
  4. 2021ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Trainingcited by 27
  5. 2020A Federated Learning Framework for Healthcare IoT devicescited by 22
  6. 2020Heterogeneous Federated Learningcited by 19
  7. 2020Federated Learning for Localization: A Privacy-Preserving Crowdsourcing Methodcited by 9
  8. 2022CoCoFL: Communication- and Computation-Aware Federated Learning via Partial NN Freezing and Quantizationcited by 7
  9. 2023Aggregating Capacity in FL through Successive Layer Training for Computationally-Constrained Devicescited by 7

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Integration of TinyML and LargeML: A Survey of 6G and Beyondcited by 0