mlsanityData to 7 Oct 2026

Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devices

arXiv 2412.20004 ↗First posted 28 Dec 2024cs.DC cs.AI cs.NIcited by 5

Not on the map. Why?

Abstract (arXiv)

Federated fine-tuning (FedFT) has been proposed to fine-tune the pre-trained language models in a distributed manner. However, there are two critical challenges for efficient FedFT in practical applications, i.e., resource constraints and system heterogeneity. Existing works rely on parameter-efficient fine-tuning methods, e.g., low-rank adaptation (LoRA), but with major limitations. Herein, based on the inherent characteristics of FedFT, we observe that LoRA layers with higher ranks added close to the output help to save resource consumption while achieving comparable fine-tuning performance. Then we propose a novel LoRA-based FedFT framework, termed LEGEND, which faces the difficulty of determining the number of LoRA layers (called, LoRA depth) and the rank of each LoRA layer (called, rank distribution). We analyze the coupled relationship between LoRA depth and rank distribution, and design an efficient LoRA configuration algorithm for heterogeneous devices, thereby promoting fine-tuning efficiency. Extensive experiments are conducted on a physical platform with 80 commercial devices. The results show that LEGEND can achieve a speedup of 1.5-2.8× and save communication costs by about 42.3% when achieving the target accuracy, compared to the advanced solutions.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 42 of its 76 references that are in the map. First posted on arXiv 28 Dec 2024.

Builds on

42 of 76 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  12. 2019Advances and Open Problems in Federated Learningcited by 3,021
30 more
  1. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  2. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  3. 2024LoRA Learns Less and Forgets Lesscited by 313
  4. 2020SplitFed: When Federated Learning Meets Split Learningcited by 308
  5. 2019A Joint Learning and Communications Framework for Federated Learning over Wireless Networkscited by 262
  6. 2023Challenges and Applications of Large Language Modelscited by 251
  7. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  8. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  9. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 114
  10. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 102
  11. 2024Higher Layers Need More LoRA Expertscited by 101
  12. 2018Federated Learning for Keyword Spottingcited by 95
  13. 2022Layer-wised Model Aggregation for Personalized Federated Learningcited by 95
  14. 2022FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learningcited by 86
  15. 2021FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Taskscited by 84
  16. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 74
  17. 2024A Survey on LoRA of Large Language Modelscited by 67
  18. 2021Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Samplingcited by 66
  19. 2023Parameter-Efficient Fine-Tuning without Introducing New Latencycited by 59
  20. 2021AutoFL: Enabling Heterogeneity-Aware Energy Efficient Federated Learningcited by 39
  21. 2024On the Impact of Fine-Tuning on Chain-of-Thought Reasoningcited by 32
  22. 2023Federated Learning-Empowered AI-Generated Content in Wireless Networkscited by 26
  23. 2022FedAdapter: Efficient Federated Learning for Modern NLPcited by 24
  24. 2020Pretraining Federated Text Models for Next Word Predictioncited by 22
  25. 2023PockEngine: Sparse and Efficient Fine-tuning in a Pocketcited by 22
  26. 2024CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inferencecited by 21
  27. 2023MergeSFL: Split Federated Learning with Feature Merging and Batch Size Regulationcited by 13
  28. 2024Flexora: Flexible Low Rank Adaptation for Large Language Modelscited by 13
  29. 2022Convergence Acceleration in Wireless Federated Learning: A Stackelberg Game Approachcited by 10
  30. 2024Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 15
  2. 2025HiCoLoRA: Addressing Context-Prompt Misalignment via Hierarchical Collaborative LoRA for Zero-Shot DSTcited by 0
  3. 2026FED-FSTQ: Fisher-Guided Token Quantization for Communication-Efficient Federated Fine-Tuning of LLMs on Edge Devicescited by 0
  4. 2026Parameter Efficient Multi-Class Intelligent Scheduling for Multimodal Online Distributed Industrial Anomaly Detectioncited by 0
  5. 2026AlignFed: Alignment-Aware Asynchronous Federated Fine-Tuning for Large Language Models in Heterogeneous Edge Environmentscited by 0