mlsanityData to 7 Oct 2026

Resource-Efficient Federated Fine-Tuning Large Language Models for Heterogeneous Data

arXiv 2503.21213 ↗First posted 27 Mar 2025cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Fine-tuning large language models (LLMs) via federated learning, i.e., FedLLM, has been proposed to adapt LLMs for various downstream applications in a privacy-preserving way. To reduce the fine-tuning costs on resource-constrained devices, FedLoRA is proposed to fine-tune only a small subset of model parameters by integrating low-rank adaptation (LoRA) into FedLLM. However, apart from resource constraints, there is still another critical challenge, i.e., data heterogeneity, severely hindering the implementation of FedLoRA in practical applications. Herein, inspired by the previous group-based federated learning paradigm, we propose a hierarchical FedLoRA framework, termed HierFedLoRA, to address these challenges. Specifically, HierFedLoRA partitions all devices into multiple near-IID groups and adjusts the intra-group aggregation frequency for each group to eliminate the negative effects of non-IID data. Meanwhile, to reduce the computation and communication cost, HierFedLoRA dynamically assigns diverse and suitable fine-tuning depth (i.e., the number of continuous fine-tuning layers from the output) for each group. HierFedLoRA explores jointly optimizing aggregation frequency and depth upon their coupled relationship to better enhance the performance of FedLoRA. Extensive experiments are conducted on a physical platform with 80 commercial devices. The results show that HierFedLoRA improves the final model accuracy by 1.6% to 4.2%, speeding up the fine-tuning process by at least 2.1×, compared to the strong baselines.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 39 of its 62 references that are in the map. First posted on arXiv 27 Mar 2025.

Builds on

39 of 62 references.

References · 39

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  11. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  12. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
27 more
  1. 2018Federated Learning with Non-IID Datacited by 1,099
  2. 2019How Can We Know What Language Models Know?cited by 953
  3. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 262
  4. 2023Challenges and Applications of Large Language Modelscited by 251
  5. 2019Client-Edge-Cloud Hierarchical Federated Learningcited by 202
  6. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 132
  7. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 114
  8. 2018Federated Learning for Keyword Spottingcited by 95
  9. 2022Layer-wised Model Aggregation for Personalized Federated Learningcited by 95
  10. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  11. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 92
  12. 2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 87
  13. 2021FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Taskscited by 84
  14. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 74
  15. 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 73
  16. 2019On-Device Machine Learning: An Algorithms and Learning Theory Perspectivecited by 63
  17. 2023Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytescited by 55
  18. 2017FreezeOut: Accelerate Training by Progressively Freezing Layerscited by 54
  19. 2024Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 49
  20. 2024FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decompositioncited by 26
  21. 2022FedAdapter: Efficient Federated Learning for Modern NLPcited by 24
  22. 2023PockEngine: Sparse and Efficient Fine-tuning in a Pocketcited by 22
  23. 2020Towards Efficient Scheduling of Federated Mobile Devices under Computational and Statistical Heterogeneitycited by 20
  24. 2022Walk for Learning: A Random Walk Approach for Federated Learning from Heterogeneous Datacited by 18
  25. 2024Flexora: Flexible Low Rank Adaptation for Large Language Modelscited by 13
  26. 2020MAB-based Client Selection for Federated Learning with Uncertain Resources in Mobile Networkscited by 11
  27. 2020Accurate and Fast Federated Learning via IID and Communication-Aware Groupingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025FFT-MoE: Efficient Federated Fine-Tuning for Foundation Models via Large-scale Sparse MoE under Heterogeneous Edgecited by 7
  2. 2026FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuningcited by 3
  3. 2025FedLoDrop: Federated LoRA with Dropout for Generalized LLM Fine-tuningcited by 2
  4. 2025A Federated Fine-Tuning Paradigm of Foundation Models in Heterogenous Wireless Networkscited by 0