mlsanityData to 14 Oct 2026

Adaptive Rank Allocation for Federated Parameter-Efficient Fine-Tuning of Language Models

arXiv 2501.14406 ↗First posted 24 Jan 2025cs.DC cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Pre-trained Language Models (PLMs) have demonstrated their superiority and versatility in modern Natural Language Processing (NLP), effectively adapting to various downstream tasks through further fine-tuning. Federated Parameter-Efficient Fine-Tuning (FedPEFT) has emerged as a promising solution to address privacy and efficiency challenges in distributed training for PLMs on resource-constrained local devices. However, our measurements reveal two key limitations of FedPEFT: heterogeneous data across devices exacerbates performance degradation of low-rank adaptation, and a fixed parameter configuration results in communication inefficiency. To overcome these limitations, we propose FedARA, a novel adaptive rank allocation framework for federated parameter-efficient fine-tuning of language models. Specifically, FedARA employs truncated Singular Value Decomposition (SVD) adaptation to enhance similar feature representation across clients, significantly mitigating the adverse effects of data heterogeneity. Subsequently, it utilizes dynamic rank allocation to progressively identify critical ranks, effectively improving communication efficiency. Lastly, it leverages rank-based module pruning to automatically remove inactive modules, steadily reducing local computational cost and memory usage in each federated learning round. Extensive experiments show that FedARA consistently outperforms baselines by an average of 6.95% to 8.49% across various datasets and models under heterogeneous data while significantly improving communication efficiency by 2.40 ×. Moreover, experiments on various edge devices demonstrate substantial decreases in total training time and energy consumption by up to 48.90% and 46.95%, respectively.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 38 of its 57 references that are in the map. First posted on arXiv 24 Jan 2025.

Builds on

38 of 57 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
26 more
  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  2. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  3. 2023A Survey of Large Language Modelscited by 2,576
  4. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  5. 2015Teaching Machines to Read and Comprehendcited by 1,049
  6. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  7. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  8. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  9. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
  10. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  11. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 155
  12. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 149
  13. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
  14. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 137
  15. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
  16. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 93
  17. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  18. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 86
  19. 2019SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominalscited by 63
  20. 2022Client Selection in Federated Learning: Principles, Challenges, and Opportunitiescited by 62
  21. 2022News Category Datasetcited by 43
  22. 2024FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decompositioncited by 26
  23. 2023FS-Real: Towards Real-World Cross-Device Federated Learningcited by 22
  24. 2023FedRA: A Random Allocation Strategy for Federated Tuning to Unleash the Power of Heterogeneous Clientscited by 19
  25. 2023Federated Fine-Tuning of LLMs on the Very Edge: The Good, the Bad, the Uglycited by 17
  26. 2021Accelerating Federated Learning with a Global Biased Optimisercited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuningcited by 3
  2. 2025FedLoDrop: Federated LoRA with Dropout for Generalized LLM Fine-tuningcited by 2
  3. 2026Rethinking LoRA for Data Heterogeneous Federated Learning: Subspace and State Alignmentcited by 2
  4. 2026SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Rankscited by 1