mlsanityData to 14 Oct 2026

SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models

arXiv 2308.06522 ↗First posted 12 Aug 2023cs.LG cs.AIcited by 93

Not on the map. Why?

Abstract (arXiv)

Transfer learning via fine-tuning pre-trained transformer models has gained significant success in delivering state-of-the-art results across various NLP tasks. In the absence of centralized data, Federated Learning (FL) can benefit from distributed and private data of the FL edge clients for fine-tuning. However, due to the limited communication, computation, and storage capabilities of edge devices and the huge sizes of popular transformer models, efficient fine-tuning is crucial to make federated training feasible. This work explores the opportunities and challenges associated with applying parameter efficient fine-tuning (PEFT) methods in different FL settings for language tasks. Specifically, our investigation reveals that as the data across users becomes more diverse, the gap between fully fine-tuning the model and employing PEFT methods widens. To bridge this performance gap, we propose a method called SLoRA, which overcomes the key limitations of LoRA in high heterogeneous data scenarios through a novel data-driven initialization technique. Our experimental results demonstrate that SLoRA achieves performance comparable to full fine-tuning, with significant sparse updates with approximately ~ 1% density while reducing training time by up to 90%.

Key facts. Cited by 93 papers in the map (data through 14 Oct 2026). Builds on 29 of its 40 references that are in the map. First posted on arXiv 12 Aug 2023.

Builds on

29 of 40 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2019Advances and Open Problems in Federated Learningcited by 3,046
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  10. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  11. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  12. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
17 more
  1. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  2. 2020Adaptive Federated Optimizationcited by 1,046
  3. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  4. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  5. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  6. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  7. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  8. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 185
  9. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
  10. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  11. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  12. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  13. 2019Reliable Federated Learning for Mobile Networkscited by 92
  14. 2021Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Bettercited by 55
  15. 2023AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuningcited by 49
  16. 2022ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsitycited by 45
  17. 2022Federated Learning of Large Models at the Edge via Principal Sub-Model Trainingcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 37; 2025: 33; 2026: 19.

93 papers.

Sort:

By area:

  1. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  2. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  3. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 123
  4. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
  5. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  6. 2024A Survey on LoRA of Large Language Modelscited by 67
  7. 2023Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytescited by 55
  8. 2024Dual-Personalizing Adapter for Federated Foundation Modelscited by 55
  9. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 37
  10. 2024Advances and Open Challenges in Federated Foundation Modelscited by 36
  11. 2024The Future of Large Language Model Pre-training is Federatedcited by 30
  12. 2024A Survey on Efficient Federated Learning Methods for Foundation Model Trainingcited by 29
38 more
  1. 2024Federated Large Language Models: Current Progress and Future Directionscited by 27
  2. 2024FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decompositioncited by 26
  3. 2024Federated LoRA with Sparse Communicationcited by 26
  4. 2024Parameter Efficient Fine-tuning via Explained Variance Adaptationcited by 20
  5. 2024FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Modelscited by 18
  6. 2023Federated Fine-Tuning of LLMs on the Very Edge: The Good, the Bad, the Uglycited by 17
  7. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 17
  8. 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 16
  9. 2024AdapterSwap: Continuous Training of LLMs with Data Removal and Access-Control Guaranteescited by 14
  10. 2024DAGER: Exact Gradient Inversion for Large Language Modelscited by 14
  11. 2024Synergizing Foundation Models and Federated Learning: A Surveycited by 14
  12. 2025Decentralized Low-Rank Fine-Tuning of Large Language Modelscited by 14
  13. 2023Mutual Enhancement of Large and Small Language Models with Cross-Silo Knowledge Transfercited by 13
  14. 2024Federated Learning Priorities Under the European Union Artificial Intelligence Actcited by 13
  15. 2023Sweeping Heterogeneity with Smart MoPs: Mixture of Prompts for LLM Task Adaptationcited by 12
  16. 2024Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Modelscited by 12
  17. 2024Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architecturescited by 12
  18. 2024Federated Continual Learning for Edge-AI: A Comprehensive Surveycited by 11
  19. 2024Accelerating Heterogeneous Federated Learning with Closed-form Classifierscited by 10
  20. 2024Towards Federated Low-Rank Adaptation of Language Models with Rank Heterogeneitycited by 10
  21. 2024Federated Data-Efficient Instruction Tuning for Large Language Modelscited by 10
  22. 2025Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuningcited by 10
  23. 2024RE-Adapt: Reverse Engineered Adaptation of Large Language Modelscited by 9
  24. 2025Federated Low-Rank Adaptation for Foundation Models: A Surveycited by 9
  25. 2025Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuningcited by 9
  26. 2025Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocationcited by 7
  27. 2025The Primacy of Magnitude in Low-Rank Adaptationcited by 7
  28. 2025A Comprehensive Data-centric Overview of Federated Graph Learningcited by 7
  29. 2024Fisher Information-based Efficient Curriculum Federated Learning with Large Language Modelscited by 6
  30. 2025Personalized Federated Fine-tuning for Heterogeneous Data: An Automatic Rank Learning Approach via Two-Level LoRAcited by 6
  31. 2025Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuningcited by 6
  32. 2024SPD-CFL: Stepwise Parameter Dropout for Efficient Continual Federated Learningcited by 5
  33. 2026FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignmentcited by 5
  34. 2024Probing the Efficacy of Federated Parameter-Efficient Fine-Tuning of Vision Transformers for Medical Image Classificationcited by 4
  35. 2024DEeR: Deviation Eliminating and Noise Regulating for Privacy-preserving Federated Low-rank Adaptationcited by 4
  36. 2025Adaptive Rank Allocation for Federated Parameter-Efficient Fine-Tuning of Language Modelscited by 4
  37. 2025Resource-Efficient Federated Fine-Tuning Large Language Models for Heterogeneous Datacited by 4
  38. 2025FLoRIST: Singular Value Thresholding for Efficient and Accurate Federated Fine-Tuning of Large Language Modelscited by 4

50 most cited of 93 All 93 (JSON)