mlsanityData to 14 Oct 2026

Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clients

arXiv 2410.22815 ↗First posted 30 Oct 2024cs.LG cs.AI cs.DCcited by 22

Not on the map. Why?

Abstract (arXiv)

Federated fine-tuning for Large Language Models (LLMs) faces significant challenges due to the heavy communication overhead of transmitting large model updates. Although Low Rank Adaptation (LoRA) has been proposed as a solution, yet its application in federated learning is complicated by discordance in aggregation. Existing methods addressing this discordance often suffer from performance degradation at low ranks in heterogeneous data settings. In response, we introduce LoRA-A2 (Low Rank Adaptation with Alternating freeze and Adaptive rank selection), which demonstrates robustness in challenging settings with low ranks and high data heterogeneity. Our experimental findings reveal that LoRA-A2 maintains performance even under extreme heterogeneity and low rank conditions, achieving up to a significant reduction in uploaded parameters compared to full fine-tuning without compromising performance. This adaptive mechanism increases robustness and communication efficiency in federated fine-tuning, enabling the practical deployment of LLMs in resource-constrained environments.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 24 of its 33 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

24 of 33 references.

References · 24

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  7. 2019Advances and Open Problems in Federated Learningcited by 3,046
  8. 2016Deep Learning with Differential Privacycited by 2,728
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  10. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  11. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  12. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
12 more
  1. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  2. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  3. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 455
  4. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
  5. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  6. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  7. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 370
  8. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
  9. 2016Distributed Mean Estimation with Limited Communicationcited by 136
  10. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
  11. 2024DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distributioncited by 43
  12. 2023Efficient Personalized Federated Learning via Sparse Model-Adaptationcited by 38

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 10; 2026: 10.

22 papers.

Sort:

By area:

  1. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 37
  2. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 17
  3. 2025Decentralized Low-Rank Fine-Tuning of Large Language Modelscited by 14
  4. 2025FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Modelscited by 11
  5. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  6. 2025Federated Low-Rank Adaptation for Foundation Models: A Surveycited by 9
  7. 2024Covariances for Free: Exploiting Mean Distributions for Training-free Federated Learningcited by 4
  8. 2025Fed-SE: Federated Self-Evolution for Privacy-Constrained Multi-Environment LLM Agentscited by 3
  9. 2025Federated Sketching LoRA: A Flexible Framework for Heterogeneous Collaborative Fine-Tuning of LLMscited by 2
  10. 2025Convergence Analysis of Aggregation-Broadcast in LoRA-enabled Distributed Fine-Tuningcited by 2
  11. 2025Adaptive Federated LoRA in Heterogeneous Wireless Networks with Independent Samplingcited by 1
  12. 2026Making Models Unmergeable via Scaling-Sensitive Loss Landscapecited by 1
10 more
  1. 2026WinFLoRA: Incentivizing Client-Adaptive Aggregation in Federated LoRA under Privacy Heterogeneitycited by 1
  2. 2026Adaptive Selection of LoRA Components in Privacy-Preserving Federated Learningcited by 1
  3. 2026SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Rankscited by 1
  4. 2025Fed-PELAD: Communication-Efficient Federated Learning for Massive MIMO CSI Feedback with Personalized Encoders and a LoRA-Adapted Shared Decodercited by 0
  5. 2026Beyond Parameter Aggregation: Semantic Consensus for Federated Fine-Tuning of LLMscited by 0
  6. 2026Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learningcited by 0
  7. 2026FraQ: Efficient Coordinate-Space Recompression for Federated Low-Rank Adaptationcited by 0
  8. 2026Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approachcited by 0
  9. 2026CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregationcited by 0
  10. 2026RoboFL: Federated Expert Assembly for World Action Modelscited by 0