mlsanityData to 7 Oct 2026

Towards Federated Low-Rank Adaptation of Language Models with Rank Heterogeneity

arXiv 2406.17477 ↗First posted 25 Jun 2024cs.DC cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Low-rank adaptation (LoRA) offers an efficient alternative to full-weight adaptation in federated fine-tuning of language models, significantly reducing computational costs. By adjusting ranks for each client, federated LoRA enables flexible resource allocation. However, we observe that heterogeneous ranks among clients lead to unstable performance. Our analysis attributes this instability to the conventional zero-padding aggregation strategy, which dilutes information from high-rank clients during model aggregation. To address this issue, we propose a replication-based padding strategy that better retains valuable information from clients with high-quality data. Empirically, this approach accelerates convergence and enhances the global model's predictive performance.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 12 of its 17 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

12 of 17 references.

References · 12

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  8. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  9. 2021Federated Learning on Non-IID Data: A Surveycited by 345
  10. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 114
  11. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 92
  12. 2021FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Taskscited by 84

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 5.

10 papers.

  1. 2025Federated Low-Rank Adaptation for Foundation Models: A Surveycited by 9
  2. 2026FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignmentcited by 5
  3. 2025Federated Sketching LoRA: A Flexible Framework for Heterogeneous Collaborative Fine-Tuning of LLMscited by 2
  4. 2025Adaptive Federated LoRA in Heterogeneous Wireless Networks with Independent Samplingcited by 1
  5. 2025FediLoRA: Practical Federated Fine-Tuning of Foundation Models Under Missing-Modality Constraintscited by 1
  6. 2026Preventing Rank Collapse in Federated Low-Rank Adaptation with Client Heterogeneitycited by 1
  7. 2026SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Rankscited by 1
  8. 2025Communication-Efficient and Accurate Approach for Aggregation in Federated Low-Rank Adaptationcited by 0
  9. 2026Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approachcited by 0
  10. 2026CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregationcited by 0