mlsanityData to 7 Oct 2026

Aggregating Low Rank Adapters in Federated Fine-tuning

arXiv 2501.06332 ↗First posted 10 Jan 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Fine-tuning large language models requires high computational and memory resources, and is therefore associated with significant costs. When training on federated datasets, an increased communication effort is also needed. For this reason, parameter-efficient methods (PEFT) are becoming increasingly important. In this context, very good results have already been achieved by fine-tuning with low-rank adaptation methods (LoRA). The application of LoRA methods in Federated Learning, and especially the aggregation of adaptation matrices, is a current research field. In this article, we propose a novel aggregation method and compare it with different existing aggregation methods of low rank adapters trained in a federated fine-tuning of large machine learning models and evaluate their performance with respect to selected GLUE benchmark datasets.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 30 references that are in the map. First posted on arXiv 10 Jan 2025.

Builds on

22 of 30 references.

References · 22

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  8. 2018Federated Learning with Non-IID Datacited by 1,099
  9. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  10. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  11. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
  12. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 174
10 more
  1. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 158
  2. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  3. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 114
  4. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 103
  5. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  6. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 92
  7. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 74
  8. 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 55
  9. 2024FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decompositioncited by 26
  10. 2023Low-Parameter Federated Learning with Large Language Modelscited by 22

Built on by

Not cited yet.