mlsanityData to 14 Oct 2026

Selective Aggregation for Low-Rank Adaptation in Federated Learning

arXiv 2410.01463 ↗First posted 2 Oct 2024cs.LGcited by 80

Not on the map. Why?

Abstract (arXiv)

We investigate LoRA in federated learning through the lens of the asymmetry analysis of the learned A and B matrices. In doing so, we uncover that A matrices are responsible for learning general knowledge, while B matrices focus on capturing client-specific knowledge. Based on this finding, we introduce Federated Share-A Low-Rank Adaptation (FedSA-LoRA), which employs two low-rank trainable matrices A and B to model the weight update, but only A matrices are shared with the server for aggregation. Moreover, we delve into the relationship between the learned A and B matrices in other LoRA variants, such as rsLoRA and VeRA, revealing a consistent pattern. Consequently, we extend our FedSA-LoRA method to these LoRA variants, resulting in FedSA-rsLoRA and FedSA-VeRA. In this way, we establish a general paradigm for integrating LoRA with FL, offering guidance for future work on subsequent LoRA variants combined with FL. Extensive experimental results on natural language understanding and generation tasks demonstrate the effectiveness of the proposed method. Our code is available at https://github.com/Pengxin-Guo/FedSA-LoRA.

Key facts. Cited by 80 papers in the map (data through 14 Oct 2026). Builds on 61 of its 73 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

61 of 73 references.

References · 61

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  11. 2019Advances and Open Problems in Federated Learningcited by 3,046
  12. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
38 more
  1. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  2. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  3. 2018Federated Learning with Non-IID Datacited by 1,108
  4. 2016An overview of gradient descent optimization algorithmscited by 1,033
  5. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  6. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  7. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  8. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 654
  9. 2020Personalized Federated Learning with Moreau Envelopescited by 618
  10. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  11. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  12. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  13. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 370
  14. 2021Federated Learning on Non-IID Data: A Surveycited by 346
  15. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
  16. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 243
  17. 2019Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computationscited by 181
  18. 2020FetchSGD: Communication-Efficient Federated Learning with Sketchingcited by 179
  19. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  20. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 149
  21. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
  22. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 137
  23. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 136
  24. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 134
  25. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 123
  26. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
  27. 2021Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learningcited by 111
  28. 2023Personalized Federated Learning with Feature Alignment and Classifier Collaborationcited by 97
  29. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 86
  30. 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 74
  31. 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
  32. 2022Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learningcited by 71
  33. 2023Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimizationcited by 65
  34. 2024SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 62
  35. 2024FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuningcited by 59
  36. 2024Dual-Personalizing Adapter for Federated Foundation Modelscited by 55
  37. 2024Global and Local Prompts Cooperation via Optimal Transport for Federated Learningcited by 46
  38. 2023Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learningcited by 41

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 30; 2026: 46.

80 papers.

Sort:

By area:

  1. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 17
  2. 2025Decentralized Low-Rank Fine-Tuning of Large Language Modelscited by 14
  3. 2025FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRAcited by 13
  4. 2024Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Modelscited by 12
  5. 2025FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRAcited by 11
  6. 2025Exploring the Vulnerabilities of Federated Learning: A Deep Dive into Gradient Inversion Attackscited by 9
  7. 2025Federated Low-Rank Adaptation for Foundation Models: A Surveycited by 9
  8. 2025Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuningcited by 9
  9. 2026FedRot-LoRA: Mitigating Rotational Misalignment in Federated LoRAcited by 8
  10. 2024A New Federated Learning Framework Against Gradient Inversion Attackscited by 6
  11. 2025FedVLMBench: Benchmarking Federated Fine-Tuning of Vision-Language Modelscited by 6
  12. 2025Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuningcited by 6
38 more
  1. 2026FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignmentcited by 5
  2. 2026Rethinking LoRA for Privacy-Preserving Federated Learning in Large Modelscited by 5
  3. 2025A Shared Low-Rank Adaptation Approach to Personalized RLHFcited by 4
  4. 2025Learning Like Humans: Resource-Efficient Federated Fine-Tuning through Cognitive Developmental Stagescited by 4
  5. 2025ADF-LoRA: Alternating Low-Rank Aggregation for Decentralized Federated Fine-Tuningcited by 4
  6. 2024Fed-pilot: Optimizing LoRA Allocation for Efficient Federated Fine-Tuning with Heterogeneous Clientscited by 3
  7. 2025A Survey on Foundation Models for Personalized Federated Intelligencecited by 3
  8. 2025Converge Faster, Talk Less: Hessian-Informed Federated Zeroth-Order Optimizationcited by 3
  9. 2025Federated Sketching LoRA: A Flexible Framework for Heterogeneous Collaborative Fine-Tuning of LLMscited by 2
  10. 2025Exploring Federated Pruning for Large Language Modelscited by 2
  11. 2025DeCAF: Decentralized Consensus-And-Factorization for Low-Rank Adaptation of Foundation Modelscited by 2
  12. 2025FLEx: Personalized Federated Learning for Mixture-of-Experts LLMs via Expert Graftingcited by 2
  13. 2025Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrixcited by 2
  14. 2025Convergence Analysis of Aggregation-Broadcast in LoRA-enabled Distributed Fine-Tuningcited by 2
  15. 2026FedKRSO: Communication and Memory Efficient Federated Fine-Tuning of Large Language Modelscited by 2
  16. 2026FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuningcited by 2
  17. 2026Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRAcited by 2
  18. 2024Sequential Compression Layers for Efficient Federated Learning in Foundational Modelscited by 1
  19. 2025FedReFT: Federated Representation Fine-Tuning with All-But-Me Aggregationcited by 1
  20. 2025FediLoRA: Practical Federated Fine-Tuning of Foundation Models Under Missing-Modality Constraintscited by 1
  21. 2025SEQR: Secure and Efficient QR-based LoRA Routingcited by 1
  22. 2025Communication-Efficient and Accurate Approach for Aggregation in Federated Low-Rank Adaptationcited by 1
  23. 2026SDFLoRA: Selective Decoupled Federated LoRA for Privacy-preserving Fine-tuning with Heterogeneous Clientscited by 1
  24. 2026Toward Federated Large Language Models in Medicine: A Parameter-Efficient Framework for Privacy-Preserving, Multi-Institutional Adaptationcited by 1
  25. 2026Preventing Rank Collapse in Federated Low-Rank Adaptation with Client Heterogeneitycited by 1
  26. 2026Stabilized Fine-Tuning with LoRA in Federated Learning: Mitigating the Side Effect of Client Size and Rank via the Scaling Factorcited by 1
  27. 2026Task-Centric Personalized Federated Fine-Tuning of Language Modelscited by 1
  28. 2026Fine-Tuning Impairs the Balancedness of Foundation Models in Long-tailed Personalized Federated Learningcited by 1
  29. 2026SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Rankscited by 1
  30. 2026FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRAcited by 1
  31. 2026RW-LoRA: Communication-Efficient Decentralized LoRA Fine-Tuning via Random Walkscited by 1
  32. 2025FedSCA: Federated Tuning with Similarity-guided Collaborative Aggregation for Heterogeneous Medical Image Segmentationcited by 0
  33. 2025FLAME: Towards Federated Fine-Tuning Large Language Models Through Adaptive SMoEcited by 0
  34. 2025Exploring and Reshaping the Weight Distribution in LLMcited by 0
  35. 2025Bi-level Personalization for Federated Foundation Models: A Task-vector Aggregation Approachcited by 0
  36. 2025Personalized Federated Fine-Tuning of Vision Foundation Models for Healthcarecited by 0
  37. 2025OvA-LP: A Simple and Efficient Framework for Federated Learning on Non-IID Datacited by 0
  38. 2026Stabilizing Decentralized Federated Fine-Tuning via Topology-Aware Alternating LoRAcited by 0

50 most cited of 80 All 80 (JSON)