# Selective Aggregation for Low-Rank Adaptation in Federated Learning

arXiv: https://arxiv.org/abs/2410.01463 · HTML: https://mlsanity.com/paper/2410.01463/

## Key facts

- Cited by 80 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 73 references that are in the map.
- First posted on arXiv 2 Oct 2024.

## Builds on (most cited first, 30 of 61)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [Personalized Federated Learning with Moreau Envelopes](https://mlsanity.com/paper/2006.08848/) (2020)
- [Federated Learning on Non-IID Data Silos: An Experimental Study](https://mlsanity.com/paper/2102.02079/) (2021)
- [Exploiting Shared Representations for Personalized Federated Learning](https://mlsanity.com/paper/2102.07078/) (2021)
- [LoRA+: Efficient Low Rank Adaptation of Large Models](https://mlsanity.com/paper/2402.12354/) (2024)
- [Measuring the Intrinsic Dimension of Objective Landscapes](https://mlsanity.com/paper/1804.08838/) (2018)
- [Federated Learning on Non-IID Data: A Survey](https://mlsanity.com/paper/2106.06843/) (2021)
- [Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning](https://mlsanity.com/paper/1807.06629/) (2018)
- [A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA](https://mlsanity.com/paper/2312.03732/) (2023)
- [Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computations](https://mlsanity.com/paper/1906.02367/) (2019)
- [FetchSGD: Communication-Efficient Federated Learning with Sketching](https://mlsanity.com/paper/2007.07682/) (2020)

## Built on by (most cited first, 30 of 80)

- [A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learning](https://mlsanity.com/paper/2504.21099/) (2025)
- [Decentralized Low-Rank Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2501.15361/) (2025)
- [FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRA](https://mlsanity.com/paper/2503.11880/) (2025)
- [Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2410.13097/) (2024)
- [FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA](https://mlsanity.com/paper/2505.12805/) (2025)
- [Exploring the Vulnerabilities of Federated Learning: A Deep Dive into Gradient Inversion Attacks](https://mlsanity.com/paper/2503.11514/) (2025)
- [Federated Low-Rank Adaptation for Foundation Models: A Survey](https://mlsanity.com/paper/2505.13502/) (2025)
- [Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuning](https://mlsanity.com/paper/2506.05568/) (2025)
- [FedRot-LoRA: Mitigating Rotational Misalignment in Federated LoRA](https://mlsanity.com/paper/2602.23638/) (2026)
- [A New Federated Learning Framework Against Gradient Inversion Attacks](https://mlsanity.com/paper/2412.07187/) (2024)
- [FedVLMBench: Benchmarking Federated Fine-Tuning of Vision-Language Models](https://mlsanity.com/paper/2506.09638/) (2025)
- [Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning](https://mlsanity.com/paper/2509.15087/) (2025)
- [FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment](https://mlsanity.com/paper/2602.17095/) (2026)
- [Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models](https://mlsanity.com/paper/2602.19926/) (2026)
- [A Shared Low-Rank Adaptation Approach to Personalized RLHF](https://mlsanity.com/paper/2503.19201/) (2025)
- [Learning Like Humans: Resource-Efficient Federated Fine-Tuning through Cognitive Developmental Stages](https://mlsanity.com/paper/2508.00041/) (2025)
- [ADF-LoRA: Alternating Low-Rank Aggregation for Decentralized Federated Fine-Tuning](https://mlsanity.com/paper/2511.18291/) (2025)
- [Fed-pilot: Optimizing LoRA Allocation for Efficient Federated Fine-Tuning with Heterogeneous Clients](https://mlsanity.com/paper/2410.10200/) (2024)
- [A Survey on Foundation Models for Personalized Federated Intelligence](https://mlsanity.com/paper/2505.06907/) (2025)
- [Converge Faster, Talk Less: Hessian-Informed Federated Zeroth-Order Optimization](https://mlsanity.com/paper/2506.02370/) (2025)
- [Federated Sketching LoRA: A Flexible Framework for Heterogeneous Collaborative Fine-Tuning of LLMs](https://mlsanity.com/paper/2501.19389/) (2025)
- [Exploring Federated Pruning for Large Language Models](https://mlsanity.com/paper/2505.13547/) (2025)
- [DeCAF: Decentralized Consensus-And-Factorization for Low-Rank Adaptation of Foundation Models](https://mlsanity.com/paper/2505.21382/) (2025)
- [FLEx: Personalized Federated Learning for Mixture-of-Experts LLMs via Expert Grafting](https://mlsanity.com/paper/2506.00965/) (2025)
- [Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix](https://mlsanity.com/paper/2507.09990/) (2025)
- [Convergence Analysis of Aggregation-Broadcast in LoRA-enabled Distributed Fine-Tuning](https://mlsanity.com/paper/2508.01348/) (2025)
- [FedKRSO: Communication and Memory Efficient Federated Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2602.03019/) (2026)
- [FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning](https://mlsanity.com/paper/2603.13282/) (2026)
- [Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRA](https://mlsanity.com/paper/2605.06733/) (2026)
- [Sequential Compression Layers for Efficient Federated Learning in Foundational Models](https://mlsanity.com/paper/2412.07021/) (2024)

## Abstract (arXiv)

We investigate LoRA in federated learning through the lens of the asymmetry analysis of the learned A and B matrices. In doing so, we uncover that A matrices are responsible for learning general knowledge, while B matrices focus on capturing client-specific knowledge. Based on this finding, we introduce Federated Share-A Low-Rank Adaptation (FedSA-LoRA), which employs two low-rank trainable matrices A and B to model the weight update, but only A matrices are shared with the server for aggregation. Moreover, we delve into the relationship between the learned A and B matrices in other LoRA variants, such as rsLoRA and VeRA, revealing a consistent pattern. Consequently, we extend our FedSA-LoRA method to these LoRA variants, resulting in FedSA-rsLoRA and FedSA-VeRA. In this way, we establish a general paradigm for integrating LoRA with FL, offering guidance for future work on subsequent LoRA variants combined with FL. Extensive experimental results on natural language understanding and generation tasks demonstrate the effectiveness of the proposed method. Our code is available at https://github.com/Pengxin-Guo/FedSA-LoRA.