SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Ranks
Not on the map. Why?
Abstract (arXiv)
We consider federated parameter efficient fine-tuning of large neural networks with low-rank adaptation (LoRA,~Hu et al. 2022). Combining LoRA with federated PEFT introduces challenges absent from either setting alone: clients may use different LoRA ranks, making their factor matrices dimension-incompatible, and factor-wise averaging suffers from a bilinear mismatch. We propose SeFoRA, a sketch-aggregated federated LoRA algorithm in which each client transmits a linear sketch of its local updates, enabling direct aggregation at the federator. As a result, SeFoRA alleviates the bilinear mismatch, and allows for aggregation in a small subspace of the full model. We introduce a rank-homogeneous version called SeFoRA-Ho which allows for direct adapter aggregation in this setting. We prove convergence to a neighborhood of the first-order stationary point at rate cO(1/T) for the rank-homogeneous setting. Numerical experiments on fine-tuning RoBERTa-Large on GLUE datasets show how our algorithms outperform the state-of-the-art.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 74 of its 86 references that are in the map. First posted on arXiv 10 Aug 2026.
Builds on
74 of 86 references.
References · 74
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2023GPT-4 Technical Reportcited by 18,396
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
- 2019Advances and Open Problems in Federated Learningcited by 3,046
- 2022Large Language Models Encode Clinical Knowledgecited by 1,830
38 more
- 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
- 2023PaLM 2 Technical Reportcited by 1,162
- 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
- 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
- 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
- 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
- 2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 374
- 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
- 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 233
- 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
- 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
- 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 149
- 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
- 2021Federated Learning for Open Bankingcited by 141
- 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 128
- 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 123
- 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
- 2023Sparse Low-rank Adaptation of Pre-trained Language Modelscited by 96
- 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 93
- 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
- 2022FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learningcited by 91
- 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
- 2023FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Modelscited by 63
- 2023Parameter-Efficient Fine-Tuning without Introducing New Latencycited by 59
- 2024FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuningcited by 59
- 2023Fair Federated Medical Image Segmentation via Client Contribution Estimationcited by 52
- 2023Learning Federated Visual Prompt in Null Space for MRI Reconstructioncited by 40
- 2016Practical sketching algorithms for low-rank matrix approximationcited by 38
- 2024LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluationcited by 38
- 2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 38
- 2021Specificity-Preserving Federated Learning for MR Image Reconstructioncited by 35
- 2024FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Modelscited by 30
- 2025FedSWA: Improving Generalization in Federated Learning with Highly Heterogeneous Data via Momentum-Based Stochastic Controlled Weight Averagingcited by 25
- 2024Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clientscited by 22
- 2025LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)cited by 20
- 2024FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Modelscited by 18
- 2023Text-driven Prompt Generation for Vision-Language Models in Federated Learningcited by 17
- 2025Robust Federated Finetuning of LLMs via Alternating Optimization of LoRAcited by 17
50 most cited of 74 All 74 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.