Adaptive Selection of LoRA Components in Privacy-Preserving Federated Learning
Not on the map. Why?
Abstract (arXiv)
Differentially private federated fine-tuning of large models with LoRA suffers from aggregation error caused by LoRA's multiplicative structure, which is further amplified by DP noise and degrades both stability and accuracy. Existing remedies apply a single update mode uniformly across all layers and all communication rounds (or alternate them on a fixed schedule), ignoring both the structural asymmetry between the two LoRA factors and the round-wise dynamics of training. We propose AS-LoRA, an adaptive framework defined by three axes (i) layer-wise freedom, in which each layer independently selects its active component, (ii) round-wise adaptivity, in which the selection updates over communication rounds, and (iii) a curvature-aware score derived from a second-order approximation of the loss. Theoretically, AS-LoRA eliminates the reconstruction-error floor of layer-tied schedules, accelerates convergence, implicitly biases solutions toward flatter minima, and incurs no additional privacy cost. Across GLUE, SQuAD, CIFAR-100, and Tiny-ImageNet under strict DP budgets and non-IID partitions, AS-LoRA improves over the federated LoRA baselines by up to +7.5 pp on GLUE and +12.5 pp on MNLI-mm for example, while matching or exceeding SVD-based aggregation methods at 33--180 × lower aggregation cost and with negligible communication overhead. Code for the proposed method is available at https://anonymous.4open.science/r/aslora-F75F/.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 48 of its 61 references that are in the map. First posted on arXiv 7 May 2026.
Builds on
48 of 61 references.
References · 48
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2023GPT-4 Technical Reportcited by 18,396
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
- 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
- 2016Deep Learning with Differential Privacycited by 2,728
- 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
36 more
- 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
- 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
- 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
- 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
- 2023PaLM 2 Technical Reportcited by 1,162
- 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
- 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
- 2019Deep Leakage from Gradientscited by 1,021
- 2019Towards Federated Learning at Scale: System Designcited by 832
- 2017Renyi Differential Privacycited by 650
- 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
- 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 383
- 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
- 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
- 2018Subsampled R'enyi Differential Privacy and Analytical Moments Accountantcited by 236
- 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 233
- 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
- 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
- 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
- 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
- 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
- 2020Local and Central Differential Privacy for Robustness and Privacy in Federated Learningcited by 78
- 2022When Do Flat Minima Optimizers Work?cited by 78
- 2024Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 49
- 2023pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuningcited by 40
- 2021Federated f-Differential Privacycited by 29
- 2024Federated LoRA with Sparse Communicationcited by 26
- 2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 25
- 2024Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clientscited by 22
- 2024FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Modelscited by 18
- 2025Robust Federated Finetuning of LLMs via Alternating Optimization of LoRAcited by 17
- 2019DP-LSSGD: A Stochastic Optimization Method to Lift the Utility in Privacy-Preserving ERMcited by 15
- 2025FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRAcited by 11
- 2025GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuningcited by 10
- 2020Differentially Private Federated Learning with Laplacian Smoothingcited by 8
- 2026Rethinking LoRA for Privacy-Preserving Federated Learning in Large Modelscited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.