DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networks
Not on the map. Why?
Abstract (arXiv)
Parameter-efficient fine-tuning (PEFT) methods have become the standard paradigm for adapting large-scale models. Among these techniques, Weight-Decomposed Low-Rank Adaptation (DoRA) has been shown to improve both the learning capacity and training stability of the Low-Rank Adaptation (LoRA) method by explicitly decomposing pre-trained weights into magnitude and directional components. In this work, we propose DoRAN, a new technique designed to stabilize training and boost the sample efficiency of DoRA. Our framework introduces two key components: (i) the injection of learnable noise into the denominator of DoRA weight decomposition, which serves as an adaptive regularizer to mitigate instabilities and improve the estimation rate of low-rank matrices; and (ii) the replacement of static low-rank matrices with auxiliary networks that generate them dynamically, enabling parameter coupling between the query and value projection matrices, leading to improved sample efficiency both theoretically and empirically. Comprehensive experiments on vision and language benchmarks show that DoRAN consistently outperforms LoRA, DoRA, and other PEFT baselines, underscoring the effectiveness of combining noise-based regularization with network-based parameter generation.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 49 of its 60 references that are in the map. First posted on arXiv 5 Oct 2025.
Builds on
49 of 60 references.
References · 49
- 2017Attention Is All You Needcited by 60,361
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2022Emergent Abilities of Large Language Modelscited by 2,512
- 2022Visual Prompt Tuningcited by 1,681
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
37 more
- 2016HyperNetworkscited by 1,027
- 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
- 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
- 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
- 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
- 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
- 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
- 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
- 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
- 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 284
- 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
- 2019Continual learning with hypernetworkscited by 273
- 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
- 2021Hash Layers For Large Sparse Modelscited by 237
- 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 233
- 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 112
- 2022StableMoE: Stable Routing Strategy for Mixture of Expertscited by 100
- 2016Learning feed-forward one-shot learnerscited by 93
- 2023Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterizationcited by 64
- 2024MoH: Multi-Head Attention as Mixture-of-Head Attentioncited by 44
- 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 37
- 2023SwitchHead: Accelerating Transformers with Mixture-of-Experts Attentioncited by 34
- 2023Demystifying Softmax Gating Function in Gaussian Mixture of Expertscited by 33
- 2024AutoLoRA: Automatically Tuning Matrix Ranks in Low-Rank Adaptation Based on Meta Learningcited by 31
- 2023Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Expertscited by 29
- 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Expertscited by 24
- 2024On Least Square Estimation in Softmax Gating Mixture of Expertscited by 23
- 2022Refined Convergence Rates for Maximum Likelihood Estimation under Finite Mixture Modelscited by 22
- 2024On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functionscited by 16
- 2024Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Promptscited by 15
- 2024DP-DyLoRA: Fine-Tuning Transformer-Based Models On-Device under Differentially Private Federated Learning using Dynamic Low-Rank Adaptationcited by 9
- 2024MLAE: Masked LoRA Experts for Visual Parameter-Efficient Fine-Tuningcited by 6
- 2025RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Expertscited by 6
- 2024Improving Generalization with Flat Hilbert Bayesian Inferencecited by 5
- 2025Revisit Visual Prompt Tuning: The Expressiveness of Prompt Expertscited by 5
- 2025On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimationcited by 4
- 2025Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Modelscited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.