mlsanityData to 14 Oct 2026

DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution

arXiv 2405.17357 ↗First posted 27 May 2024cs.CLcited by 43

Not on the map. Why?

Abstract (arXiv)

Fine-tuning large-scale pre-trained models is inherently a resource-intensive task. While it can enhance the capabilities of the model, it also incurs substantial computational costs, posing challenges to the practical application of downstream tasks. Existing parameter-efficient fine-tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) rely on a bypass framework that ignores the differential parameter budget requirements across weight matrices, which may lead to suboptimal fine-tuning outcomes. To address this issue, we introduce the Dynamic Low-Rank Adaptation (DoRA) method. DoRA decomposes high-rank LoRA layers into structured single-rank components, allowing for dynamic pruning of parameter budget based on their importance to specific tasks during training, which makes the most of the limited parameter budget. Experimental results demonstrate that DoRA can achieve competitive performance compared with LoRA and full model fine-tuning, and outperform various strong baselines with the same storage parameter budget. Our code is available at https://github.com/MIkumikumi0116/DoRA

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Builds on 41 of its 61 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

41 of 61 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  11. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
29 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  3. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  5. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  6. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  7. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  8. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  9. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  10. 2018Neural Network Acceptability Judgmentscited by 958
  11. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  12. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  13. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  14. 2017Learning multiple visual domains with residual adapterscited by 557
  15. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
  16. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  17. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  18. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  19. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 233
  20. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  21. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  22. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  23. 2021Counter-Interference Adapter for Multilingual Machine Translationcited by 40
  24. 2021MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translatorscited by 35
  25. 2022Efficient Fine-Tuning of BERT Models on the Edgecited by 29
  26. 2023Is Fine-tuning Needed? Pre-trained Language Models Are Near Perfect for Out-of-Domain Detectioncited by 29
  27. 2021NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMMcited by 19
  28. 2023One Network, Many Masks: Towards More Parameter-Efficient Transfer Learningcited by 18
  29. 2024Mixture of Latent Experts Using Tensor Productscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 23; 2026: 11.

43 papers.

Sort:

By area:

  1. 2024A Survey on LoRA of Large Language Modelscited by 67
  2. 2024Unlearning Backdoor Attacks for LLMs with Weak-to-Strong Knowledge Distillationcited by 27
  3. 2025AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mappingcited by 23
  4. 2024Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clientscited by 22
  5. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  6. 2025Sensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language Modelscited by 14
  7. 2024Flexora: Flexible Low Rank Adaptation for Large Language Modelscited by 13
  8. 2024WildSAT: Learning Satellite Image Representations from Wildlife Observationscited by 13
  9. 2025HyperAdaLoRA: Accelerating LoRA Rank Allocation During Training via Hypernetworks without Sacrificing Performancecited by 10
  10. 2025Dynamical Low-Rank Compression of Neural Networks with Robustness under Adversarial Attackscited by 6
  11. 2024GeoLoRA: Geometric integration for parameter efficient fine-tuningcited by 5
  12. 2025ElaLoRA: Elastic & Learnable Low-Rank Adaptation for Efficient Model Fine-Tuningcited by 5
31 more
  1. 2025A geometric framework for momentum-based optimizers for low-rank trainingcited by 5
  2. 2025ARD-LoRA: Dynamic Rank Allocation for Parameter-Efficient Fine-Tuning of Foundation Models with Heterogeneous Adaptation Needscited by 5
  3. 2026FlexLoRA: Entropy-Guided Flexible Low-Rank Adaptationcited by 5
  4. 2025WeightLoRA: Keep Only Necessary Adapterscited by 4
  5. 2025PFMBench: Protein Foundation Model Benchmarkcited by 4
  6. 2026Post-Optimization Adaptive Rank Allocation for LoRAcited by 4
  7. 2025Parameter Efficient Continual Learning with Dynamic Low-Rank Adaptationcited by 3
  8. 2025Few-Shot Segmentation of Historical Maps via Linear Probing of Vision Foundation Modelscited by 3
  9. 2026Decomposing and Composing: Towards Efficient Vision-Language Continual Learning via Rank-1 Expert Pool in a Single LoRAcited by 3
  10. 2026Localized LoRA-MoE: Block-wise Low-Rank Experts With Adaptive Routingcited by 3
  11. 2024A Bayesian Interpretation of Adaptive Low-Rank Adaptationcited by 2
  12. 2025DiffoRA: Enabling Parameter-Efficient Fine-Tuning via Differential Module Selectioncited by 2
  13. 2025HaLoRA: Hardware-aware Low-Rank Adaptation for Large Language Models Based on Hybrid Compute-in-Memory Architecturecited by 2
  14. 2025MLLP-VRAIN UPV system for the IWSLT 2025 Simultaneous Speech Translation Translation taskcited by 2
  15. 2025StyleQoRA: Quality-Aware Low-Rank Adaptation for Few-Shot Multi-Style Editingcited by 2
  16. 2026A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluationcited by 2
  17. 2024Dynamic Adapter with Semantics Disentangling for Cross-lingual Cross-modal Retrievalcited by 1
  18. 2025AROMA: Autonomous Rank-one Matrix Adaptationcited by 1
  19. 2025Localized LoRA: A Structured Low-Rank Approximation for Efficient Fine-Tuningcited by 1
  20. 2025Performance Trade-offs of Optimizing Small Language Models for E-Commercecited by 1
  21. 2024Accurate and Efficient Fine-Tuning of Quantized Large Language Models Through Optimal Balancecited by 0
  22. 2025TwinTURBO: Semi-Supervised Fine-Tuning of Foundation Models via Mutual Information Decompositions for Downstream Task and Latent Spacescited by 0
  23. 2025Progtuning: Progressive Fine-tuning Framework for Transformer-based Language Modelscited by 0
  24. 2025AILoRA: Function-Aware Asymmetric Initialization for Low-Rank Adaptation of Large Language Modelscited by 0
  25. 2025Collaborative Low-Rank Adaptation for Pre-Trained Vision Transformerscited by 0
  26. 2026Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptationcited by 0
  27. 2026Trade-offs in Medical LLM Adaptation: An Empirical Study in French QAcited by 0
  28. 2026LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuningcited by 0
  29. 2026What Does the Rank Buy? A Spectral and Distributional Analysis of Low-Rank Adaptationcited by 0
  30. 2026Resource-Aware Parameter-Efficient Model Adaptation for Onboard High-Dimensional Datacited by 0
  31. 2026Adaptive Utilization of Low-Rank Adaptation via Conditioned Gatingcited by 0