mlsanityData to 7 Oct 2026

DoRA: Weight-Decomposed Low-Rank Adaptation

arXiv 2402.09353 ↗First posted 14 Feb 2024cs.CL cs.CVcited by 740

Not on the map. Why?

Cited by 46 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Among the widely used parameter-efficient fine-tuning (PEFT) methods, LoRA and its variants have gained considerable popularity because of avoiding additional inference costs. However, there still often exists an accuracy gap between these methods and full fine-tuning (FT). In this work, we first introduce a novel weight decomposition analysis to investigate the inherent differences between FT and LoRA. Aiming to resemble the learning capacity of FT from the findings, we propose Weight-Decomposed Low-Rank Adaptation (DoRA). DoRA decomposes the pre-trained weight into two components, magnitude and direction, for fine-tuning, specifically employing LoRA for directional updates to efficiently minimize the number of trainable parameters. By employing ours, we enhance both the learning capacity and training stability of LoRA while avoiding any additional inference overhead. ours~consistently outperforms LoRA on fine-tuning LLaMA, LLaVA, and VL-BART on various downstream tasks, such as commonsense reasoning, visual instruction tuning, and image/video-text understanding. Code is available at https://github.com/NVlabs/DoRA.

Key facts. Cited by 740 papers in the map (data through 7 Oct 2026). Builds on 50 of its 59 references that are in the map. First posted on arXiv 14 Feb 2024.

Builds on

50 of 59 references.

References · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
38 more
  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  7. 2019Towards VQA Models That Can Readcited by 1,696
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  10. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  11. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  12. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  13. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  14. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  15. 2023Instruction Tuning with GPT-4cited by 676
  16. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  17. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  18. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  19. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  20. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  21. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  22. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  23. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  24. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  25. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  26. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  27. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  28. 2020TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrievalcited by 215
  29. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 207
  30. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 140
  31. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  32. 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluationcited by 108
  33. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  34. 2024Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learningcited by 80
  35. 2023Tied-Lora: Enhancing parameter efficiency of LoRA with weight tyingcited by 65
  36. 2023Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterizationcited by 64
  37. 2022Combining Modular Skills in Multitask Learningcited by 27
  38. 2023Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modelingcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 201; 2025: 317; 2026: 222.

740 papers.

Sort:

By area:

  1. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  2. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  3. 2024What matters when building vision-language models?cited by 321
  4. 2024LoRA Learns Less and Forgets Lesscited by 313
  5. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  6. 2025Small Language Models are the Future of Agentic AIcited by 254
  7. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  8. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  9. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  10. 2024An Introduction to Vision-Language Modelingcited by 175
  11. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  12. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
38 more
  1. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  2. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  3. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 102
  4. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  5. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 89
  6. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  7. 2024OpenELM: An Efficient Language Model Family with Open Training and Inference Frameworkcited by 74
  8. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameterscited by 73
  9. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?cited by 70
  10. 2024A Survey on LoRA of Large Language Modelscited by 67
  11. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 65
  12. 2024Prompt Compression for Large Language Models: A Surveycited by 65
  13. 2025SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learningcited by 60
  14. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 53
  15. 2024(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Textscited by 49
  16. 2024Still-Moving: Customized Video Generation without Customized Video Datacited by 49
  17. 2024Mixture-of-Subspaces in Low-Rank Adaptationcited by 47
  18. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 44
  19. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  20. 2024WebCode2M: A Real-World Dataset for Code Generation from Webpage Designscited by 43
  21. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  22. 2024DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distributioncited by 41
  23. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 41
  24. 2025SaLoRA: Safety-Alignment Preserved Low-Rank Adaptationcited by 39
  25. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 36
  26. 2024Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMscited by 36
  27. 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 35
  28. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  29. 2025RandLoRA: Full-rank parameter-efficient fine-tuning of large modelscited by 34
  30. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 34
  31. 2024MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuningcited by 31
  32. 2024Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Modelscited by 31
  33. 2024Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMscited by 31
  34. 2024LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimizationcited by 31
  35. 2025Accurate and Efficient Low-Rank Model Merging in Core Spacecited by 28
  36. 2024FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decompositioncited by 26
  37. 2024Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Passcited by 26
  38. 2024To Forget or Not? Towards Practical Knowledge Unlearning for Large Language Modelscited by 25

50 most cited of 740 All 740 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition
  2. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
  3. 2026ID-LoRA: Efficient Low-Rank Adaptation Inspired by Matrix Interpolative Decomposition
  4. 2025AFA-LoRA: Enabling Non-Linear Adaptations in LoRA with Activation Function Annealing
  5. 2024VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks
  6. 2025GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning
  7. 2026SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
  8. 2026Echo-LoRA: Parameter-Efficient Fine-Tuning via Cross-Layer Representation Injection