mlsanityData to 14 Oct 2026

DeLoRA: Decoupling Angles and Strength in Low-rank Adaptation

arXiv 2503.18225 ↗First posted 23 Mar 2025cs.LG cs.CL cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Parameter-Efficient FineTuning (PEFT) methods have recently gained significant popularity thanks to the widespread availability of large-scale pretrained models. These methods allow for quick adaptation to downstream tasks with minimal computational cost. However, popular finetuning methods such as LoRA exhibit limited robustness when it comes to hyperparameter choices or extended training regimes, preventing optimal out-of-the-box performance. In contrast, bounded approaches, such as ETHER, provide greater robustness but are limited to extremely low-rank adaptations and fixed-strength transformations, reducing their adaptation expressive power. In this work, we propose Decoupled Low-rank Adaptation (DeLoRA), a novel finetuning method that normalizes and scales learnable low-rank matrices. By bounding the distance of the transformation, DeLoRA effectively decouples the angular learning from the adaptation strength, enhancing robustness without compromising performance. Through evaluations on subject-driven image generation, natural language understanding, and instruction tuning, we show that DeLoRA matches or surpasses performance of competing PEFT methods, while exhibiting stronger robustness. Code is available at https://github.com/ExplainableML/DeLoRA.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 40 of its 45 references that are in the map. First posted on arXiv 23 Mar 2025.

Builds on

40 of 45 references.

References · 40

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
28 more
  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  3. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  4. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  5. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  6. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  7. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  8. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  9. 2021GPT Understands, Toocited by 716
  10. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  11. 2024LoRA Learns Less and Forgets Lesscited by 328
  12. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  13. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 208
  14. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  15. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 114
  16. 2023Sparse Low-rank Adaptation of Pre-trained Language Modelscited by 96
  17. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 94
  18. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 86
  19. 2024Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learningcited by 82
  20. 2023Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matricescited by 74
  21. 2024Parameter-Efficient Fine-Tuning with Discrete Fourier Transformcited by 71
  22. 2023Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuningcited by 64
  23. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 54
  24. 2024Advancing Parameter Efficiency in Fine-tuning via Representation Editingcited by 37
  25. 2024ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Modelscited by 36
  26. 2024AutoLoRA: Automatically Tuning Matrix Ranks in Low-Rank Adaptation Based on Meta Learningcited by 31
  27. 2024MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuningcited by 19
  28. 2024ETHER: Efficient Finetuning of Large-Scale Models with Hyperplane Reflectionscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025LoFT: Low-Rank Adaptation That Behaves Like Full Fine-Tuningcited by 14
  2. 2025StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifoldcited by 14
  3. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 10
  4. 2025Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptationcited by 4
  5. 2025Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Modelscited by 3
  6. 2026High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Modelcited by 3
  7. 2026A Language-Guided Bayesian Optimization for Efficient LoRA Hyperparameter Searchcited by 2
  8. 2026TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language Modelscited by 1
  9. 2026ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuningcited by 1
  10. 2026AntiPaSTO: Self-Supervised Honesty Steering via Anti-Parallel Representationscited by 0