mlsanityData to 7 Oct 2026

Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matrices

arXiv 2309.02411 ↗First posted 5 Sep 2023cs.LGcited by 72

Not on the map. Why?

Abstract (arXiv)

In this paper, we present Delta-LoRA, which is a novel parameter-efficient approach to fine-tune large language models (LLMs). In contrast to LoRA and other low-rank adaptation methods such as AdaLoRA, Delta-LoRA not only updates the low-rank matrices bA and bB, but also propagate the learning to the pre-trained weights bW via updates utilizing the delta of the product of two low-rank matrices (bA(t+1)bB(t+1) - bA(t)bB(t)). Such a strategy effectively addresses the limitation that the incremental update of low-rank matrices is inadequate for learning representations capable for downstream tasks. Moreover, as the update of bW does not need to compute the gradients of bW and store their momentums, Delta-LoRA shares comparable memory requirements and computational costs with LoRA. Extensive experiments show that Delta-LoRA significantly outperforms existing low-rank adaptation methods. We further support these results with comprehensive analyses that underscore the effectiveness of Delta-LoRA.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Builds on 27 of its 37 references that are in the map. First posted on arXiv 5 Sep 2023.

Builds on

27 of 37 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
15 more
  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  3. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  4. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  5. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  6. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  7. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  8. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  9. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  10. 2018A Survey on Deep Transfer Learningcited by 417
  11. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  12. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  13. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  14. 2023One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuningcited by 84
  15. 2021Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigmcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 26; 2025: 27; 2026: 16.

72 papers.

Sort:

By area:

  1. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  2. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  3. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  4. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  5. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  6. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 103
  7. 2024A Survey on LoRA of Large Language Modelscited by 67
  8. 2024Riemannian Preconditioned LoRA for Fine-Tuning Foundation Modelscited by 51
  9. 2025K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAscited by 48
  10. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  11. 2024LoRA Dropout as a Sparsity Regularizer for Overfitting Controlcited by 35
  12. 2024Spectral Adapter: Fine-Tuning in Spectral Spacecited by 21
38 more
  1. 2024Parameter Efficient Fine-tuning via Explained Variance Adaptationcited by 17
  2. 2024MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuningcited by 16
  3. 2024Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuningcited by 16
  4. 2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 16
  5. 2025FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRAcited by 11
  6. 2024SVFit: Parameter-Efficient Fine-Tuning of Large Pre-Trained Models Using Singular Valuescited by 10
  7. 2025DeLoRA: Decoupling Angles and Strength in Low-rank Adaptationcited by 10
  8. 2024A Fine-tuning Enhanced RAG System with Quantized Influence Measure as AI Judgecited by 9
  9. 2025Quantum-PEFT: Ultra parameter-efficient fine-tuningcited by 9
  10. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9
  11. 2025AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projectionscited by 8
  12. 2026Why Steering Works: Toward a Unified View of Language Model Parameter Dynamicscited by 8
  13. 2024BA-LoRA: Bias-Alleviating Low-Rank Adaptation to Mitigate Catastrophic Inheritance in Large Language Modelscited by 7
  14. 2024AutoLoRA: AutoGuidance Meets Low-Rank Adaptation for Diffusion Modelscited by 7
  15. 2025Orthogonal Finetuning Made Scalablecited by 7
  16. 2025The Primacy of Magnitude in Low-Rank Adaptationcited by 7
  17. 2024KD-LoRA: A Hybrid Approach to Efficient Fine-Tuning with LoRA and Knowledge Distillationcited by 6
  18. 2025Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretrainingcited by 6
  19. 2024LORS: Low-rank Residual Structure for Parameter-Efficient Network Stackingcited by 5
  20. 2024Structured Unrestricted-Rank Matrices for Parameter Efficient Fine-tuningcited by 5
  21. 2025Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Modelscited by 5
  22. 2026Reclaiming Lost Text Layers for Source-Free Cross-Domain Few-Shot Learningcited by 5
  23. 2026IGU-LoRA: Adaptive Rank Allocation via Integrated Gradients and Uncertainty-Aware Scoringcited by 5
  24. 2024Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networkscited by 4
  25. 2025MetaTT: A Global Tensor-Train Adapter for Parameter-Efficient Fine-Tuningcited by 4
  26. 2025Efficient Compositional Multi-tasking for On-device Large Language Modelscited by 4
  27. 2026Mind the Discriminability Trap in Source-Free Cross-domain Few-shot Learningcited by 4
  28. 2024CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuningcited by 3
  29. 2025Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAscited by 3
  30. 2025K-Merge: Online Continual Merging of Adapters for On-device Large Language Modelscited by 3
  31. 2024Matrix-Transformation Based Low-Rank Adaptation (MTLoRA): A Brain-Inspired Method for Parameter-Efficient Fine-Tuningcited by 2
  32. 2024SwitchLoRA: Switched Low-Rank Adaptation Can Learn Full-Rank Informationcited by 2
  33. 2024CE-SSL: Computation-Efficient Semi-Supervised Learning for ECG-based Cardiovascular Diseases Detectioncited by 2
  34. 2025R-LoRA: Randomized Multi-Head LoRA for Efficient Multi-Task Learningcited by 2
  35. 2025SingLoRA: Low Rank Adaptation Using a Single Matrixcited by 2
  36. 2025ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraintscited by 2
  37. 2025TiMoE: Time-Aware Mixture of Language Expertscited by 2
  38. 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2

50 most cited of 72 All 72 (JSON)