mlsanityData to 7 Oct 2026

ResLoRA: Identity Residual Mapping in Low-Rank Adaption

arXiv 2402.18039 ↗First posted 28 Feb 2024cs.CL cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

As one of the most popular parameter-efficient fine-tuning (PEFT) methods, low-rank adaptation (LoRA) is commonly applied to fine-tune large language models (LLMs). However, updating the weights of LoRA blocks effectively and expeditiously is challenging due to the long calculation path in the original model. To address this, we propose ResLoRA, an improved framework of LoRA. By adding residual paths during training and using merging approaches to eliminate these extra paths during inference, our method can achieve better results in fewer training steps without any extra trainable parameters or inference cost compared to LoRA. The experiments on NLG, NLU, and text-to-image tasks demonstrate the effectiveness of our method. To the best of our knowledge, ResLoRA is the first work that combines the residual path with LoRA. The code of our method is available at https://github.com/microsoft/LMOps/tree/main/reslora .

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 30 of its 44 references that are in the map. First posted on arXiv 28 Feb 2024.

Builds on

30 of 44 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  9. 2016Densely Connected Convolutional Networkscited by 6,691
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
18 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  3. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  4. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  5. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  6. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  7. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  8. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  9. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  10. 2021RepVGG: Making VGG-style ConvNets Great Againcited by 520
  11. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  12. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  13. 2015Highway Networkscited by 257
  14. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  15. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 140
  16. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  17. 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluationcited by 108
  18. 2023The Expressive Power of Low-Rank Adaptationcited by 100

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 5; 2026: 3.

18 papers.

Sort:

By area:

  1. 2024LoRA Learns Less and Forgets Lesscited by 313
  2. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  3. 2024A Survey on LoRA of Large Language Modelscited by 67
  4. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 44
  5. 2024MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuningcited by 31
  6. 2024MTL-LoRA: Low-Rank Adaptation for Multi-Task Learningcited by 16
  7. 2024Propulsion: Steering LLM with Tiny Fine-Tuningcited by 14
  8. 2025UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter-Efficient Fine-Tuning of Large Modelscited by 11
  9. 2024Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuningcited by 10
  10. 2025A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Modelscited by 9
  11. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9
  12. 2024PETALface: Parameter Efficient Transfer Learning for Low-resolution Face Recognitioncited by 7
6 more
  1. 2025LoR2C : Low-Rank Residual Connection Adaptation for Parameter-Efficient Fine-Tuningcited by 4
  2. 2025zFLoRA: Zero-Latency Fused Low-Rank Adapterscited by 1
  3. 2026Structural Correspondence and Universal Approximation in Diagonal plus Low-Rank Neural Networkscited by 1
  4. 2024Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Modelscited by 0
  5. 2025FLoRA: Fused forward-backward adapters for parameter efficient fine-tuning and reducing inference-time latencies of LLMscited by 0
  6. 2026Move What Matters: Parameter-Efficient Domain Adaptation via Optimal Transport Flow for Collaborative Perceptioncited by 0