mlsanityData to 7 Oct 2026

LoRA Learns Less and Forgets Less

arXiv 2405.09673 ↗First posted 15 May 2024cs.LG cs.AI cs.CLcited by 313

Not on the map. Why?

Abstract (arXiv)

Low-Rank Adaptation (LoRA) is a widely-used parameter-efficient finetuning method for large language models. LoRA saves memory by training only low rank perturbations to selected weight matrices. In this work, we compare the performance of LoRA and full finetuning on two target domains, programming and mathematics. We consider both the instruction finetuning (approximately 100K prompt-response pairs) and continued pretraining (20B unstructured tokens) data regimes. Our results show that, in the standard low-rank settings, LoRA substantially underperforms full finetuning. Nevertheless, LoRA better maintains the base model's performance on tasks outside the target domain. We show that LoRA mitigates forgetting more than common regularization techniques such as weight decay and dropout; it also helps maintain more diverse generations. Finally, we show that full finetuning learns perturbations with a rank that is 10-100X greater than typical LoRA configurations, possibly explaining some of the reported gaps. We conclude by proposing best practices for finetuning with LoRA.

Key facts. Cited by 313 papers in the map (data through 7 Oct 2026). Builds on 62 of its 89 references that are in the map. First posted on arXiv 15 May 2024.

Builds on

62 of 89 references.

References · 62

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2023Qwen Technical Reportcited by 3,273
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  8. 2020GLU Variants Improve Transformercited by 1,724
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  10. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  11. 2023StarCoder: may the source be with you!cited by 790
  12. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  13. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  14. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  15. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  16. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  17. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  18. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  19. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  20. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  21. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  22. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  23. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  24. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  25. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  26. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  27. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  28. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  29. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  30. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
  31. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  32. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
  33. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  34. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  35. 2023SantaCoder: don't reach for the stars!cited by 169
  36. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 156
  37. 2022Fine-tuned Language Models are Continual Learnerscited by 152
  38. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 151

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 64; 2025: 116; 2026: 133.

313 papers.

Sort:

By area:

  1. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  2. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 134
  3. 2025LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!cited by 132
  4. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 102
  5. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 100
  6. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 89
  7. 2024A Survey on LoRA of Large Language Modelscited by 67
  8. 2024APOLLO: SGD-like Memory, AdamW-level Performancecited by 50
  9. 2025Multi-Attribute Steering of Language Models via Targeted Interventioncited by 44
  10. 2024Mitigating Catastrophic Forgetting in Language Transfer via Model Mergingcited by 36
  11. 2024Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMscited by 36
  12. 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 36
38 more
  1. 2025Diff2Flow: Training Flow Matching Models via Diffusion Model Alignmentcited by 27
  2. 2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 24
  3. 2024AgroGPT: Efficient Agricultural Vision-Language Model with Expert Tuningcited by 23
  4. 2025PET-MAD, a lightweight universal interatomic potential for advanced materials modelingcited by 23
  5. 2025SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMscited by 23
  6. 2024Unlocking Continual Learning Abilities in Language Modelscited by 22
  7. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22
  8. 2024SoMA: Singular Value Decomposed Minor Components Adaptation for Domain Generalizable Representation Learningcited by 22
  9. 2025Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignmentcited by 22
  10. 2025AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mappingcited by 22
  11. 2025OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting during Parameter-Efficient Fine-Tuningcited by 20
  12. 2024SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixturecited by 19
  13. 2025Reinforced Refinement with Self-Aware Expansion for End-to-End Autonomous Drivingcited by 19
  14. 2025Scaling Language-Centric Omnimodal Representation Learningcited by 19
  15. 2024A Second-Order Perspective on Model Compositionality and Incremental Learningcited by 18
  16. 2024VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projectionscited by 18
  17. 2025The Rise of Small Language Models in Healthcare: A Comprehensive Surveycited by 18
  18. 2024Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradientscited by 17
  19. 2024Distillation of Diffusion Features for Semantic Correspondencecited by 17
  20. 2025Continual Learning via Sparse Memory Finetuningcited by 17
  21. 2024MTL-LoRA: Low-Rank Adaptation for Multi-Task Learningcited by 16
  22. 2025Upweighting Easy Samples in Fine-Tuning Mitigates Forgettingcited by 16
  23. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  24. 2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 16
  25. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 16
  26. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 15
  27. 2025T-LoRA: Single Image Diffusion Model Customization Without Overfittingcited by 15
  28. 2024HexaCoder: Secure Code Generation via Oracle-Guided Synthetic Training Datacited by 14
  29. 2024LLaVA-SLT: Visual Language Tuning for Sign Language Translationcited by 14
  30. 2025Put the Space of LoRA Initialization to the Extreme to Preserve Pre-trained Knowledgecited by 14
  31. 2025Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Qualitycited by 14
  32. 2024MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuningcited by 13
  33. 2024Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialectcited by 13
  34. 2025LoFT: Low-Rank Adaptation That Behaves Like Full Fine-Tuningcited by 13
  35. 2026Learning to Reason in 13 Parameterscited by 13
  36. 2026Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learningcited by 13
  37. 2024DAGER: Exact Gradient Inversion for Large Language Modelscited by 12
  38. 2024Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptationcited by 12

50 most cited of 313 All 313 (JSON)