mlsanityData to 14 Oct 2026

Mitigating Forgetting in Low Rank Adaptation

arXiv 2512.17720 ↗First posted 19 Dec 2025cs.LG stat.MLcited by 2

Not on the map. Why?

Abstract (arXiv)

Parameter-efficient fine-tuning methods, such as Low-Rank Adaptation (LoRA), enable fast specialization of large pre-trained models to different downstream applications. However, this process often leads to catastrophic forgetting of the model's prior domain knowledge. We address this issue with LaLoRA, a weight-space regularization technique that applies a Laplace approximation to Low-Rank Adaptation. Our approach estimates the model's confidence in each parameter and constrains updates in high-curvature directions, preserving prior knowledge while enabling efficient target-domain learning. By applying the Laplace approximation only to the LoRA weights, the method remains lightweight. We evaluate LaLoRA by fine-tuning a Llama model for mathematical reasoning and demonstrate an improved learning-forgetting trade-off, which can be directly controlled via the method's regularization strength. We further explore different loss landscape curvature approximations for estimating parameter confidence, analyze the effect of the data used for the Laplace approximation, and study robustness across hyperparameters.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 39 of its 45 references that are in the map. First posted on arXiv 19 Dec 2025.

Builds on

39 of 45 references.

References · 39

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2020GLU Variants Improve Transformercited by 1,804
  9. 2019Root Mean Square Layer Normalizationcited by 1,480
  10. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  11. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  12. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
27 more
  1. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  2. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
  3. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  4. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  5. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 230
  6. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  7. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 201
  8. 2022Language model compression with weighted low-rank factorizationcited by 195
  9. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  10. 2022Fine-tuned Language Models are Continual Learnerscited by 156
  11. 2024Continual Learning for Large Language Models: A Surveycited by 147
  12. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 132
  13. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 115
  14. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 109
  15. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 94
  16. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugincited by 82
  17. 2024A Closer Look at the Limitations of Instruction Tuningcited by 77
  18. 2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
  19. 2024InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructionscited by 58
  20. 2024Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Modelscited by 54
  21. 2024LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Reportcited by 47
  22. 2023Continual Learning with Low Rank Adaptationcited by 29
  23. 2024Astraios: Parameter-Efficient Instruction Tuning Code Large Language Modelscited by 23
  24. 2024Unlocking Continual Learning Abilities in Language Modelscited by 23
  25. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscapecited by 20
  26. 2023Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Modelscited by 13
  27. 2024Debiasing Mini-Batch Quadratics for Applications in Deep Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Robust Stochastic Gradient Posterior Sampling with Lattice Based Discretisationcited by 1
  2. 2026One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning When to Adapt
  2. 2024MoR: Mixture of Ranks for Low-Rank Adaptation Tuning
  3. 2025RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models
  4. 2026Adaptive Utilization of Low-Rank Adaptation via Conditioned Gating
  5. 2026Strategic Over-Parameterization for Generalizable Low-Rank Adaptation
  6. 2026Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning
  7. 2025ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning
  8. 2026Parameter-Efficient Fine-Tuning with Learnable Rank