mlsanityData to 14 Oct 2026

Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models

arXiv 2410.16801 ↗First posted 22 Oct 2024cs.CL cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) exhibit remarkable capabilities in natural language processing but face catastrophic forgetting when learning new tasks, where adaptation to a new domain leads to a substantial decline in performance on previous tasks. In this paper, we propose Controlled LoRA (CLoRA), a sub-space regularization method on LoRA structure. Aiming to reduce the scale of output change while introduce minimal constraint on model capacity, CLoRA imposes constraint on the direction of updating matrix's null space. Experimental results on one-stage LLM finetuning tasks and continual learning settings highlight the superority of CLoRA as a effective parameter efficient finetuning method with catastrophic forgetting mitigating.Further investigation for model parameters indicates that CLoRA effectively balances the trade-off between model capacity and degree of forgetting.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 30 of its 48 references that are in the map. First posted on arXiv 22 Oct 2024.

Builds on

30 of 48 references.

References · 30

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  10. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
18 more
  1. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  2. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  3. 2021Learning to Prompt for Continual Learningcited by 944
  4. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  5. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  6. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  7. 2019Orthogonal Gradient Descent for Continual Learningcited by 369
  8. 2024LoRA Learns Less and Forgets Lesscited by 328
  9. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
  10. 2023Progressive Prompts: Continual Learning for Language Modelscited by 197
  11. 2019Episodic Memory in Lifelong Language Learningcited by 188
  12. 2024Continual Learning for Large Language Models: A Surveycited by 147
  13. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 141
  14. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 123
  15. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 104
  16. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 94
  17. 2022NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networkscited by 39
  18. 2024Learning Attentional Mixture of LoRAs for Language Model Continual Learningcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2025SEAT: Sparse Entity-Aware Tuning for Knowledge Adaptation while Preserving Epistemic Abstentioncited by 3
  2. 2025The Effectiveness of Approximate Regularized Replay for Efficient Supervised Fine-Tuning of Large Language Modelscited by 2
  3. 2026Principled Steering via Null-space Projection for Jailbreak Defense in Vision-Language Modelscited by 2
  4. 2025Patching LLM Like Software: A Lightweight Method for Improving Safety Policy in Large Language Modelscited by 1
  5. 2026Attribution-Guided Continual Learning for Large Language Modelscited by 1
  6. 2026Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgettingcited by 1
  7. 2026Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Modelscited by 1
  8. 2026Learn Here, Move Less Elsewhere: Input-Conditioned Plasticity from Retained-Domain Activation Atlasescited by 0
  9. 2026A Dynamical Theory of LoRA in Continual Learningcited by 0