mlsanityData to 14 Oct 2026

Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models

arXiv 2505.15424 ↗First posted 21 May 2025cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Continual learning (CL), which requires the model to learn multiple tasks sequentially, is crucial for large language models (LLMs). Recently, low-rank adaptation~(LoRA), one of the most representative parameter-efficient fine-tuning (PEFT) methods, has gained increasing attention in CL of LLMs. However, most existing CL methods based on LoRA typically expand a new LoRA branch to learn each new task and force the new and old LoRA branches to influence old tasks equally, potentially leading to forgetting. In this work, we propose a new method, called gated integration of low-rank adaptation (GainLoRA), for CL of LLMs. GainLoRA expands a new LoRA branch for each new task and introduces gating modules to integrate the new and old LoRA branches. Furthermore, GainLoRA leverages the new gating module to minimize the influence from the new LoRA branch to old tasks, effectively mitigating forgetting and improving the model's overall performance. Experimental results on CL benchmarks demonstrate that GainLoRA outperforms existing state-of-the-art methods.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 63 of its 78 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

63 of 78 references.

References · 63

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
38 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  3. 2016Learning without Forgettingcited by 2,496
  4. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  5. 2016Progressive Neural Networkscited by 1,361
  6. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  7. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  8. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  9. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  10. 2021Learning to Prompt for Continual Learningcited by 944
  11. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  12. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  13. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  14. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
  15. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  16. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  17. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 450
  18. 2021Gradient Projection Memory for Continual Learningcited by 348
  19. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
  20. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  21. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  22. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  23. 2019Compacting, Picking and Growing for Unforgetting Continual Learningcited by 207
  24. 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 198
  25. 2023Progressive Prompts: Continual Learning for Language Modelscited by 197
  26. 2019Episodic Memory in Lifelong Language Learningcited by 188
  27. 2023SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Modelcited by 180
  28. 2023RanPAC: Random Projections and Pre-trained Models for Continual Learningcited by 164
  29. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 160
  30. 2021Training Networks in Null Space of Feature Covariance for Continual Learningcited by 143
  31. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 141
  32. 2023Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimalitycited by 121
  33. 2022Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning Systemcited by 112
  34. 2020Continual Learning with Node-Importance based Adaptive Group Sparse Regularizationcited by 105
  35. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 104
  36. 2020Continual Learning in Low-rank Orthogonal Subspacescited by 99
  37. 2023TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Modelscited by 66
  38. 2024Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learningcited by 53

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 12.

13 papers.

Sort:

By area:

  1. 2026Harness Continual Learning: Continual Adaptation Beyond Model Parameterscited by 6
  2. 2026Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuningcited by 5
  3. 2026Continual Safety Alignment via Gradient-Based Sample Selectioncited by 4
  4. 2026CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learningcited by 2
  5. 2026Rethinking Transfer in Continual Learning: A Replay-Based Realisationcited by 1
  6. 2026Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Modelscited by 1
  7. 2025Mixtures of SubExperts for Large Language Continual Learningcited by 0
  8. 2026SafeAnchor: Preventing Cumulative Safety Erosion in Continual Domain Adaptation of Large Language Modelscited by 0
  9. 2026Learning When to Adaptcited by 0
  10. 2026DeCoFlow: Structural Decomposition of Normalizing Flows for Continual Anomaly Detectioncited by 0
  11. 2026Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learningcited by 0
  12. 2026Parameter Efficient Continual Learning for Sparse Event-Based Transformerscited by 0
1 more
  1. 2026Making LoRA Identifiable: Orthogonal Alignment for Continual Learningcited by 0