mlsanityData to 7 Oct 2026

Continual Learning with Low Rank Adaptation

arXiv 2311.17601 ↗First posted 29 Nov 2023cs.LG cs.AIcited by 27

Not on the map. Why?

Abstract (arXiv)

Recent work using pretrained transformers has shown impressive performance when fine-tuned with data from the downstream problem of interest. However, they struggle to retain that performance when the data characteristics changes. In this paper, we focus on continual learning, where a pre-trained transformer is updated to perform well on new data, while retaining its performance on data it was previously trained on. Earlier works have tackled this primarily through methods inspired from prompt tuning. We question this choice, and investigate the applicability of Low Rank Adaptation (LoRA) to continual learning. On a range of domain-incremental learning benchmarks, our LoRA-based solution, CoLoR, yields state-of-the-art performance, while still being as parameter efficient as the prompt tuning based methods.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 28 of its 37 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

28 of 37 references.

References · 28

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2016Learning without Forgettingcited by 2,465
  8. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  9. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  10. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  11. 2021Learning to Prompt for Continual Learningcited by 938
  12. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
16 more
  1. 2019Large Scale Incremental Learningcited by 727
  2. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  3. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 446
  4. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  5. 2017CORe50: a New Dataset and Benchmark for Continuous Object Recognitioncited by 287
  6. 2018Learning without Memorizingcited by 249
  7. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  8. 2021Co2L: Contrastive Continual Learningcited by 243
  9. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 197
  10. 2023Progressive Prompts: Continual Learning for Language Modelscited by 195
  11. 2017FearNet: Brain-Inspired Model for Incremental Learningcited by 177
  12. 2019Latent Replay for Real-Time Continual Learningcited by 97
  13. 2021On Transferability of Prompt Tuning for Natural Language Processingcited by 74
  14. 2019Incremental learning for the detection and classification of GAN-generated imagescited by 58
  15. 2022Memory Efficient Continual Learning with Transformerscited by 43
  16. 2020Learning latent representations across multiple data domains using Lifelong VAEGANcited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 14; 2026: 7.

27 papers.

Sort:

By area:

  1. 2025Memory in the Age of AI Agentscited by 285
  2. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  3. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  4. 2025PDE-Transformer: Efficient and Versatile Transformers for Physics Simulationscited by 29
  5. 2024Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptationcited by 26
  6. 2025CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learningcited by 26
  7. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22
  8. 2025Pre-Act: Multi-Step Planning and Reasoning Improves Acting in LLM Agentscited by 20
  9. 2024Incremental Learning of Retrievable Skills For Efficient Continual Task Adaptationcited by 19
  10. 2025Continual Gradient Low-Rank Projection Fine-Tuning for LLMscited by 10
  11. 2026CORAL: Scalable Multi-Task Robot Learning via LoRA Expertscited by 8
  12. 2024An Empirical Analysis of Forgetting in Pre-trained Models with Incremental Low-Rank Updatescited by 6
15 more
  1. 2025PLAN: Proactive Low-Rank Allocation for Continual Learningcited by 5
  2. 2025A Comparative Analysis of LLM Adaptation: SFT, LoRA, and ICL in Data-Scarce Scenarioscited by 4
  3. 2025Scalable Strategies for Continual Learning with Replaycited by 3
  4. 2025One Rank at a Time: Cascading Error Dynamics in Sequential Learningcited by 3
  5. 2026Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Lesscited by 3
  6. 2025CLoRA: Parameter-Efficient Continual Learning with Low-Rank Adaptationcited by 2
  7. 2025Mitigating Forgetting in Low Rank Adaptationcited by 2
  8. 2025LoRAX: LoRA eXpandable Networks for Continual Synthetic Image Attributioncited by 1
  9. 2025Proxy-Anchor and EVT-Driven Continual Learning Method for Generalized Category Discoverycited by 1
  10. 2025CoLoR-GAN: Continual Few-Shot Learning with Low-Rank Adaptation in Generative Adversarial Networkscited by 1
  11. 2026LoRA-based Parameter-Efficient LLMs for Continuous Learning in Edge-based Malware Detectioncited by 1
  12. 2026Continual Learning Mechanisms Compose for Long-Horizon Memorizationcited by 1
  13. 2026SAILS: Segment Anything with Incrementally Learned Semantics for Task-Invariant and Training-Free Continual Learningcited by 0
  14. 2026SLAD : Shared LoRA Adapters for Task Specific Distillationcited by 0
  15. 2026Remembering by Reconstructing: Domain Incremental Learning With Test-Time Training on Video Streamscited by 0