# Continual Learning with Low Rank Adaptation

arXiv: https://arxiv.org/abs/2311.17601 · HTML: https://mlsanity.com/paper/2311.17601/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 37 references that are in the map.
- First posted on arXiv 29 Nov 2023.

## Builds on (most cited first, 28 of 28)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning](https://mlsanity.com/paper/2204.04799/) (2022)
- [CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2211.13218/) (2022)
- [DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion](https://mlsanity.com/paper/2111.11326/) (2021)
- [CORe50: a New Dataset and Benchmark for Continuous Object Recognition](https://mlsanity.com/paper/1705.03550/) (2017)
- [Learning without Memorizing](https://mlsanity.com/paper/1811.08051/) (2018)
- [S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning](https://mlsanity.com/paper/2207.12819/) (2022)
- [Co2L: Contrastive Continual Learning](https://mlsanity.com/paper/2106.14413/) (2021)
- [Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1805.07810/) (2018)
- [Progressive Prompts: Continual Learning for Language Models](https://mlsanity.com/paper/2301.12314/) (2023)
- [FearNet: Brain-Inspired Model for Incremental Learning](https://mlsanity.com/paper/1711.10563/) (2017)
- [Latent Replay for Real-Time Continual Learning](https://mlsanity.com/paper/1912.01100/) (2019)
- [On Transferability of Prompt Tuning for Natural Language Processing](https://mlsanity.com/paper/2111.06719/) (2021)
- [Incremental learning for the detection and classification of GAN-generated images](https://mlsanity.com/paper/1910.01568/) (2019)
- [Memory Efficient Continual Learning with Transformers](https://mlsanity.com/paper/2203.04640/) (2022)
- [Learning latent representations across multiple data domains using Lifelong VAEGAN](https://mlsanity.com/paper/2007.10221/) (2020)

## Built on by (most cited first, 29 of 29)

- [Memory in the Age of AI Agents](https://mlsanity.com/paper/2512.13564/) (2025)
- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [Towards Lifelong Learning of Large Language Models: A Survey](https://mlsanity.com/paper/2406.06391/) (2024)
- [PDE-Transformer: Efficient and Versatile Transformers for Physics Simulations](https://mlsanity.com/paper/2505.24717/) (2025)
- [CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning](https://mlsanity.com/paper/2505.24816/) (2025)
- [Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptation](https://mlsanity.com/paper/2411.05663/) (2024)
- [Pre-Act: Multi-Step Planning and Reasoning Improves Acting in LLM Agents](https://mlsanity.com/paper/2505.09970/) (2025)
- [A Practitioner's Guide to Continual Multimodal Pretraining](https://mlsanity.com/paper/2408.14471/) (2024)
- [Incremental Learning of Retrievable Skills For Efficient Continual Task Adaptation](https://mlsanity.com/paper/2410.22658/) (2024)
- [Continual Gradient Low-Rank Projection Fine-Tuning for LLMs](https://mlsanity.com/paper/2507.02503/) (2025)
- [CORAL: Scalable Multi-Task Robot Learning via LoRA Experts](https://mlsanity.com/paper/2603.09298/) (2026)
- [PLAN: Proactive Low-Rank Allocation for Continual Learning](https://mlsanity.com/paper/2510.21188/) (2025)
- [An Empirical Analysis of Forgetting in Pre-trained Models with Incremental Low-Rank Updates](https://mlsanity.com/paper/2405.18069/) (2024)
- [Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less](https://mlsanity.com/paper/2605.06654/) (2026)
- [Scalable Strategies for Continual Learning with Replay](https://mlsanity.com/paper/2505.12512/) (2025)
- [A Comparative Analysis of LLM Adaptation: SFT, LoRA, and ICL in Data-Scarce Scenarios](https://mlsanity.com/paper/2511.00130/) (2025)
- [One Rank at a Time: Cascading Error Dynamics in Sequential Learning](https://mlsanity.com/paper/2505.22602/) (2025)
- [CLoRA: Parameter-Efficient Continual Learning with Low-Rank Adaptation](https://mlsanity.com/paper/2507.19887/) (2025)
- [Mitigating Forgetting in Low Rank Adaptation](https://mlsanity.com/paper/2512.17720/) (2025)
- [LoRAX: LoRA eXpandable Networks for Continual Synthetic Image Attribution](https://mlsanity.com/paper/2504.08149/) (2025)
- [Proxy-Anchor and EVT-Driven Continual Learning Method for Generalized Category Discovery](https://mlsanity.com/paper/2504.08550/) (2025)
- [CoLoR-GAN: Continual Few-Shot Learning with Low-Rank Adaptation in Generative Adversarial Networks](https://mlsanity.com/paper/2510.13869/) (2025)
- [LoRA-based Parameter-Efficient LLMs for Continuous Learning in Edge-based Malware Detection](https://mlsanity.com/paper/2602.11655/) (2026)
- [SLAD : Shared LoRA Adapters for Task Specific Distillation](https://mlsanity.com/paper/2605.29726/) (2026)
- [Continual Learning Mechanisms Compose for Long-Horizon Memorization](https://mlsanity.com/paper/2609.06986/) (2026)
- [SAILS: Segment Anything with Incrementally Learned Semantics for Task-Invariant and Training-Free Continual Learning](https://mlsanity.com/paper/2602.14767/) (2026)
- [Remembering by Reconstructing: Domain Incremental Learning With Test-Time Training on Video Streams](https://mlsanity.com/paper/2605.31108/) (2026)
- [Catastrophic Forgetting in Sequential Thermal Anti-UAV Detection: The Role of Scale-Conditioned Gradient Imbalance](https://mlsanity.com/paper/2610.08315/) (2026)
- [ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-Tuning](https://mlsanity.com/paper/2610.19700/) (2026)

## Abstract (arXiv)

Recent work using pretrained transformers has shown impressive performance when fine-tuned with data from the downstream problem of interest. However, they struggle to retain that performance when the data characteristics changes. In this paper, we focus on continual learning, where a pre-trained transformer is updated to perform well on new data, while retaining its performance on data it was previously trained on. Earlier works have tackled this primarily through methods inspired from prompt tuning. We question this choice, and investigate the applicability of Low Rank Adaptation (LoRA) to continual learning. On a range of domain-incremental learning benchmarks, our LoRA-based solution, CoLoR, yields state-of-the-art performance, while still being as parameter efficient as the prompt tuning based methods.