mlsanityData to 7 Oct 2026

InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning

arXiv 2404.00228 ↗First posted 30 Mar 2024cs.LG cs.AI cs.CVcited by 189

Not on the map. Why?

Abstract (arXiv)

Continual learning requires the model to learn multiple tasks sequentially. In continual learning, the model should possess the ability to maintain its performance on old tasks (stability) and the ability to adapt to new tasks continuously (plasticity). Recently, parameter-efficient fine-tuning (PEFT), which involves freezing a pre-trained model and injecting a small number of learnable parameters to adapt to downstream tasks, has gained increasing popularity in continual learning. Although existing continual learning methods based on PEFT have demonstrated superior performance compared to those not based on PEFT, most of them do not consider how to eliminate the interference of the new task on the old tasks, which inhibits the model from making a good trade-off between stability and plasticity. In this work, we propose a new PEFT method, called interference-free low-rank adaptation (InfLoRA), for continual learning. InfLoRA injects a small number of parameters to reparameterize the pre-trained weights and shows that fine-tuning these injected parameters is equivalent to fine-tuning the pre-trained weights within a subspace. Furthermore, InfLoRA designs this subspace to eliminate the interference of the new task on the old tasks, making a good trade-off between stability and plasticity. Experimental results show that InfLoRA outperforms existing state-of-the-art continual learning methods on multiple datasets.

Key facts. Cited by 189 papers in the map (data through 7 Oct 2026). Builds on 40 of its 53 references that are in the map. First posted on arXiv 30 Mar 2024.

Builds on

40 of 53 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  10. 2022Visual Prompt Tuningcited by 1,674
  11. 2016Progressive Neural Networkscited by 1,332
  12. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
28 more
  1. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  2. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  3. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  4. 2021Learning to Prompt for Continual Learningcited by 938
  5. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  6. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  7. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  8. 2019Gradient based sample selection for online continual learningcited by 568
  9. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  10. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 446
  11. 2019Online Continual Learning with Maximally Interfered Retrievalcited by 376
  12. 2021Gradient Projection Memory for Continual Learningcited by 338
  13. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  14. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  15. 2019Compacting, Picking and Growing for Unforgetting Continual Learningcited by 207
  16. 2023SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Modelcited by 178
  17. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 138
  18. 2022FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformercited by 133
  19. 2023A Unified Continual Learning Framework with General Parameter-Efficient Tuningcited by 124
  20. 2023Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimalitycited by 121
  21. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Modelscited by 112
  22. 2020Continual Learning with Node-Importance based Adaptive Group Sparse Regularizationcited by 102
  23. 2022TRGP: Trust Region Gradient Projection for Continual Learningcited by 79
  24. 2023Introducing Language Guidance in Prompt-based Continual Learningcited by 49
  25. 2022Transfer without Forgettingcited by 44
  26. 2022Exploring Example Influence in Continual Learningcited by 44
  27. 2022AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Taskscited by 29
  28. 2021On the importance of cross-task features for class-incremental learningcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 78; 2026: 88.

189 papers.

Sort:

By area:

  1. 2024A Survey on LoRA of Large Language Modelscited by 67
  2. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  3. 2025SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learningcited by 60
  4. 2024Visual Prompt Tuning in Null Space for Continual Learningcited by 30
  5. 2025Tensor Product Attention Is All You Needcited by 27
  6. 2024Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptationcited by 26
  7. 2025CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learningcited by 26
  8. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22
  9. 2024A Second-Order Perspective on Model Compositionality and Incremental Learningcited by 18
  10. 2024Adapter Merging with Centroid Prototype Mapping for Scalable Class-Incremental Learningcited by 18
  11. 2025LoRA Subtraction for Drift-Resistant Space in Exemplar-Free Continual Learningcited by 16
  12. 2025Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learningcited by 15
38 more
  1. 2026Mass Concept Erasure in Diffusion Models with Concept Hierarchycited by 14
  2. 2025Componential Prompt-Knowledge Alignment for Domain Incremental Learningcited by 13
  3. 2025Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Modelscited by 12
  4. 2024Dynamic Integration of Task-Specific Adapters for Class Incremental Learningcited by 11
  5. 2025RainbowPrompt: Diversity-Enhanced Prompt-Evolving for Continual Learningcited by 10
  6. 2025Harnessing Textual Semantic Priors for Knowledge Transfer and Refinement in CLIP-Driven Continual Learningcited by 10
  7. 2025Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learningcited by 10
  8. 2024KIF: Knowledge Identification and Fusion for Language Model Continual Learningcited by 9
  9. 2024LW2G: Learning Whether to Grow for Prompt-based Continual Learningcited by 9
  10. 2024CAPrompt: Cyclic Prompt Aggregation for Pre-Trained Model Based Class Incremental Learningcited by 9
  11. 2025Merge before Forget: A Single LoRA Continual Learning via Continual Mergingcited by 9
  12. 2024LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learningcited by 8
  13. 2025Continuous Subspace Optimization for Continual Learningcited by 8
  14. 2026Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learningcited by 8
  15. 2026CORAL: Scalable Multi-Task Robot Learning via LoRA Expertscited by 8
  16. 2024OpenCIL: Benchmarking Out-of-Distribution Detection in Class-Incremental Learningcited by 7
  17. 2025StPR: Spatiotemporal Preservation and Routing for Exemplar-Free Video Class-Incremental Learningcited by 7
  18. 2025SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splittingcited by 7
  19. 2025OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignmentcited by 7
  20. 2025AnaCP: Toward Upper-Bound Continual Learning via Analytic Contrastive Projectioncited by 7
  21. 2025Navigating Semantic Drift in Task-Agnostic Class-Incremental Learningcited by 6
  22. 2025Continual Learning on CLIP via Incremental Prompt Tuning with Intrinsic Textual Anchorscited by 6
  23. 2024Mamba-CL: Optimizing Selective State Space Model in Null Space for Continual Learningcited by 5
  24. 2025Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learningcited by 5
  25. 2025EKPC: Elastic Knowledge Preservation and Compensation for Class-Incremental Learningcited by 5
  26. 2025CKAA: Cross-subspace Knowledge Alignment and Aggregation for Robust Continual Learningcited by 5
  27. 2025PLAN: Proactive Low-Rank Allocation for Continual Learningcited by 5
  28. 2024CODE-CL: Conceptor-Based Gradient Projection for Deep Continual Learningcited by 4
  29. 2024DESIRE: Dynamic Knowledge Consolidation for Rehearsal-Free Continual Learningcited by 4
  30. 2025Lifelong Learning with Task-Specific Adaptation: Addressing the Stability-Plasticity Dilemmacited by 4
  31. 2025Achieving More with Less: Additive Prompt Tuning for Rehearsal-Free Class-Incremental Learningcited by 4
  32. 2025Attention in Diffusion Model: A Surveycited by 4
  33. 2025FM-LoRA: Factorized Low-Rank Meta-Prompting for Continual Learningcited by 4
  34. 2025Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Expertscited by 4
  35. 2025Instruction-Grounded Visual Projectors for Continual Learning of Generative Vision-Language Modelscited by 4
  36. 2025Null-Space Filtering for Data-Free Continual Model Merging: Preserving Stability, Promoting Plasticitycited by 4
  37. 2026Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptationcited by 4
  38. 2026NeuroLoRA: Context-Aware Neuromodulation for Parameter-Efficient Multi-Task Adaptationcited by 4

50 most cited of 189 All 189 (JSON)