mlsanityData to 7 Oct 2026

Task Arithmetic with LoRA for Continual Learning

arXiv 2311.02428 ↗First posted 4 Nov 2023cs.CV cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

Continual learning refers to the problem where the training data is available in sequential chunks, termed "tasks". The majority of progress in continual learning has been stunted by the problem of catastrophic forgetting, which is caused by sequential training of the model on streams of data. Moreover, it becomes computationally expensive to sequentially train large models multiple times. To mitigate both of these problems at once, we propose a novel method to continually train transformer-based vision models using low-rank adaptation and task arithmetic. Our method completely bypasses the problem of catastrophic forgetting, as well as reducing the computational requirement for training models on each task. When aided with a small memory of 10 samples per class, our method achieves performance close to full-set finetuning. We present rigorous ablations to support the prowess of our method.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 18 of its 27 references that are in the map. First posted on arXiv 4 Nov 2023.

Builds on

18 of 27 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2016Learning without Forgettingcited by 2,465
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  12. 2017Continual Learning with Deep Generative Replaycited by 1,152
6 more
  1. 2018Experience Replay for Continual Learningcited by 1,074
  2. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  3. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  4. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  5. 2021Avalanche: an End-to-End Library for Continual Learningcited by 124
  6. 2022Continual VQA for Disaster Response Systemscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 14; 2026: 5.

31 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2024A Survey on LoRA of Large Language Modelscited by 67
  3. 2025SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learningcited by 60
  4. 2025When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformerscited by 38
  5. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  6. 2024Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptationcited by 26
  7. 2024PILoRA: Prototype Guided Incremental LoRA for Federated Class-Incremental Learningcited by 19
  8. 2025MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Mergingcited by 14
  9. 2025Parameter-Efficient Continual Fine-Tuning: A Surveycited by 11
  10. 2025Continual Gradient Low-Rank Projection Fine-Tuning for LLMscited by 10
  11. 2024DPPA: Pruning Method for Large Language Model to Model Mergingcited by 7
  12. 2024Calibrating Bayesian Learning via Regularization, Confidence Minimization, and Selective Inferencecited by 6
19 more
  1. 2024An Empirical Analysis of Forgetting in Pre-trained Models with Incremental Low-Rank Updatescited by 6
  2. 2026When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Mergingcited by 5
  3. 2024DESIRE: Dynamic Knowledge Consolidation for Rehearsal-Free Continual Learningcited by 4
  4. 2025A Systematic Study of In-the-Wild Model Merging for Large Language Modelscited by 4
  5. 2024Disentangling Latent Shifts of In-Context Learning with Weak Supervisioncited by 3
  6. 2025Parameter Efficient Continual Learning with Dynamic Low-Rank Adaptationcited by 3
  7. 2025CLoRA: Parameter-Efficient Continual Learning with Low-Rank Adaptationcited by 2
  8. 2025Toward a Holistic Approach to Continual Model Mergingcited by 2
  9. 2025MAGIC: Achieving Superior Model Merging via Magnitude Calibrationcited by 2
  10. 2025Task Vector Bases: A Unified and Scalable Framework for Compressed Task Arithmeticcited by 1
  11. 2025Exploring and Leveraging Class Vectors for Classifier Editingcited by 1
  12. 2024Hyper-Representations: Learning from Populations of Neural Networkscited by 0
  13. 2024R-MTLLMF: Resilient Multi-Task Large Language Model Fusion at the Wireless Edgecited by 0
  14. 2025ReqBrain: Task-Specific Instruction Tuning of LLMs for AI-Assisted Requirements Generationcited by 0
  15. 2025Separation and Collaboration: Two-Level Routing Grouped Mixture-of-Experts for Multi-Domain Continual Learningcited by 0
  16. 2026Task Arithmetic with Support Languages for Low-Resource ASRcited by 0
  17. 2026SAILS: Segment Anything with Incrementally Learned Semantics for Task-Invariant and Training-Free Continual Learningcited by 0
  18. 2026Remembering by Reconstructing: Domain Incremental Learning With Test-Time Training on Video Streamscited by 0
  19. 2026EVAF: A Test-Retest Protocol for Selective Parametric Consolidationcited by 0