mlsanityData to 13 Oct 2026

Class Incremental Learning with Pre-trained Vision-Language Models

arXiv 2310.20348 ↗First posted 31 Oct 2023cs.CV cs.LGcited by 15

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

With the advent of large-scale pre-trained models, interest in adapting and exploiting them for continual learning scenarios has grown. In this paper, we propose an approach to exploiting pre-trained vision-language models (e.g. CLIP) that enables further adaptation instead of only using zero-shot learning of new tasks. We augment a pre-trained CLIP model with additional layers after the Image Encoder or before the Text Encoder. We investigate three different strategies: a Linear Adapter, a Self-attention Adapter, each operating on the image embedding, and Prompt Tuning which instead modifies prompts input to the CLIP text encoder. We also propose a method for parameter retention in the adapter layers that uses a measure of parameter importance to better maintain stability and plasticity during incremental learning. Our experiments demonstrate that the simplest solution -- a single Linear Adapter layer with parameter retention -- produces the best results. Experiments on several conventional benchmarks consistently show a significant margin of improvement over the current state-of-the-art.

Key facts. Cited by 15 papers in the map (data through 13 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 44 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

30 of 44 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,106
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,753
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,855
  5. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,323
  6. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,847
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,616
  8. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,322
  9. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  10. 2017Continual Learning with Deep Generative Replaycited by 1,155
  11. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,099
  12. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 891
18 more
  1. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 675
  2. 2018End-to-End Incremental Learningcited by 597
  3. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  4. 2019Maintaining Discrimination and Fairness in Class Incremental Learningcited by 281
  5. 2021Rainbow Memory: Continual Learning with a Memory of Diverse Samplescited by 273
  6. 2020Semantic Drift Compensation for Class-Incremental Learningcited by 241
  7. 2019REMIND Your Neural Network to Prevent Catastrophic Forgettingcited by 197
  8. 2020Adaptive Aggregation Networks for Class-Incremental Learningcited by 149
  9. 2018Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgettingcited by 127
  10. 2021Distilling Causal Effect of Data in Class-Incremental Learningcited by 117
  11. 2020A Comprehensive Study of Class Incremental Learning Algorithms for Visual Taskscited by 107
  12. 2021On Learning the Geodesic Path for Incremental Learningcited by 85
  13. 2022Continual Pre-Training Mitigates Forgetting in Language and Visioncited by 82
  14. 2021Rehearsal revealed: The limits and merits of revisiting samples in continual learningcited by 76
  15. 2022A Simple Baseline that Questions the Use of Pretrained-Models in Continual Learningcited by 54
  16. 2022Memory Efficient Continual Learning with Transformerscited by 43
  17. 2020Ternary Feature Masks: zero-forgetting for task-incremental learningcited by 14
  18. 2023POP: Prompt Of Prompts for Continual Learningcited by 7

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 7; 2025: 5; 2026: 3.

15 papers.

Sort:

By area:

  1. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 230
  2. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  3. 2024Class-Incremental Learning with CLIP: Adaptive Representation Adjustment and Parameter Fusioncited by 47
  4. 2025CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answeringcited by 36
  5. 2024Generative Multi-modal Models are Good Class-Incremental Learnerscited by 22
  6. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 16
  7. 2024PASS++: A Dual Bias Reduction Framework for Non-Exemplar Class-Incremental Learningcited by 7
  8. 2024Brain-inspired continual pre-trained learner via silent synaptic consolidationcited by 6
  9. 2024Premonition: Using Generative Models to Preempt Future Data Changes in Continual Learningcited by 5
  10. 2025Task-Core Memory Management and Consolidation for Long-term Continual Learningcited by 4
  11. 2025CalFuse: Multi-Modal Continual Learning via Feature Calibration and Parameter Fusioncited by 0
  12. 2025Representation Calibration and Uncertainty Guidance for Class-Incremental Learning based on Vision Language Modelcited by 0
3 more
  1. 2026A Simple Efficiency Incremental Learning Framework via Vision-Language Model with Nonlinear Multi-Adapterscited by 0
  2. 2026iGSP:Implicit Gradient Subspace Projection for Efficient Continual Learning of Vision-Language Modelscited by 0
  3. 2026Visual Branch is What You Need for CLIP-based Class-Incremental Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters
  2. 2025Preserving Cross-Modal Consistency for CLIP-based Class-Incremental Learning
  3. 2025Beyond CLIP Generalization: Against Forward&Backward Forgetting Adapter for Continual Learning of Vision-Language Models
  4. 2024CLIP with Generative Latent Replay: a Strong Baseline for Incremental Learning
  5. 2026Unlocking Patch-Level Features for CLIP-Based Class-Incremental Learning
  6. 2025Modular Embedding Recomposition for Incremental Learning
  7. 2025Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Models
  8. 2026Enhancing Continual Learning of Vision-Language Models via Dynamic Prefix Weighting