mlsanityData to 7 Oct 2026

A Simple Efficiency Incremental Learning Framework via Vision-Language Model with Nonlinear Multi-Adapters

arXiv 2603.11211 ↗First posted 11 Mar 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Incremental Learning (IL) aims to learn new tasks while preserving previously acquired knowledge. Integrating the zero-shot learning capabilities of pre-trained vision-language models into IL methods has marked a significant advancement. However, these methods face three primary challenges: (1) the need for improved training efficiency; (2) reliance on a memory bank to store previous data; and (3) the necessity of a strong backbone to augment the model's capabilities. In this paper, we propose SimE, a Simple and Efficient framework that employs a vision-language model with adapters designed specifically for the IL task. We report a remarkable phenomenon: there is a nonlinear correlation between the number of adaptive adapter connections and the model's IL capabilities. While increasing adapter connections between transformer blocks improves model performance, adding more adaptive connections within transformer blocks during smaller incremental steps does not enhance, and may even degrade the model's IL ability. Extensive experimental results show that SimE surpasses traditional methods by 9.6% on TinyImageNet and outperforms other CLIP-based methods by 5.3% on CIFAR-100. Furthermore, we conduct a systematic study to enhance the utilization of the zero-shot capabilities of CLIP. We suggest replacing SimE's encoder with a CLIP model trained on larger datasets (e.g., LAION2B) and stronger architectures (e.g., ViT-L/14).

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 54 references that are in the map. First posted on arXiv 11 Mar 2026.

Builds on

43 of 54 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters18 Mar
  2. 2023Class Incremental Learning with Pre-trained Vision-Language Models31 Oct
  3. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models12 Mar
  4. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  5. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  6. 2016Learning without Forgettingcited by 2,465
  7. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  8. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  9. 2019Natural Adversarial Examplescited by 1,401
  10. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  11. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  12. 2021Learning to Prompt for Continual Learningcited by 938
26 more
  1. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  2. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  3. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  4. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  5. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  6. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  7. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  8. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 446
  9. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 446
  10. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  11. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  12. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  13. 2021Co2L: Contrastive Continual Learningcited by 243
  14. 2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 183
  15. 2021Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learningcited by 140
  16. 2020iTAML: An Incremental Task-Agnostic Meta-learning Approachcited by 90
  17. 2022CLIP model is an Efficient Continual Learnercited by 64
  18. 2022Don't Stop Learning: Towards Continual Learning for the CLIP Modelcited by 61
  19. 2023AttriCLIP: A Non-Incremental Learner for Incremental Knowledge Learningcited by 46
  20. 2022Memory Efficient Continual Learning with Transformerscited by 43
  21. 2022Benchmarking Omni-Vision Representation through the Lens of Visual Realmscited by 38
  22. 2023Efficient Adaptation of Large Vision Transformer via Adapter Re-Composingcited by 35
  23. 2020Learn-Prune-Share for Lifelong Learningcited by 32
  24. 2022Continual Learning with Transformers for Image Classificationcited by 17
  25. 2022Anti-Retroactive Interference for Lifelong Learningcited by 13
  26. 2025Advancing Prompt-Based Methods for Replay-Independent General Continual Learningcited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond CLIP Generalization: Against Forward&Backward Forgetting Adapter for Continual Learning of Vision-Language Models
  2. 2026Enhancing Continual Learning of Vision-Language Models via Dynamic Prefix Weighting
  3. 2024Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models
  4. 2025IAP: Improving Continual Learning of Vision-Language Models via Instance-Aware Prompting
  5. 2025Enhanced Continual Learning of Vision-Language Models with Model Fusion
  6. 2025Modular Embedding Recomposition for Incremental Learning
  7. 2024Enhancing Vision-Language Few-Shot Adaptation with Negative Learning
  8. 2023Learning without Forgetting for Vision-Language Models