mlsanityData to 7 Oct 2026

Beyond CLIP Generalization: Against Forward&Backward Forgetting Adapter for Continual Learning of Vision-Language Models

arXiv 2505.07690 ↗First posted 12 May 2025cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

This study aims to address the problem of multi-domain task incremental learning~(MTIL), which requires that vision-language models~(VLMs) continuously acquire new knowledge while maintaining their inherent zero-shot recognition capability. Existing paradigms delegate the testing of unseen-domain samples to the original CLIP, which only prevents the degradation of the model's zero-shot capability but fails to enhance the generalization of the VLM further. To this end, we propose a novel MTIL framework, named AFA, which comprises two core modules: (1) an against forward-forgetting adapter that learns task-invariant information for each dataset in the incremental tasks to enhance the zero-shot recognition ability of VLMs; (2) an against backward-forgetting adapter that strengthens the few-shot learning capability of VLMs while supporting incremental learning. Extensive experiments demonstrate that the AFA method significantly outperforms existing state-of-the-art approaches, especially in few-shot MTIL tasks, and surpasses the inherent zero-shot performance of CLIP in terms of transferability. The code is provided in the Supplementary Material.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 52 references that are in the map. First posted on arXiv 12 May 2025.

Builds on

42 of 52 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models7 Jul
  2. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters18 Mar
  3. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models12 Mar
  4. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  5. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2016Learning without Forgettingcited by 2,465
  9. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  10. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  11. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  12. 2018Experience Replay for Continual Learningcited by 1,074
25 more
  1. 2021Learning to Prompt for Continual Learningcited by 938
  2. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  3. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  4. 2021Robust fine-tuning of zero-shot modelscited by 785
  5. 2019Large Scale Incremental Learningcited by 727
  6. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  7. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  8. 2017Learning multiple visual domains with residual adapterscited by 553
  9. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  10. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 446
  11. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 446
  12. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  13. 2017CORe50: a New Dataset and Benchmark for Continuous Object Recognitioncited by 287
  14. 2019Maintaining Discrimination and Fairness in Class Incremental Learningcited by 281
  15. 2020Few-Shot Class-Incremental Learningcited by 249
  16. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  17. 2022FOSTER: Feature Boosting and Compression for Class-Incremental Learningcited by 228
  18. 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 189
  19. 2018Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgettingcited by 127
  20. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 118
  21. 2019KnowIT VQA: Answering Knowledge-Based Questions about Videoscited by 53
  22. 2022VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMixcited by 45
  23. 2024Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Modelscited by 30
  24. 2024Beyond Prompt Learning: Continual Adapter for Efficient Rehearsal-Free Continual Learningcited by 29
  25. 2023Tem-adapter: Adapting Image-Text Pretraining for Video Question Answercited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Unleashing the Potential of All Test Samples: Mean-Shift Guided Test-Time Adaptationcited by 1
  2. 2026Learning Like Humans: Analogical Concept Learning for Generalized Category Discoverycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models
  2. 2023Class Incremental Learning with Pre-trained Vision-Language Models
  3. 2025Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Models
  4. 2025Enhanced Continual Learning of Vision-Language Models with Model Fusion
  5. 2026TASSO: TAsk-Specific Subspace Optimization for Continual Learning of Vision-Language Models
  6. 2026A Simple Efficiency Incremental Learning Framework via Vision-Language Model with Nonlinear Multi-Adapters
  7. 2025IAP: Improving Continual Learning of Vision-Language Models via Instance-Aware Prompting
  8. 2024CoLeCLIP: Open-Domain Continual Learning via Joint Task Prompt and Vocabulary Learning