mlsanityData to 7 Oct 2026

Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models

arXiv 2406.18868 ↗First posted 27 Jun 2024cs.CVcited by 15

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Continual learning (CL) with Vision-Language Models (VLMs) has overcome the constraints of traditional CL, which only focuses on previously encountered classes. During the CL of VLMs, we need not only to prevent the catastrophic forgetting on incrementally learned knowledge but also to preserve the zero-shot ability of VLMs. However, existing methods require additional reference datasets to maintain such zero-shot ability and rely on domain-identity hints to classify images across different domains. In this study, we propose Regression-based Analytic Incremental Learning (RAIL), which utilizes a recursive ridge regression-based adapter to learn from a sequence of domains in a non-forgetting manner and decouple the cross-domain correlations by projecting features to a higher-dimensional space. Cooperating with a training-free fusion module, RAIL absolutely preserves the VLM's zero-shot ability on unseen domains without any reference data. Additionally, we introduce Cross-domain Task-Agnostic Incremental Learning (X-TAIL) setting. In this setting, a CL learner is required to incrementally learn from multiple domains and classify test images from both seen and unseen domains without any domain-identity hint. We theoretically prove RAIL's absolute memorization on incrementally learned domains. Experiment results affirm RAIL's state-of-the-art performance in both X-TAIL and existing Multi-domain Task-Incremental Learning settings. The code is released at https://github.com/linghan1997/Regression-based-Analytic-Incremental-Learning.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 49 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

29 of 49 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters18 Mar
  2. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models12 Mar
  3. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  4. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  5. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  6. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2016Learning without Forgettingcited by 2,465
  6. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  7. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  9. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  11. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  12. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
11 more
  1. 2021Robust fine-tuning of zero-shot modelscited by 785
  2. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  3. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  4. 2018Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselinescited by 228
  5. 2023RanPAC: Random Projections and Pre-trained Models for Continual Learningcited by 162
  6. 2023Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimalitycited by 121
  7. 2022ACIL: Analytic Class-Incremental Learning with Absolute Memorization and Privacy Protectioncited by 56
  8. 2024Locality Sensitive Sparse Encoding for Learning World Models Onlinecited by 12
  9. 2017View Independent Vehicle Make, Model and Color Recognition Using Convolutional Neural Networkcited by 9
  10. 2016Simple and Efficient Learning using Privileged Informationcited by 4
  11. 2019Where is the Fake? Patch-Wise Supervised GANs for Texture Inpaintingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 9; 2026: 3.

15 papers.

Sort:

By area:

  1. 2024Recent Advances of Multimodal Continual Learning: A Comprehensive Surveycited by 39
  2. 2024AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportationcited by 27
  3. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 16
  4. 2025MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Mergingcited by 14
  5. 2025LADA: Scalable Label-Specific CLIP Adapter for Continual Learningcited by 11
  6. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  7. 2025Semantic Shift Estimation via Dual-Projection and Classifier Reconstruction for Exemplar-Free Class-Incremental Learningcited by 7
  8. 2024One VLM to Keep it Learning: Generation and Balancing for Data-free Continual Visual Question Answeringcited by 6
  9. 2025Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Expertscited by 4
  10. 2025Affordance-First Decomposition for Continual Learning in Video-Language Understandingcited by 4
  11. 2025Enhanced Continual Learning of Vision-Language Models with Model Fusioncited by 3
  12. 2025In Praise of Stubbornness: An Empirical Case for Cognitive-Dissonance Aware Continual Update of Knowledge in LLMscited by 2
3 more
  1. 2026HyCal: A Training-Free Prototype Calibration Method for Cross-Discipline Few-Shot Class-Incremental Learningcited by 0
  2. 2026Reasoning Portability: Guiding Continual Learning for MLLMs in the RLVR Eracited by 0
  3. 2026Non-Forgetting Knowledge Allocation with Bi-level Competition for Class-Incremental Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Learning without Forgetting for Vision-Language Models
  2. 2025Beyond CLIP Generalization: Against Forward&Backward Forgetting Adapter for Continual Learning of Vision-Language Models
  3. 2025Modular Embedding Recomposition for Incremental Learning
  4. 2024Boosting Open-Domain Continual Learning via Leveraging Intra-domain Category-aware Prototype
  5. 2026Black-Box Continual Learning for Vision-Language Models
  6. 2024Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models
  7. 2024Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models
  8. 2025Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Models