mlsanityData to 14 Oct 2026

Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting

arXiv 2508.04227 ↗First posted 6 Aug 2025cs.CV cs.LGcited by 18

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-language models (VLMs), spanning predictive architectures to generative Multimodal Large Language Models (MLLMs), have revolutionized artificial intelligence through powerful cross-modal alignment and zero-shot generalization. However, enabling them to learn continually from non-stationary data remains a major challenge, as their cross-modal alignment and generalization capabilities are particularly vulnerable to catastrophic forgetting. Unlike traditional unimodal continual learning (CL), VLMs face unique challenges such as cross-modal feature drift, parameter interference due to shared architectures, and zero-shot capability erosion. Furthermore, generative MLLMs exhibit a unique "alignment tax," where catastrophic forgetting manifests not merely as factual amnesia, but as a systemic collapse of deep Chain-of-Thought (CoT) reasoning. This survey presents the first comprehensive diagnostic review bridging continual learning across predictive VLMs and generative MLLMs. We systematically deconstruct the aforementioned failure modes and propose a challenge-driven taxonomy comprising four core paradigms: (1) Multi-Modal Replay Strategies addressing explicit and implicit memory drift; (2) Cross-Modal Regularization enforcing topological and geometric alignment; (3) Parameter-Efficient Adaptation utilizing dynamic routing and subspace projections; and the emerging (4) Model Fusion and Decoupling paradigms. We critically analyze the evolution of evaluation protocols, highlighting the essential shift toward dual-track benchmarks (Domain vs. Ability CL). Finally, we chart a roadmap for future research, emphasizing compositional zero-shot learning, embodied AI with sensor fusion, and autonomous agentic ecosystems. All resources are available at: https://github.com/YuyangSunshine/Awesome-Continual-learning-of-Vision-Language-Models

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 140 of its 208 references that are in the map. First posted on arXiv 6 Aug 2025.

Builds on

140 of 208 references.

Closest ancestors · 29

Same direction, newest first

  1. 2025Instruction-Grounded Visual Projectors for Continual Learning of Generative Vision-Language Models1 Aug
  2. 2025MLLM-CTBench: A Benchmark for Continual Instruction Tuning with Reasoning Process Diagnosis31 Jul
  3. 2025Branch, or Layer? Zeroth-Order Optimization for Continual Learning of Vision-Language Models14 Jun
  4. 2025Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning13 Jun
  5. 2025MLLM-CL: Continual Learning for Multimodal Large Language Models5 Jun
  6. 2025Enhancing Multimodal Continual Instruction Tuning with BranchLoRA31 May
  7. 2025SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning5 May
  8. 2025Federated Continual Instruction Tuning17 Mar
  9. 2025Synthetic Data is an Elegant GIFT for Continual Vision-Language Models6 Mar
  10. 2025CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering1 Mar
  11. 2024SMoLoRA: Exploring and Defying Dual Catastrophic Forgetting in Continual Visual Instruction Tuning21 Nov
  12. 2024ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Prompt8 Oct
17 more
  1. 2024Boosting Open-Domain Continual Learning via Leveraging Intra-domain Category-aware Prototype19 Aug
  2. 2024CLIP with Generative Latent Replay: a Strong Baseline for Incremental Learning22 Jul
  3. 2024Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models7 Jul
  4. 2024Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models27 Jun
  5. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models11 Apr
  6. 2024CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Models28 Mar
  7. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters18 Mar
  8. 2024CoLeCLIP: Open-Domain Continual Learning via Joint Task Prompt and Vocabulary Learning15 Mar
  9. 2024Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models14 Mar
  10. 2024CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model13 Mar
  11. 2023Continual Instruction Tuning for Large Multimodal Models27 Nov
  12. 2023Class Incremental Learning with Pre-trained Vision-Language Models31 Oct
  13. 2023CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation14 Aug
  14. 2023Learning without Forgetting for Vision-Language Models30 May
  15. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models12 Mar
  16. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Models28 Nov
  17. 2022Generative Negative Text Replay for Continual Vision-Language Pretraining31 Oct

Other references · 111

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  10. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
38 more
  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  8. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  9. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  11. 2015VQA: Visual Question Answeringcited by 2,506
  12. 2016Learning without Forgettingcited by 2,496
  13. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  14. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  15. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  16. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  17. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  18. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  19. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  20. 2016Progressive Neural Networkscited by 1,361
  21. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  22. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  23. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  24. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  25. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  26. 2017Continual Learning with Deep Generative Replaycited by 1,158
  27. 2021Grounded Language-Image Pre-trainingcited by 1,156
  28. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
  29. 2021Learning to Prompt for Continual Learningcited by 944
  30. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  31. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  32. 2021Robust fine-tuning of zero-shot modelscited by 797
  33. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  34. 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
  35. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  36. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
  37. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  38. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561

50 most cited of 111 All 111 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 17.

18 papers.

Sort:

By area:

  1. 2026SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuningcited by 8
  2. 2026CGL: Advancing Continual GUI Learning via Reinforcement Fine-Tuningcited by 3
  3. 2026Information-Theoretic Constraints for Continual Vision-Language-Action Alignmentcited by 3
  4. 2026IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignmentcited by 3
  5. 2026DynaTokens: Controlling Token Dynamics for Continual Video-Language Understandingcited by 1
  6. 2026DSCA: Dynamic Subspace Concept Alignment for Lifelong VLM Editingcited by 1
  7. 2026ImageHD: Energy-Efficient On-Device Continual Learning of Visual Representations via Hyperdimensional Computingcited by 1
  8. 2025KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Controlscited by 0
  9. 2026Unlocking the Potential of Grounding DINO in Videos: Parameter-Efficient Adaptation for Limited-Data Spatial-Temporal Localizationcited by 0
  10. 2026MAny: Merge Anything for Multimodal Continual Instruction Tuningcited by 0
  11. 2026AIM: Asymmetric Information Masking for Visual Question Answering Continual Learningcited by 0
  12. 2026Reasoning Portability: Guiding Continual Learning for MLLMs in the RLVR Eracited by 0
6 more
  1. 2026iGSP:Implicit Gradient Subspace Projection for Efficient Continual Learning of Vision-Language Modelscited by 0
  2. 2026CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuningcited by 0
  3. 2026ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuningcited by 0
  4. 2026MOSAIC: Modality-Specific Adaptation for Incremental Continual Learning in Parkinson's Disease Gait Assessmentcited by 0
  5. 2026Lifelong Representations: A Survey on Continual Self-Supervised Learning for Vision Modelscited by 0
  6. 2026TASSO: TAsk-Specific Subspace Optimization for Continual Learning of Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Understanding Cross-Modal Contributions in Continual Vision-Language Models: A Theoretical Perspective
  2. 2025Continual Learning for Generative AI: From LLMs to MLLMs and Beyond
  3. 2025Continual Learning in Vision-Language Models via Aligned Model Merging
  4. 2026Lifelong Learning in Vision-Language Models: Enhanced EWC with Cross-Modal Knowledge Retention
  5. 2026Black-Box Continual Learning for Vision-Language Models
  6. 2024Improving Multimodal Large Language Models Using Continual Learning
  7. 2026Continual-NExT: A Unified Comprehension And Generation Continual Learning Framework
  8. 2025When Continue Learning Meets Multimodal Large Language Model: A Survey