mlsanityData to 14 Oct 2026

Lifelong Learning in Vision-Language Models: Enhanced EWC with Cross-Modal Knowledge Retention

arXiv 2605.12789 ↗First posted 12 May 2026cs.ROcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Large language-vision models (LVLMs) such as CLIP, Flamingo, and BLIP have revolutionized AI by enabling understanding across textual and visual modalities. These models excel at tasks like image captioning, visual question answering, and cross-modal retrieval. However, they face catastrophic forgetting when learning new tasks sequentially, particularly challenging in multi-modal settings where preserving cross-modal alignments adds complexity to the learning process. This paper presents a comprehensive continual learning framework for LVLMs that combines enhanced Elastic Weight Consolidation (EWC) with parameter-efficient fine-tuning techniques. We integrate multi-modal Fisher Information Matrix calculation, consistency preservation across modalities, and adaptive regularization that considers dependencies across visual and textual encoders. The framework achieves a 78% reduction in forgetting rates relative to naive sequential training approaches through extensive evaluation testing. The framework also preserves alignment between modalities during sequential learning with only 15% additional computational cost. This work advances the state of the art in lifelong learning for multi-modal AI systems, with direct applications to autonomous driving, intelligent robotic assistants, and adaptive robotic systems that must continuously learn in dynamic real-world environments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 12 of its 18 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

12 of 18 references.

References · 12

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2016Learning without Forgettingcited by 2,496
  5. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  6. 2016Progressive Neural Networkscited by 1,361
  7. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  8. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
  9. 2021Learning to Prompt for Continual Learningcited by 944
  10. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  11. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 237
  12. 2023Causal Interpretation of Self-Attention in Pre-Trained Transformerscited by 30

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Understanding Cross-Modal Contributions in Continual Vision-Language Models: A Theoretical Perspective
  2. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting
  3. 2025MLLM-CL: Continual Learning for Multimodal Large Language Models
  4. 2025Continual Learning in Vision-Language Models via Aligned Model Merging
  5. 2025Enhanced Continual Learning of Vision-Language Models with Model Fusion
  6. 2026Black-Box Continual Learning for Vision-Language Models
  7. 2026Information-Theoretic Constraints for Continual Vision-Language-Action Alignment
  8. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters