mlsanityData to 7 Oct 2026

Caption, Create, Continue: Continual Learning with Pre-trained Generative Vision-Language Models

arXiv 2409.17806 ↗First posted 26 Sep 2024cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Continual learning (CL) enables models to adapt to evolving data streams without catastrophic forgetting, a fundamental requirement for real-world AI systems. However, the current methods often depend on large replay buffers or heavily annotated datasets which are impractical due to storage, privacy, and cost constraints. We propose CLTS (Continual Learning via Text-Image Synergy), a novel class-incremental framework that mitigates forgetting without storing real task data. CLTS leverages pre-trained vision-language models, BLIP (Bootstrapping Language-Image Pre-training) for caption generation and stable diffusion for sample generation. Each task is handled by a dedicated Task Head, while a Task Router learns to assign inputs to the correct Task Head using the generated data. On three benchmark datasets, CLTS improves average task accuracy by up to 54% and achieves 63 times better memory efficiency compared to four recent continual learning baselines, demonstrating improved retention and adaptability. CLTS introduces a novel perspective by integrating generative text-image augmentation for scalable continual learning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 24 references that are in the map. First posted on arXiv 26 Sep 2024.

Builds on

16 of 24 references.

References · 16

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2016Learning without Forgettingcited by 2,465
  3. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  4. 2016Progressive Neural Networkscited by 1,332
  5. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  6. 2018Experience Replay for Continual Learningcited by 1,074
  7. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  8. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
  9. 2021Self-Supervised Models are Continual Learnerscited by 149
  10. 2020Adversarial Continual Learningcited by 140
  11. 2023Class-Incremental Learning using Diffusion Model for Distillation and Replaycited by 31
  12. 2019Unsupervised Progressive Learning and the STAM Architecturecited by 29
4 more
  1. 2019Generative Continual Concept Learningcited by 26
  2. 2021Unsupervised Continual Learning via Self-Adaptive Deep Clustering Approachcited by 19
  3. 2022SCALE: Online Self-Supervised Lifelong Learning without Prior Knowledgecited by 16
  4. 2024U-TELL: Unsupervised Task Expert Lifelong Learningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Continual Learning for Generative AI: From LLMs to MLLMs and Beyond
  2. 2025Harnessing Textual Semantic Priors for Knowledge Transfer and Refinement in CLIP-Driven Continual Learning
  3. 2025Synthetic Data is an Elegant GIFT for Continual Vision-Language Models
  4. 2025Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Models
  5. 2024Dual-Phase Continual Learning: Supervised Adaptation Meets Unsupervised Retention
  6. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting
  7. 2024Improving Multimodal Large Language Models Using Continual Learning
  8. 2026Black-Box Continual Learning for Vision-Language Models