mlsanityData to 14 Oct 2026

LADA: Scalable Label-Specific CLIP Adapter for Continual Learning

arXiv 2505.23271 ↗First posted 29 May 2025cs.CV cs.LGcited by 11

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Continual learning with vision-language models like CLIP offers a pathway toward scalable machine learning systems by leveraging its transferable representations. Existing CLIP-based methods adapt the pre-trained image encoder by adding multiple sets of learnable parameters, with each task using a partial set of parameters. This requires selecting the expected parameters for input images during inference, which is prone to error that degrades performance. To address this problem, we introduce LADA (Label-specific ADApter). Instead of partitioning parameters across tasks, LADA appends lightweight, label-specific memory units to the frozen CLIP image encoder, enabling discriminative feature generation by aggregating task-agnostic knowledge. To prevent catastrophic forgetting, LADA employs feature distillation for seen classes, preventing their features from being interfered with by new classes. Positioned after the image encoder, LADA prevents gradient flow to the frozen CLIP parameters, ensuring efficient training. Extensive results show that LADA achieves state-of-the-art performance in continual learning settings. The implementation code is available at https://github.com/MaolinLuo/LADA.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 60 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

44 of 60 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Vision-Language Models are Strong Noisy Label Detectors29 Sep
  2. 2024Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models7 Jul
  3. 2024Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models27 Jun
  4. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters18 Mar
  5. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models12 Mar
  6. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  7. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct

Other references · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2016Learning without Forgettingcited by 2,496
  6. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  7. 2013Describing Textures in the Wildcited by 2,125
  8. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  10. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  11. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  12. 2017Continual Learning with Deep Generative Replaycited by 1,158
25 more
  1. 2018Experience Replay for Continual Learningcited by 1,095
  2. 2021Learning to Prompt for Continual Learningcited by 944
  3. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  4. 2021Robust fine-tuning of zero-shot modelscited by 797
  5. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  6. 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
  7. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
  8. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  9. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 450
  10. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  11. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  12. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  13. 2018Selective Experience Replay for Lifelong Learningcited by 241
  14. 2018Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselinescited by 228
  15. 2023SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Modelcited by 180
  16. 2023RanPAC: Random Projections and Pre-trained Models for Continual Learningcited by 164
  17. 2019Uncertainty-based Continual Learning with Adaptive Regularizationcited by 146
  18. 2023Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimalitycited by 121
  19. 2023Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurtscited by 58
  20. 2024Class-Incremental Learning with CLIP: Adaptive Representation Adjustment and Parameter Fusioncited by 47
  21. 2023When Prompt-based Incremental Learning Does Not Meet Strong Pretrainingcited by 41
  22. 2018Continual Classification Learning Using Generative Modelscited by 28
  23. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 23
  24. 2024Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learningcited by 19
  25. 2023NPCL: Neural Processes for Uncertainty-Aware Continual Learningcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 10.

11 papers.

  1. 2026Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptationcited by 5
  2. 2026HyCal: A Training-Free Prototype Calibration Method for Cross-Discipline Few-Shot Class-Incremental Learningcited by 1
  3. 2026Hidden Forgetting in Continual Multimodal Learning: When Accuracy Survives but Grounding Failscited by 1
  4. 2025Prompt-Based Continual Compositional Zero-Shot Learningcited by 0
  5. 2026GR4CIL: Gap-compensated Routing for CLIP-based Class Incremental Learningcited by 0
  6. 2026SHED: Style-Homogenized Embedding Alignment for Domain Generalizationcited by 0
  7. 2026COVD: Continual Open-Vocabulary Object Detection with Novel Concept Injectioncited by 0
  8. 2026KeepLoRA++: Continual Learning with Layer-Scaled Residual Gradient Adaptationcited by 0
  9. 2026Black-Box Continual Learning for Vision-Language Modelscited by 0
  10. 2026InduceKV: Fixed-Footprint Continual Adaptation of Multimodal LLMs via Inducing KV Memoriescited by 0
  11. 2026OrthoSkillVLA: Continual Skill Learning via Gradient-Informed Skill Subspace Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LVP-CLIP:Revisiting CLIP for Continual Learning with Label Vector Pool
  2. 2023Class Incremental Learning with Pre-trained Vision-Language Models
  3. 2022CLIP model is an Efficient Continual Learner
  4. 2025Beyond CLIP Generalization: Against Forward&Backward Forgetting Adapter for Continual Learning of Vision-Language Models
  5. 2025Harnessing Textual Semantic Priors for Knowledge Transfer and Refinement in CLIP-Driven Continual Learning
  6. 2024CLIP with Generative Latent Replay: a Strong Baseline for Incremental Learning
  7. 2025Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Models
  8. 2026Decomposing and Composing: Towards Efficient Vision-Language Continual Learning via Rank-1 Expert Pool in a Single LoRA