mlsanityData to 7 Oct 2026

Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters

arXiv 2403.11549 ↗First posted 18 Mar 2024cs.CVcited by 199

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

also in 1 more
Abstract (arXiv)

Continual learning can empower vision-language models to continuously acquire new knowledge, without the need for access to the entire historical dataset. However, mitigating the performance degradation in large-scale models is non-trivial due to (i) parameter shifts throughout lifelong learning and (ii) significant computational burdens associated with full-model tuning. In this work, we present a parameter-efficient continual learning framework to alleviate long-term forgetting in incremental learning with vision-language models. Our approach involves the dynamic expansion of a pre-trained CLIP model, through the integration of Mixture-of-Experts (MoE) adapters in response to new tasks. To preserve the zero-shot recognition capability of vision-language models, we further introduce a Distribution Discriminative Auto-Selector (DDAS) that automatically routes in-distribution and out-of-distribution inputs to the MoE Adapter and the original CLIP, respectively. Through extensive experiments across various settings, our proposed method consistently outperforms previous state-of-the-art approaches while concurrently reducing parameter training burdens by 60%. Our code locates at https://github.com/JiazuoYu/MoE-Adapters4CL

Key facts. Cited by 199 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 56 of its 84 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

56 of 84 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Class Incremental Learning with Pre-trained Vision-Language Models31 Oct
  2. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models12 Mar
  3. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  4. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  5. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct

Other references · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  10. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2016Learning without Forgettingcited by 2,465
38 more
  1. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  2. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  3. 2013Describing Textures in the Wildcited by 2,110
  4. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  5. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  6. 2022Visual Prompt Tuningcited by 1,674
  7. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  8. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  9. 2017Continual Learning with Deep Generative Replaycited by 1,152
  10. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  11. 2019When Does Label Smoothing Help?cited by 985
  12. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  13. 2021Robust fine-tuning of zero-shot modelscited by 785
  14. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  15. 2019Large Scale Incremental Learningcited by 727
  16. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  17. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  18. 2018End-to-End Incremental Learningcited by 596
  19. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  20. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 422
  21. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
  22. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  23. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  24. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  25. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  26. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  27. 2018Selective Experience Replay for Lifelong Learningcited by 239
  28. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  29. 2023Lifelong Language Pretraining with Distribution-Specialized Expertscited by 64
  30. 2022Continual Learning with Foundation Models: An Empirical Study of Latent Replaycited by 61
  31. 2022Don't Stop Learning: Towards Continual Learning for the CLIP Modelcited by 61
  32. 2022Mixture of Attention Heads: Selecting Attention Heads Per Tokencited by 58
  33. 2023Dense Network Expansion for Class Incremental Learningcited by 45
  34. 2023Class-Incremental Learning using Diffusion Model for Distillation and Replaycited by 31
  35. 2018Continual Classification Learning Using Generative Modelscited by 28
  36. 2023VL-PET: Vision-and-Language Parameter-Efficient Tuning via Granularity Controlcited by 21
  37. 2023On the Effectiveness of LayerNorm Tuning for Continual Learning in Vision Transformerscited by 14
  38. 2023Mobile V-MoEs: Scaling Down Vision Transformers via Sparse Mixture-of-Expertscited by 14

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 43; 2025: 82; 2026: 74.

199 papers.

Sort:

By area:

  1. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  2. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  3. 2024Mixture of A Million Expertscited by 70
  4. 2024Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learningcited by 53
  5. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  6. 2024Theory on Mixture-of-Experts in Continual Learningcited by 50
  7. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  8. 2024Mixture of Experts Meets Prompt-Based Continual Learningcited by 43
  9. 2024SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domaincited by 39
  10. 2024Recent Advances of Multimodal Continual Learning: A Comprehensive Surveycited by 39
  11. 2024FoundIR: Unleashing Million-scale Training Data to Advance Foundation Models for Image Restorationcited by 37
  12. 2025VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Modelscited by 31
38 more
  1. 2025IDEA: Inverted Text with Cooperative Deformable Aggregation for Multi-modal Object Re-Identificationcited by 27
  2. 2025Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuningcited by 27
  3. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22
  4. 2025Synthetic Data is an Elegant GIFT for Continual Vision-Language Modelscited by 21
  5. 2025A Survey on Efficient Vision-Language Modelscited by 19
  6. 2024CLIP with Generative Latent Replay: a Strong Baseline for Incremental Learningcited by 17
  7. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 16
  8. 2024Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Modelscited by 15
  9. 2025MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Mergingcited by 14
  10. 2024Learning More Generalized Experts by Merging Experts in Mixture-of-Expertscited by 12
  11. 2024LLMs Can Evolve Continually on Modality for X-Modal Reasoningcited by 12
  12. 2024Human-inspired Perspectives: A Survey on AI Long-term Memorycited by 12
  13. 2025Continual Text-to-Video Retrieval with Frame Fusion and Task-Aware Routingcited by 12
  14. 2025Addressing Imbalanced Domain-Incremental Learning through Dual-Balance Collaborative Expertscited by 12
  15. 2024Dynamic Integration of Task-Specific Adapters for Class Incremental Learningcited by 11
  16. 2025LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Modelscited by 11
  17. 2025Parameter-Efficient Continual Fine-Tuning: A Surveycited by 11
  18. 2025LADA: Scalable Label-Specific CLIP Adapter for Continual Learningcited by 11
  19. 2024Continual Adapter Tuning with Semantic Shift Compensation for Class-Incremental Learningcited by 10
  20. 2025External Knowledge Injection for CLIP-Based Class-Incremental Learningcited by 10
  21. 2025How to Teach Large Multimodal Models New Skillscited by 10
  22. 2025Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learningcited by 10
  23. 2024LW2G: Learning Whether to Grow for Prompt-based Continual Learningcited by 9
  24. 2024CAPrompt: Cyclic Prompt Aggregation for Pre-Trained Model Based Class Incremental Learningcited by 9
  25. 2024Onboard Satellite Image Classification for Earth Observation: A Comparative Study of ViT Modelscited by 7
  26. 2025IAP: Improving Continual Learning of Vision-Language Models via Instance-Aware Promptingcited by 7
  27. 2025StPR: Spatiotemporal Preservation and Routing for Exemplar-Free Video Class-Incremental Learningcited by 7
  28. 2025Continual Learning in Vision-Language Models via Aligned Model Mergingcited by 7
  29. 2024Towards Incremental Learning in Large Language Models: A Critical Reviewcited by 6
  30. 2024Exploiting the Semantic Knowledge of Pre-trained Text-Encoders for Continual Learningcited by 6
  31. 2024Brain-inspired continual pre-trained learner via silent synaptic consolidationcited by 6
  32. 2024One VLM to Keep it Learning: Generation and Balancing for Data-free Continual Visual Question Answeringcited by 6
  33. 2025VLM-Assisted Continual learning for Visual Question Answering in Self-Drivingcited by 6
  34. 2025Merge then Realign: Simple and Effective Modality-Incremental Continual Learning for Multimodal LLMscited by 6
  35. 2025Continual Learning on CLIP via Incremental Prompt Tuning with Intrinsic Textual Anchorscited by 6
  36. 2025Foundation Models for Scientific Discovery: From Paradigm Enhancement to Paradigm Transitioncited by 6
  37. 2025BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learningcited by 6
  38. 2025Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Modelscited by 6

50 most cited of 199 All 199 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Lifelong Learning in Vision-Language Models: Enhanced EWC with Cross-Modal Knowledge Retention
  2. 2024Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models
  3. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Models
  4. 2025Harnessing Textual Semantic Priors for Knowledge Transfer and Refinement in CLIP-Driven Continual Learning
  5. 2026MERGETUNE: Continued Fine-Tuning of Vision-Language Models
  6. 2024Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models
  7. 2026SAMBAR: Selective Anchoring via Method of Multipliers for Balanced Knowledge Acquisition and Retention in Vision-Language-Action Models
  8. 2025MLLM-CL: Continual Learning for Multimodal Large Language Models