mlsanityData to 7 Oct 2026

DIMoE-Adapters: Dynamic Expert Evolution for Continual Learning in Vision-Language Models

arXiv 2605.07494 ↗First posted 8 May 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Continual learning enables vision-language models to accumulate knowledge and adapt to evolving tasks without retraining from scratch. However, in multi-domain task-incremental learning, large domain shifts intensify the stability-plasticity dilemma. Most existing methods rely on fixed architectures with statically allocated parameters, which limits adaptation to new domains and aggravates catastrophic forgetting. To address these challenges, we propose DIMoE-Adapters, a Dynamic Incremental Mixture-of-Experts Adapters framework that introduces a dynamic expert evolution paradigm to balance stability and plasticity. This paradigm is implemented through two collaborative components: Self-Calibrated Expert Evolution (SCEE) and Prototype-Guided Expert Selection (PGES). SCEE constructs and evolves a sparse expert pool through expert optimization dynamics, improving plasticity while reducing redundant capacity. PGES controls expert utilization based on the pool shaped by SCEE, improving stability across both previously encountered and unseen tasks. Extensive experiments show that DIMoE-Adapters outperforms previous state-of-the-art methods across various settings.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 59 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

42 of 59 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Synthetic Data is an Elegant GIFT for Continual Vision-Language Models6 Mar
  2. 2024CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Models28 Mar
  3. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters18 Mar
  4. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models12 Mar
  5. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  6. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2016Learning without Forgettingcited by 2,465
  6. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  7. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  8. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  9. 2013Describing Textures in the Wildcited by 2,110
  10. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  11. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  12. 2022Visual Prompt Tuningcited by 1,674
24 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  3. 2016Progressive Neural Networkscited by 1,332
  4. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  5. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  6. 2014How transferable are features in deep neural networks?cited by 1,216
  7. 2017Continual Learning with Deep Generative Replaycited by 1,152
  8. 2019When Does Label Smoothing Help?cited by 985
  9. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  10. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  11. 2021Robust fine-tuning of zero-shot modelscited by 785
  12. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  13. 2018Generalizing to Unseen Domains via Adversarial Data Augmentationcited by 451
  14. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
  15. 2019Transfusion: Understanding Transfer Learning for Medical Imagingcited by 366
  16. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  17. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  18. 2018Learning without Memorizingcited by 249
  19. 2021GCR: Gradient Coreset Based Replay Buffer Selection For Continual Learningcited by 95
  20. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Modelscited by 71
  21. 2023Lifelong Language Pretraining with Distribution-Specialized Expertscited by 64
  22. 2022Don't Stop Learning: Towards Continual Learning for the CLIP Modelcited by 61
  23. 2024Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learningcited by 53
  24. 2020Incremental Embedding Learning via Zero-Shot Translationcited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Decomposing and Composing: Towards Efficient Vision-Language Continual Learning via Rank-1 Expert Pool in a Single LoRA
  2. 2025LLaVA-CMoE: Towards Continual Mixture of Experts for Large Vision-Language Models
  3. 2025Continual Learning in Vision-Language Models via Aligned Model Merging
  4. 2026CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
  5. 2026On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Models
  6. 2026Dynamic Mixture of Latent Memories for Self-Evolving Agents
  7. 2026CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning
  8. 2026M2Note: Continual Evolution of Vision Language Models via Mistake Notebook Learning