mlsanityData to 7 Oct 2026

CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering

arXiv 2503.00413 ↗First posted 1 Mar 2025cs.CV cs.LGcited by 36

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

also in 1 more
Abstract (arXiv)

Multimodal large language models (MLLMs) have garnered widespread attention from researchers due to their remarkable understanding and generation capabilities in visual language tasks (e.g., visual question answering). However, the rapid pace of knowledge updates in the real world makes offline training of MLLMs costly, and when faced with non-stationary data streams, MLLMs suffer from catastrophic forgetting during learning. In this paper, we propose an MLLMs-based dual momentum Mixture-of-Experts (CL-MoE) framework for continual visual question answering (VQA). We integrate MLLMs with continual learning to utilize the rich commonsense knowledge in LLMs. We introduce a Dual-Router MoE (RMoE) strategy to select the global and local experts using task-level and instance-level routers, to robustly assign weights to the experts most appropriate for the task. Then, we design a dynamic Momentum MoE (MMoE) to update the parameters of experts dynamically based on the relationships between the experts and tasks/instances, so that the model can absorb new knowledge while maintaining existing knowledge. The extensive experimental results indicate that our method achieves state-of-the-art performance on 10 VQA tasks, proving the effectiveness of our approach.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 49 references that are in the map. First posted on arXiv 1 Mar 2025.

Builds on

35 of 49 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Class Incremental Learning with Pre-trained Vision-Language Models31 Oct
  2. 2023Learning without Forgetting for Vision-Language Models30 May

Other references · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
21 more
  1. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  2. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  3. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  5. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  6. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  7. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  8. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  9. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  10. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  11. 2019On Tiny Episodic Memories in Continual Learningcited by 299
  12. 2023Pengi: An Audio Language Model for Audio Taskscited by 217
  13. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  14. 2024Mixture of LoRA Expertscited by 179
  15. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  16. 2019Overcoming Catastrophic Forgetting by Neuron-level Plasticity Controlcited by 36
  17. 2024VLM-PL: Advanced Pseudo Labeling Approach for Class Incremental Object Detection via Vision-Language Modelcited by 23
  18. 2021Video-Text Pre-training with Learned Regionscited by 20
  19. 2022Continual Learning for Visual Search with Backward Consistent Feature Embeddingcited by 17
  20. 2024Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learningcited by 14
  21. 2024Boosting Large Language Models with Continual Learning for Aspect-based Sentiment Analysiscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 25.

36 papers.

Sort:

By area:

  1. 2025EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecyclecited by 129
  2. 2025MLLM-CL: Continual Learning for Multimodal Large Language Modelscited by 21
  3. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  4. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 16
  5. 2025MCITlib: Multimodal Continual Instruction Tuning Library and Benchmarkcited by 10
  6. 2025Multimodal Continual Instruction Tuning with Dynamic Gradient Guidancecited by 6
  7. 2025Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Modelscited by 6
  8. 2025Task-Core Memory Management and Consolidation for Long-term Continual Learningcited by 4
  9. 2025Affordance-First Decomposition for Continual Learning in Video-Language Understandingcited by 4
  10. 2026Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptationcited by 4
  11. 2025Reinforced Interactive Continual Learning via Real-time Noisy Human Feedbackcited by 3
  12. 2025Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuningcited by 3
24 more
  1. 2026Elastic Weight Consolidation Done Right for Continual Learningcited by 3
  2. 2026φ-DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Modelscited by 2
  3. 2026On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Modelscited by 2
  4. 2026Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuningcited by 2
  5. 2026CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learningcited by 2
  6. 2026CL-VISTA: Benchmarking Continual Learning in Video Large Language Modelscited by 1
  7. 2026CoGR-MoE: Concept-Guided Expert Routing with Consistent Selection and Flexible Reasoning for Visual Question Answeringcited by 1
  8. 2026Stable Routing for Mixture-of-Experts in Class-Incremental Learningcited by 1
  9. 2026Attention-Spectrum Regularization for Replay-Free Continual Multimodal LLMscited by 1
  10. 2026Hidden Forgetting in Continual Multimodal Learning: When Accuracy Survives but Grounding Failscited by 1
  11. 2026CL-Anomaly: Layer-Adaptive Mixture-of-Experts with Multimodal Large Language Model for Continual Learning in Anomaly Detectioncited by 1
  12. 2026From Isolation to Integration: Building an Adaptive Expert Forest for Pre-Trained Model-based Class-Incremental Learningcited by 0
  13. 2026Toward Structural Multimodal Representations: Specialization, Selection, and Sparsification via Mixture-of-Expertscited by 0
  14. 2026Octopus: History-Free Gradient Orthogonalization for Continual Learning in Multimodal Large Language Modelscited by 0
  15. 2026ACE-LoRA: Adaptive Orthogonal Decoupling for Continual Image Editingcited by 0
  16. 2026StrLoRA: Towards Streaming Continual Visual Instruction Tuning for MLLMscited by 0
  17. 2026iGSP:Implicit Gradient Subspace Projection for Efficient Continual Learning of Vision-Language Modelscited by 0
  18. 2026Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Surveycited by 0
  19. 2026GC-MoE: Genomics-Guided Cell-Type-Specific Mixture of Experts for Histology-Based Single-Cell Spatial Transcriptomicscited by 0
  20. 2026FOGO: Forgetting-aware Orthogonalization Optimizercited by 0
  21. 2026KeepLoRA++: Continual Learning with Layer-Scaled Residual Gradient Adaptationcited by 0
  22. 2026General Incomplete Multimodal Learning via Dynamic Quality Perceptioncited by 0
  23. 2026SiGMA: Sign-Guided Merging and Adaptation for Multimodal Continual Instruction Tuningcited by 0
  24. 2026ReCAP: Retrieval-Guided Capability Reuse for Multimodal Continual Instruction Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024One VLM to Keep it Learning: Generation and Balancing for Data-free Continual Visual Question Answering
  2. 2022CL-CrossVQA: A Continual Learning Benchmark for Cross-Domain Visual Question Answering
  3. 2024CluMo: Cluster-based Modality Fusion Prompt for Continual Learning in Visual Question Answering
  4. 2025LLaVA-CMoE: Towards Continual Mixture of Experts for Large Vision-Language Models
  5. 2026Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation
  6. 2026Fine-tuning MLLMs Without Forgetting Is Easier Than You Think
  7. 2024Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering
  8. 2026AIM: Asymmetric Information Masking for Visual Question Answering Continual Learning