mlsanityData to 14 Oct 2026

MLLM-CL: Continual Learning for Multimodal Large Language Models

arXiv 2506.05453 ↗First posted 5 Jun 2025cs.CL cs.AI cs.CVcited by 23

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Recent Multimodal Large Language Models (MLLMs) excel in vision-language understanding but face challenges in adapting to dynamic real-world scenarios that require continuous integration of new knowledge and skills. While continual learning (CL) offers a potential solution, existing benchmarks and methods suffer from critical limitations. In this paper, we introduce MLLM-CL, a novel benchmark encompassing domain and ability continual learning, where the former focuses on independently and identically distributed (IID) evaluation across evolving mainstream domains, whereas the latter evaluates on non-IID scenarios with new model abilities. Methodologically, we propose preventing catastrophic interference through parameter isolation and an MLLM-based routing mechanism. Extensive experiments demonstrate that our approach can integrate domain-specific knowledge and functional abilities with minimal forgetting, significantly outperforming existing methods. Our benchmark and code are available at https://github.com/bjzhb666/MLLM-CL.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 65 of its 74 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

65 of 74 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning5 May
  2. 2025Federated Continual Instruction Tuning17 Mar
  3. 2025CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering1 Mar
  4. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Models4 Nov
  5. 2024CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model13 Mar
  6. 2023Continual Instruction Tuning for Large Multimodal Models27 Nov

Other references · 59

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2025Qwen2.5-VL Technical Reportcited by 5,587
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  12. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
38 more
  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  3. 2016Learning without Forgettingcited by 2,496
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  5. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  6. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  7. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
  8. 2021Learning to Prompt for Continual Learningcited by 944
  9. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
  10. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  11. 2016A Diagram Is Worth A Dozen Imagescited by 804
  12. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  13. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
  14. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
  15. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 450
  16. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  17. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
  18. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  19. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  20. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 205
  21. 2020RSVQA: Visual Question Answering for Remote Sensing Datacited by 203
  22. 2023Progressive Prompts: Continual Learning for Language Modelscited by 197
  23. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 197
  24. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 193
  25. 2021Towards Continual Knowledge Learning of Language Modelscited by 174
  26. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
  27. 2018TallyQA: Answering Complex Counting Questionscited by 153
  28. 2024Continual Learning for Large Language Models: A Surveycited by 147
  29. 2024Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Modelscited by 133
  30. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 120
  31. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 119
  32. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Modelscited by 119
  33. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 113
  34. 2022CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generationcited by 87
  35. 2022MapQA: A Dataset for Question Answering on Choropleth Mapscited by 85
  36. 2022Continual Pre-Training Mitigates Forgetting in Language and Visioncited by 83
  37. 2022ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshotscited by 66
  38. 2024Class-Incremental Learning with CLIP: Adaptive Representation Adjustment and Parameter Fusioncited by 47

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 18.

23 papers.

Sort:

By area:

  1. 2025Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Trainingcited by 47
  2. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 18
  3. 2025MCITlib: Multimodal Continual Instruction Tuning Library and Benchmarkcited by 13
  4. 2026Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptationcited by 5
  5. 2025Harmonious Parameter Adaptation in Continual Visual Instruction Tuning for Safety-Aligned MLLMscited by 3
  6. 2026Model-Dowser: Data-Free Importance Probing to Mitigate Catastrophic Forgetting in Multimodal Large Language Modelscited by 2
  7. 2026φ-DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Modelscited by 2
  8. 2026Fine-tuning MLLMs Without Forgetting Is Easier Than You Thinkcited by 2
  9. 2026CL-VISTA: Benchmarking Continual Learning in Video Large Language Modelscited by 2
  10. 2026Towards Human-like Physical Intelligence: Lifelong Vision-Language-Action Learning for Robotic Manipulationcited by 2
  11. 2025CurLL: A Developmental Framework to Evaluate Continual Learning in Language Modelscited by 1
  12. 2026Hidden Forgetting in Continual Multimodal Learning: When Accuracy Survives but Grounding Failscited by 1
11 more
  1. 2026CL-Anomaly: Layer-Adaptive Mixture-of-Experts with Multimodal Large Language Model for Continual Learning in Anomaly Detectioncited by 1
  2. 2026Continual Test-Time Adaptation in Computer Vision: Methods, Benchmarks, and Future Directionscited by 1
  3. 2026Structural Abstraction as an Inductive Bias for Non-Stationary Language Model Trainingcited by 0
  4. 2026Continual Vision-Language Learning for Remote Sensing: Benchmarking and Analysiscited by 0
  5. 2026MAny: Merge Anything for Multimodal Continual Instruction Tuningcited by 0
  6. 2026Reasoning Portability: Guiding Continual Learning for MLLMs in the RLVR Eracited by 0
  7. 2026KeepLoRA++: Continual Learning with Layer-Scaled Residual Gradient Adaptationcited by 0
  8. 2026Is Our Benchmark Enough? An Analysis of Continual Learning for MLLMscited by 0
  9. 2026SiGMA: Sign-Guided Merging and Adaptation for Multimodal Continual Instruction Tuningcited by 0
  10. 2026Beyond Routing Saturation: A Long-Horizon Class-Incremental Perspective on Expert Routing in Multimodal Continual Instruction Tuningcited by 0
  11. 2026Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Improving Multimodal Large Language Models Using Continual Learning
  2. 2025When Continue Learning Meets Multimodal Large Language Model: A Survey
  3. 2024Modality-Inconsistent Continual Learning of Multimodal Large Language Models
  4. 2026Continual-NExT: A Unified Comprehension And Generation Continual Learning Framework
  5. 2026Black-Box Continual Learning for Vision-Language Models
  6. 2025Merge then Realign: Simple and Effective Modality-Incremental Continual Learning for Multimodal LLMs
  7. 2025Routing-Based Continual Learning for Multimodal Large Language Models
  8. 2025Multimodal Continual Learning with MLLMs from Multi-scenario Perspectives