# Is Our Benchmark Enough? An Analysis of Continual Learning for MLLMs

arXiv: https://arxiv.org/abs/2606.20961 · HTML: https://mlsanity.com/paper/2606.20961/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 18 Jun 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 25 of 25)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Orthogonal Subspace Learning for Language Model Continual Learning](https://mlsanity.com/paper/2310.14152/) (2023)
- [InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning](https://mlsanity.com/paper/2404.00228/) (2024)
- [Mixture of LoRA Experts](https://mlsanity.com/paper/2404.13628/) (2024)
- [MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts](https://mlsanity.com/paper/2404.15159/) (2024)
- [When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications](https://mlsanity.com/paper/2310.18339/) (2023)
- [Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts](https://mlsanity.com/paper/2405.11273/) (2024)
- [Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning](https://mlsanity.com/paper/2312.12379/) (2023)
- [CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks](https://mlsanity.com/paper/2206.09059/) (2022)
- [SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning](https://mlsanity.com/paper/2501.13198/) (2025)
- [CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model](https://mlsanity.com/paper/2403.08350/) (2024)
- [Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models](https://mlsanity.com/paper/2403.03432/) (2024)
- [AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Experts](https://mlsanity.com/paper/2405.00361/) (2024)
- [SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2505.02486/) (2025)
- [SiRA: Sparse Mixture of Low Rank Adaptation](https://mlsanity.com/paper/2311.09179/) (2023)
- [Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptation](https://mlsanity.com/paper/2411.05663/) (2024)
- [MLLM-CL: Continual Learning for Multimodal Large Language Models](https://mlsanity.com/paper/2506.05453/) (2025)
- [LoRA Subtraction for Drift-Resistant Space in Exemplar-Free Continual Learning](https://mlsanity.com/paper/2503.18985/) (2025)
- [MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark](https://mlsanity.com/paper/2508.07307/) (2025)
- [Learning Attentional Mixture of LoRAs for Language Model Continual Learning](https://mlsanity.com/paper/2409.19611/) (2024)
- [MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2506.05928/) (2025)

## Abstract (arXiv)

Continual adaptation is essential for multimodal large language models (MLLMs) deployed across evolving domains, but the state-of-the-art MR-LoRA method highly relies on the assumption that a MLLM-based router is necessary to process complex multimodal inputs. This paper revisits this claim on the MLLM-CL benchmark and argues for two claims. First, routing does not require an MLLM: a simple training-free, replay-free ptotypical routing method (textscRePRo), uses frozen pretrained features and task prototypes to match the MLLM-based router of MR-LoRA at far lower computational cost. Second, shared experts do not improve continual learning for MLLMs, despite their theoretical appeal. We show that these findings arise from two structural limitations of MLLM-CL: (1) its tasks are highly separable in representation space, and (2) its fixed task order makes conclusions sensitive to a single curriculum rather than robust across diverse continual-learning trajectories. As a result, the benchmark primarily rewards learning in isolation rather than genuine continual transfer. This motivates a new design for future benchmarks of continual MLLM learning, with overlapping task manifolds, multiple task orders, fine-grained domain shifts, and evaluation protocols that reward forward transfer as well as retention.