mlsanityData to 7 Oct 2026

Merge then Realign: Simple and Effective Modality-Incremental Continual Learning for Multimodal LLMs

arXiv 2503.07663 ↗First posted 8 Mar 2025cs.LG cs.AIcited by 6

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Recent advances in Multimodal Large Language Models (MLLMs) have enhanced their versatility as they integrate a growing number of modalities. Considering the heavy cost of training MLLMs, it is efficient to reuse the existing ones and extend them to more modalities through Modality-incremental Continual Learning (MCL). The exploration of MCL is in its early stages. In this work, we dive into the causes of performance degradation in MCL. We uncover that it suffers not only from forgetting as in traditional continual learning, but also from misalignment between the modality-agnostic and modality-specific components. To this end, we propose an elegantly simple MCL paradigm called "MErge then ReAlign" (MERA) to address both forgetting and misalignment. MERA avoids introducing heavy model budgets or modifying model architectures, hence is easy to deploy and highly reusable in the MLLM community. Extensive experiments demonstrate the impressive performance of MERA, holding an average of 99.84% Backward Relative Gain when extending to four modalities, achieving nearly lossless MCL performance. Our findings underscore the misalignment issue in MCL. More broadly, our work showcases how to adjust different components of MLLMs during continual learning.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 70 references that are in the map. First posted on arXiv 8 Mar 2025.

Builds on

48 of 70 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters18 Mar
  2. 2023Continual Instruction Tuning for Large Multimodal Models27 Nov

Other references · 46

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  9. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  10. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  11. 2018Experience Replay for Continual Learningcited by 1,074
  12. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
34 more
  1. 2021Robust fine-tuning of zero-shot modelscited by 785
  2. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  3. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  4. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  5. 2019Three scenarios for continual learningcited by 610
  6. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  7. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  8. 2020What is being transferred in transfer learning?cited by 365
  9. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  10. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  11. 2021New Insights on Reducing Abrupt Representation Change in Online Continual Learningcited by 215
  12. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  13. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  14. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 172
  15. 2022Fine-tuned Language Models are Continual Learnerscited by 152
  16. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  17. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 127
  18. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 107
  19. 2024Representation Surgery for Multi-Task Model Mergingcited by 95
  20. 2021Achieving Forgetting Prevention and Knowledge Transfer in Continual Learningcited by 91
  21. 2022Clotho-AQA: A Crowdsourced Dataset for Audio Question Answeringcited by 84
  22. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
  23. 2023VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understandingcited by 62
  24. 2022Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfercited by 59
  25. 2024InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructionscited by 54
  26. 2024What Matters for Model Merging at Scale?cited by 52
  27. 2023Multimodal Neurons in Pretrained Text-Only Transformerscited by 49
  28. 2022Parameter-Efficient and Student-Friendly Knowledge Distillationcited by 47
  29. 2023LM-Cocktail: Resilient Tuning of Language Models via Model Mergingcited by 44
  30. 2023Maintaining Plasticity in Deep Continual Learningcited by 42
  31. 2024Cross-Modal Projection in Multimodal LLMs Doesn't Really Project Visual Attributes to Textual Spacecited by 21
  32. 2024CommonIT: Commonality-Aware Instruction Tuning for Large Language Models via Data Partitionscited by 13
  33. 2024LLMs Can Evolve Continually on Modality for X-Modal Reasoningcited by 12
  34. 2023I2I: Initializing Adapters with Improvised Knowledgecited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learningcited by 7
  2. 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2
  3. 2026MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inferencecited by 1
  4. 2026Fast-Slow Efficient Training for Multimodal Large Language Models via Visual Token Pruningcited by 0
  5. 2026Continual Vision-Language Learning for Remote Sensing: Benchmarking and Analysiscited by 0
  6. 2026VFA: Empowering Multilingual MLLMs via Vision-Free Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Modality-Inconsistent Continual Learning of Multimodal Large Language Models
  2. 2025MLLM-CL: Continual Learning for Multimodal Large Language Models
  3. 2025When Continue Learning Meets Multimodal Large Language Model: A Survey
  4. 2026Is Our Benchmark Enough? An Analysis of Continual Learning for MLLMs
  5. 2024Improving Multimodal Large Language Models Using Continual Learning
  6. 2025Multi-Modality Expansion and Retention for LLMs through Parameter Merging and Decoupling
  7. 2026Continual-NExT: A Unified Comprehension And Generation Continual Learning Framework
  8. 2026MAny: Merge Anything for Multimodal Continual Instruction Tuning