mlsanityData to 14 Oct 2026

Model Composition for Multimodal Large Language Models

arXiv 2402.12750 ↗First posted 20 Feb 2024cs.CV cs.AI cs.CLcited by 12

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recent developments in Multimodal Large Language Models (MLLMs) have shown rapid progress, moving towards the goal of creating versatile MLLMs that understand inputs from various modalities. However, existing methods typically rely on joint training with paired multimodal instruction data, which is resource-intensive and challenging to extend to new modalities. In this paper, we propose a new paradigm through the model composition of existing MLLMs to create a new model that retains the modal understanding capabilities of each original model. Our basic implementation, NaiveMC, demonstrates the effectiveness of this paradigm by reusing modality encoders and merging LLM parameters. Furthermore, we introduce DAMC to address parameter interference and mismatch issues during the merging process, thereby enhancing the model performance. To facilitate research in this area, we propose MCUB, a benchmark for assessing ability of MLLMs to understand inputs from diverse modalities. Experiments on this benchmark and four other multimodal understanding tasks show significant improvements over baselines, proving that model composition can create a versatile model capable of processing inputs from multiple modalities.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 42 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

32 of 42 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  3. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration15 Jun
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 28

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  7. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  8. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  9. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  10. 2021Robust fine-tuning of zero-shot modelscited by 797
  11. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  12. 2021Merging Models with Fisher-Weighted Averagingcited by 621
16 more
  1. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  2. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  3. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  4. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  5. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  6. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  7. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 267
  8. 2023Pengi: An Audio Language Model for Audio Taskscited by 220
  9. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 191
  10. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  11. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 128
  12. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 80
  13. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 66
  14. 2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64
  15. 2023An Empirical Study of Multimodal Model Mergingcited by 52
  16. 2023Multimodal Attention Merging for Improved Speech Recognition and Audio Event Classificationcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 6.

12 papers.

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  2. 2024View Selection for 3D Captioning via Diffusion Rankingcited by 32
  3. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  4. 2025AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimizationcited by 25
  5. 2025MokA: Multimodal Low-Rank Adaptation for MLLMscited by 6
  6. 2026Per-parameter Task Arithmetic for Unlearning in Large Language Modelscited by 2
  7. 2026PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Mergingcited by 2
  8. 2025Sample-efficient Integration of New Modalities into Large Language Modelscited by 1
  9. 2026Dynamic Model Merging Made Slimcited by 1
  10. 2026ES-Merging: Biological MLLM Merging via Embedding Space Signalscited by 0
  11. 2026SSAM: Singular Subspace Alignment for Merging Multimodal Large Language Modelscited by 0
  12. 2026Tracing and Coordinating Cross-Layer Influence for Multimodal Model Mergingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniques
  2. 2025OrchMLLM: Orchestrate Multimodal Data with Batch Post-Balancing to Accelerate Multimodal Large Language Model Training
  3. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective
  4. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  5. 2024The Revolution of Multimodal Large Language Models: A Survey
  6. 2026Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs
  7. 2025Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs
  8. 2026Alignment Is All You Need: Instruction-Free Training for General Audio-Language Models