mlsanityData to 14 Oct 2026

AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization

arXiv 2503.23733 ↗First posted 31 Mar 2025cs.CL cs.CVcited by 25

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recently, model merging methods have demonstrated powerful strengths in combining abilities on various tasks from multiple Large Language Models (LLMs). While previous model merging methods mainly focus on merging homogeneous models with identical architecture, they meet challenges when dealing with Multimodal Large Language Models (MLLMs) with inherent heterogeneous property, including differences in model architecture and the asymmetry in the parameter space. In this work, we propose AdaMMS, a novel model merging method tailored for heterogeneous MLLMs. Our method tackles the challenges in three steps: mapping, merging and searching. Specifically, we first design mapping function between models to apply model merging on MLLMs with different architecture. Then we apply linear interpolation on model weights to actively adapt the asymmetry in the heterogeneous MLLMs. Finally in the hyper-parameter searching step, we propose an unsupervised hyper-parameter selection method for model merging. As the first model merging method capable of merging heterogeneous MLLMs without labeled data, extensive experiments on various model combinations demonstrated that AdaMMS outperforms previous model merging methods on various vision-language benchmarks.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 32 references that are in the map. First posted on arXiv 31 Mar 2025.

Builds on

25 of 32 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2024A Survey on Benchmarks of Multimodal Large Language Models16 Aug
  3. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  4. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models17 Jul
  5. 2024Model Composition for Multimodal Large Language Models20 Feb
  6. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  7. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  8. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 16

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  5. 2019Towards VQA Models That Can Readcited by 1,718
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  7. 2022Editing Models with Task Arithmeticcited by 1,120
  8. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  9. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  10. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  11. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  12. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
4 more
  1. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  2. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 194
  3. 2024MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmeticcited by 53
  4. 2023An Empirical Study of Multimodal Model Mergingcited by 52

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 21.

25 papers.

Sort:

By area:

  1. 2025MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agentcited by 18
  2. 2025FedMerge: Federated Personalization via Model Mergingcited by 4
  3. 2026Label-Free Cross-Task LoRA Merging with Null-Space Compressioncited by 4
  4. 2026Per-parameter Task Arithmetic for Unlearning in Large Language Modelscited by 2
  5. 2026Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropycited by 2
  6. 2026PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Mergingcited by 2
  7. 2025StatsMerging: Statistics-Guided Model Merging via Task-Specific Teacher Distillationcited by 1
  8. 2026Linear Model Merging Unlocks Simple and Scalable Multimodal Data Mixture Optimizationcited by 1
  9. 2026DiMtextsuperscript3: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Mergingcited by 1
  10. 2026Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspectivecited by 1
  11. 2026Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruningcited by 1
  12. 2026From Parameters to Behaviors: A Survey of Model Fusion for Large Language Modelscited by 1
13 more
  1. 2025Efficient Multi-Source Knowledge Transfer by Model Mergingcited by 0
  2. 2026SSAM: Singular Subspace Alignment for Merging Multimodal Large Language Modelscited by 0
  3. 2026Task Alignment: A Simple Proxy for Practical Model Merging Across Diverse Vision Taskscited by 0
  4. 2026Investigating Cross-Modal Skill Injection: Scenarios, Methods, and Hyperparameterscited by 0
  5. 2026Enhancing Multilingual Reasoning via Steerable Model Mergingcited by 0
  6. 2026Understanding Knowledge Transfer Mechanism in Heterogeneous MLLM Fusion: A Simple Linear Approachcited by 0
  7. 2026AgentPatch: Coarse-to-Fine Weak-Task Repair for Merging Agentic Multimodal Large Language Modelscited by 0
  8. 2026VFA: Empowering Multilingual MLLMs via Vision-Free Adaptationcited by 0
  9. 2026CoMerge: Conflict-Driven Preference Optimization for Multi-Task Model Mergingcited by 0
  10. 2026Tracing and Coordinating Cross-Layer Influence for Multimodal Model Mergingcited by 0
  11. 2026One Domain, Many Tongues: Composing Domain and Language LoRAs for Cross-Lingual Remote-Sensing MLLMs without Paired Datacited by 0
  12. 2026MergeHEIR: Mitigating Multimodal Hallucinations as the Tax of Model Mergingcited by 0
  13. 2026Exploring Heterogeneous Model Merging Approach for Complex Knowledge Transfercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Model Merging in the Era of Large Language Models: Methods, Applications, and Future Directions
  2. 2025Model Merging in Pre-training of Large Language Models
  3. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities
  4. 2025Fine, I'll Merge It Myself: A Multi-Fidelity Framework for Automated Model Merging
  5. 2025OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging
  6. 2024Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglement
  7. 2025Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging
  8. 2025Expert Merging: Model Merging with Unsupervised Expert Alignment and Importance-Guided Layer Chunking