# ES-Merging: Biological MLLM Merging via Embedding Space Signals

arXiv: https://arxiv.org/abs/2603.14405 · HTML: https://mlsanity.com/paper/2603.14405/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 45 of its 68 references that are in the map.
- First posted on arXiv 15 Mar 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 30 of 45)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022)
- [LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment](https://mlsanity.com/paper/2310.01852/) (2023)
- [LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition](https://mlsanity.com/paper/2307.13269/) (2023)
- [ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction](https://mlsanity.com/paper/2010.09885/) (2020)
- [Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development](https://mlsanity.com/paper/2102.09548/) (2021)
- [AdaMerging: Adaptive Model Merging for Multi-Task Learning](https://mlsanity.com/paper/2310.02575/) (2023)
- [Learning to Answer Questions in Dynamic Audio-Visual Scenarios](https://mlsanity.com/paper/2203.14072/) (2022)
- [Translation between Molecules and Natural Language](https://mlsanity.com/paper/2204.11817/) (2022)
- [Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models](https://mlsanity.com/paper/2306.08018/) (2023)
- [Localizing Task Information for Improved Model Merging and Compression](https://mlsanity.com/paper/2405.07813/) (2024)
- [MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter](https://mlsanity.com/paper/2310.12798/) (2023)
- [LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset](https://mlsanity.com/paper/2402.09391/) (2024)
- [Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging](https://mlsanity.com/paper/2406.15479/) (2024)
- [EMR-Merging: Tuning-Free High-Performance Model Merging](https://mlsanity.com/paper/2405.17461/) (2024)
- [BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations](https://mlsanity.com/paper/2310.07276/) (2023)
- [InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discovery](https://mlsanity.com/paper/2311.16208/) (2023)
- [ProtST: Multi-Modality Learning of Protein Sequences and Biomedical Texts](https://mlsanity.com/paper/2301.12040/) (2023)
- [Parameter Competition Balancing for Model Merging](https://mlsanity.com/paper/2410.02396/) (2024)
- [Towards 3D Molecule-Text Interpretation in Language Models](https://mlsanity.com/paper/2401.13923/) (2024)
- [Prot2Text: Multimodal Protein's Function Generation with GNNs and Transformers](https://mlsanity.com/paper/2307.14367/) (2023)
- [RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustness](https://mlsanity.com/paper/2502.17159/) (2025)

## Abstract (arXiv)

Biological multimodal large language models (MLLMs) have emerged as powerful foundation models for scientific discovery. However, existing models are specialized to a single modality, limiting their ability to solve inherently cross-modal scientific problems. While model merging is an efficient method to combine the different modalities into a unified MLLM, existing methods rely on input-agnostic parameter space heuristics that fail to faithfully capture modality specialization. To overcome this limitation, we propose the Embedding-Signal-based MLLM Merging (ES-Merging), a framework that estimates merging coefficients from embedding space signals, moving the merging paradigm from the parameter signals to the embedding signals. ES-Merging exploits coarse-grained and fine-grained signals from embedding space to estimate the layer-wise and element-wise merging coefficients, respectively, which are jointly combined for complementary coefficient estimation. Through extensive experiments, we demonstrate that ES-Merging outperforms existing merging methods not only on the cross-modal reasoning but also on the single-modal knowledge preservation, establishing that embedding space signals provide a principled and effective foundation for MLLM merging.