# Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging

arXiv: https://arxiv.org/abs/2505.05464 · HTML: https://mlsanity.com/paper/2505.05464/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 31 of its 35 references that are in the map.
- First posted on arXiv 8 May 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://mlsanity.com/paper/2403.20330/) (2024)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://mlsanity.com/paper/2309.12284/) (2023)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?](https://mlsanity.com/paper/2403.14624/) (2024)
- [Dataless Knowledge Fusion by Merging Weights of Language Models](https://mlsanity.com/paper/2212.09849/) (2022)
- [Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing](https://mlsanity.com/paper/2406.08464/) (2024)
- [What matters when building vision-language models?](https://mlsanity.com/paper/2405.02246/) (2024)
- [AdaMerging: Adaptive Model Merging for Multi-Task Learning](https://mlsanity.com/paper/2310.02575/) (2023)
- [Evolutionary Optimization of Model Merging Recipes](https://mlsanity.com/paper/2403.13187/) (2024)
- [Composing Parameter-Efficient Modules with Arithmetic Operations](https://mlsanity.com/paper/2306.14870/) (2023)
- [DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models](https://mlsanity.com/paper/2411.00836/) (2024)
- [MAmmoTH2: Scaling Instructions from the Web](https://mlsanity.com/paper/2405.03548/) (2024)
- [Improve Vision Language Model Chain-of-thought Reasoning](https://mlsanity.com/paper/2410.16198/) (2024)
- [DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving](https://mlsanity.com/paper/2407.13690/) (2024)
- [What Matters for Model Merging at Scale?](https://mlsanity.com/paper/2410.03617/) (2024)
- [LogiCoT: Logical Chain-of-Thought Instruction-Tuning](https://mlsanity.com/paper/2305.12147/) (2023)
- [How Far Are We from Intelligent Visual Deductive Reasoning?](https://mlsanity.com/paper/2403.04732/) (2024)

## Built on by (most cited first, 30 of 40)

- [Detection and Mitigation of Hallucination in Large Reasoning Models: A Mechanistic Perspective](https://mlsanity.com/paper/2505.12886/) (2025)
- [Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training](https://mlsanity.com/paper/2509.26625/) (2025)
- [MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agent](https://mlsanity.com/paper/2511.18810/) (2025)
- [Perception Before Reasoning: Two-Stage Reinforcement Learning for Visual Reasoning in Vision-Language Models](https://mlsanity.com/paper/2509.13031/) (2025)
- [VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models](https://mlsanity.com/paper/2508.09945/) (2025)
- [CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMs](https://mlsanity.com/paper/2510.12184/) (2025)
- [Mapping Post-Training Forgetting in Language Models at Scale](https://mlsanity.com/paper/2510.17776/) (2025)
- [Towards Long-horizon Agentic Multimodal Search](https://mlsanity.com/paper/2604.12890/) (2026)
- [ProxyThinker: Test-Time Guidance through Small Visual Reasoners](https://mlsanity.com/paper/2505.24872/) (2025)
- [Unlocking Multilingual Reasoning Capability of LLMs and LVLMs through Representation Engineering](https://mlsanity.com/paper/2511.23231/) (2025)
- [L2V-CoT: Cross-Modal Transfer of Chain-of-Thought Reasoning via Latent Intervention](https://mlsanity.com/paper/2511.17910/) (2025)
- [What does RL improve for Visual Reasoning? A Frankenstein-Style Analysis](https://mlsanity.com/paper/2602.12395/) (2026)
- [Label-Free Cross-Task LoRA Merging with Null-Space Compression](https://mlsanity.com/paper/2603.26317/) (2026)
- [SVAgent: Storyline-Guided Long Video Understanding via Cross-Modal Multi-Agent Collaboration](https://mlsanity.com/paper/2604.05079/) (2026)
- [Training-Free Reasoning and Reflection in MLLMs](https://mlsanity.com/paper/2505.16151/) (2025)
- [MMedExpert-R1: Strengthening Multimodal Medical Reasoning via Domain-Specific Adaptation and Clinical Guideline Reinforcement](https://mlsanity.com/paper/2601.10949/) (2026)
- [PlanGPT-VL: Enhancing Urban Planning with Domain-Specific Vision-Language Models](https://mlsanity.com/paper/2505.14481/) (2025)
- [Can MLLMs Absorb Math Reasoning Abilities from LLMs as Free Lunch?](https://mlsanity.com/paper/2510.14387/) (2025)
- [Do All Individual Layers Help? An Empirical Study of Task-Interfering Layers in Vision-Language Models](https://mlsanity.com/paper/2602.01167/) (2026)
- [Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropy](https://mlsanity.com/paper/2603.26299/) (2026)
- [Lost in Adaptation: Layer-Selective Recovery of Temporal Reasoning in Video-Language Models](https://mlsanity.com/paper/2604.11399/) (2026)
- [From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Models](https://mlsanity.com/paper/2511.12861/) (2025)
- [Linear Model Merging Unlocks Simple and Scalable Multimodal Data Mixture Optimization](https://mlsanity.com/paper/2602.04937/) (2026)
- [ConsisGuard: Aligning Safety Deliberation with Policy Enforcement in LLM Guardrails](https://mlsanity.com/paper/2605.31073/) (2026)
- [Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning](https://mlsanity.com/paper/2606.01558/) (2026)
- [Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Merging](https://mlsanity.com/paper/2608.05541/) (2026)
- [Transferring Visual Explainability of Self-Explaining Models to Prediction-Only Models without Additional Training](https://mlsanity.com/paper/2507.04380/) (2025)
- [Beyond the Linear Separability Ceiling: Aligning Representations in VLMs](https://mlsanity.com/paper/2507.07574/) (2025)
- [Beyond Parameter Arithmetic: Sparse Complementary Fusion for Distribution-Aware Model Merging](https://mlsanity.com/paper/2602.11717/) (2026)
- [Securing the Floor and Raising the Ceiling: A Merging-based Paradigm for Multi-modal Search Agents](https://mlsanity.com/paper/2603.01416/) (2026)

## Abstract (arXiv)

Vision-Language Models (VLMs) combine visual perception with the general capabilities, such as reasoning, of Large Language Models (LLMs). However, the mechanisms by which these two abilities can be combined and contribute remain poorly understood. In this work, we explore to compose perception and reasoning through model merging that connects parameters of different models. Unlike previous works that often focus on merging models of the same kind, we propose merging models across modalities, enabling the incorporation of the reasoning capabilities of LLMs into VLMs. Through extensive experiments, we demonstrate that model merging offers a successful pathway to transfer reasoning abilities from LLMs to VLMs in a training-free manner. Moreover, we utilize the merged models to understand the internal mechanism of perception and reasoning and how merging affects it. We find that perception capabilities are predominantly encoded in the early layers of the model, whereas reasoning is largely facilitated by the middle-to-late layers. After merging, we observe that all layers begin to contribute to reasoning, whereas the distribution of perception abilities across layers remains largely unchanged. These observations shed light on the potential of model merging as a tool for multimodal integration and interpretation.