mlsanityData to 14 Oct 2026

Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging

arXiv 2505.05464 ↗First posted 8 May 2025cs.CLcited by 40

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Vision-Language Models (VLMs) combine visual perception with the general capabilities, such as reasoning, of Large Language Models (LLMs). However, the mechanisms by which these two abilities can be combined and contribute remain poorly understood. In this work, we explore to compose perception and reasoning through model merging that connects parameters of different models. Unlike previous works that often focus on merging models of the same kind, we propose merging models across modalities, enabling the incorporation of the reasoning capabilities of LLMs into VLMs. Through extensive experiments, we demonstrate that model merging offers a successful pathway to transfer reasoning abilities from LLMs to VLMs in a training-free manner. Moreover, we utilize the merged models to understand the internal mechanism of perception and reasoning and how merging affects it. We find that perception capabilities are predominantly encoded in the early layers of the model, whereas reasoning is largely facilitated by the middle-to-late layers. After merging, we observe that all layers begin to contribute to reasoning, whereas the distribution of perception abilities across layers remains largely unchanged. These observations shed light on the potential of model merging as a tool for multimodal integration and interpretation.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 35 references that are in the map. First posted on arXiv 8 May 2025.

Builds on

31 of 35 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models29 Oct
  2. 2024Improve Vision Language Model Chain-of-thought Reasoning21 Oct
  3. 2024How Far Are We from Intelligent Visual Deductive Reasoning?7 Mar
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2023Mistral 7Bcited by 3,164
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  9. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  10. 2022Editing Models with Task Arithmeticcited by 1,120
  11. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  12. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
15 more
  1. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  2. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  3. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  4. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  5. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  6. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  7. 2024What matters when building vision-language models?cited by 326
  8. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 252
  9. 2024Evolutionary Optimization of Model Merging Recipescited by 208
  10. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 156
  11. 2024MAmmoTH2: Scaling Instructions from the Webcited by 145
  12. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 95
  13. 2024What Matters for Model Merging at Scale?cited by 55
  14. 2023LogiCoT: Logical Chain-of-Thought Instruction-Tuningcited by 31
  15. 2024Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Modelscited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 16; 2026: 24.

40 papers.

Sort:

By area:

  1. 2025Detection and Mitigation of Hallucination in Large Reasoning Models: A Mechanistic Perspectivecited by 34
  2. 2025Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-trainingcited by 23
  3. 2025MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agentcited by 18
  4. 2025Perception Before Reasoning: Two-Stage Reinforcement Learning for Visual Reasoning in Vision-Language Modelscited by 17
  5. 2025VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Modelscited by 16
  6. 2025CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMscited by 14
  7. 2025Mapping Post-Training Forgetting in Language Models at Scalecited by 11
  8. 2026Towards Long-horizon Agentic Multimodal Searchcited by 11
  9. 2025ProxyThinker: Test-Time Guidance through Small Visual Reasonerscited by 8
  10. 2025Unlocking Multilingual Reasoning Capability of LLMs and LVLMs through Representation Engineeringcited by 8
  11. 2025L2V-CoT: Cross-Modal Transfer of Chain-of-Thought Reasoning via Latent Interventioncited by 5
  12. 2026What does RL improve for Visual Reasoning? A Frankenstein-Style Analysiscited by 4
28 more
  1. 2026Label-Free Cross-Task LoRA Merging with Null-Space Compressioncited by 4
  2. 2026SVAgent: Storyline-Guided Long Video Understanding via Cross-Modal Multi-Agent Collaborationcited by 4
  3. 2025Training-Free Reasoning and Reflection in MLLMscited by 3
  4. 2026MMedExpert-R1: Strengthening Multimodal Medical Reasoning via Domain-Specific Adaptation and Clinical Guideline Reinforcementcited by 3
  5. 2025PlanGPT-VL: Enhancing Urban Planning with Domain-Specific Vision-Language Modelscited by 2
  6. 2025Can MLLMs Absorb Math Reasoning Abilities from LLMs as Free Lunch?cited by 2
  7. 2026Do All Individual Layers Help? An Empirical Study of Task-Interfering Layers in Vision-Language Modelscited by 2
  8. 2026Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropycited by 2
  9. 2026Lost in Adaptation: Layer-Selective Recovery of Temporal Reasoning in Video-Language Modelscited by 2
  10. 2025From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Modelscited by 1
  11. 2026Linear Model Merging Unlocks Simple and Scalable Multimodal Data Mixture Optimizationcited by 1
  12. 2026ConsisGuard: Aligning Safety Deliberation with Policy Enforcement in LLM Guardrailscited by 1
  13. 2026Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoningcited by 1
  14. 2026Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Mergingcited by 1
  15. 2025Transferring Visual Explainability of Self-Explaining Models to Prediction-Only Models without Additional Trainingcited by 0
  16. 2025Beyond the Linear Separability Ceiling: Aligning Representations in VLMscited by 0
  17. 2026Beyond Parameter Arithmetic: Sparse Complementary Fusion for Distribution-Aware Model Mergingcited by 0
  18. 2026Securing the Floor and Raising the Ceiling: A Merging-based Paradigm for Multi-modal Search Agentscited by 0
  19. 2026CapVector: Learning Transferable Capability Vectors in Parametric Space for Vision-Language-Action Modelscited by 0
  20. 2026Investigating Cross-Modal Skill Injection: Scenarios, Methods, and Hyperparameterscited by 0
  21. 2026PlanBench-V: A Spatial Planning Map Benchmark for Vision-Language Modelscited by 0
  22. 2026Towards Truly Multilingual ASR: Generalizing Code-Switching ASR to Unseen Language Pairscited by 0
  23. 2026Recursive Vision Language Models for General Symbolic Reasoningcited by 0
  24. 2026Is Visual Prompting All You Need? Studying VLM Spatial Reasoning under Progressive Visual Scaffoldscited by 0
  25. 2026VFA: Empowering Multilingual MLLMs via Vision-Free Adaptationcited by 0
  26. 2026Video-FLAIR: Not Whether to Reason, But Howcited by 0
  27. 2026Can Linguistic Reasoning Vectors Enhance Multimodal Reasoning Ability?cited by 0
  28. 2026Orthogonal Yet Coupled: Decoupling Geometric Components for Model Mergingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models
  2. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Models
  3. 2026Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models
  4. 2025A Cognitive Paradigm Approach to Probe the Perception-Reasoning Interface in VLMs
  5. 2025See, Think, Learn: A Self-Taught Multimodal Reasoner
  6. 2025VERIFY: A Benchmark of Visual Explanation and Reasoning for Investigating Multimodal Reasoning Fidelity
  7. 2026The Last Visible Pixel: Probing Fine-Scale Perception in Vision-Language Models
  8. 2026Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap