mlsanityData to 14 Oct 2026

Fusion to Enhance: Fusion Visual Encoder to Enhance Multimodal Language Model

arXiv 2509.00664 ↗First posted 31 Aug 2025cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have made significant progress in bridging visual perception with high-level textual reasoning. However, they face a fundamental contradiction: while excelling at complex semantic understanding, these models often fail at basic visual tasks that require precise detail perception. This deficiency primarily stems from the prevalent architectural reliance on a single vision encoder optimized for high-level semantic alignment, which inherently sacrifices the ability to capture fine-grained visual information. To address this issue, we introduce Fusion to Enhance (FtZ), a novel vision tower framework. FtZ moves beyond the single-encoder design by innovatively composing a semantically powerful anchor encoder with a perception-rich augmenting encoder via a lightweight Multi-Head Cross-Attention mechanism. Experimental results demonstrate that on several challenging benchmarks demanding fine-grained visual understanding, such as TextVQA, POPE, MMMU, MME and MM-Vet, our FtZ model significantly outperforms baselines that use only a single encoder or existing feature fusion methods. This work proves that composing heterogeneous expert encoders is an efficient and effective path to overcoming the visual perception bottleneck in current MLLMs, offering a new design paradigm for building next-generation AI systems with stronger perceptual capabilities.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 39 references that are in the map. First posted on arXiv 31 Aug 2025.

Builds on

29 of 39 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs24 Feb
  2. 2025Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMs8 Jan
  3. 2024Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusion5 Dec
  4. 2024Improving Multi-modal Large Language Model through Boosting Vision Capabilities17 Oct
  5. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders28 Aug
  6. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks2 Aug
  7. 2024TinyLLaVA Factory: A Modularized Codebase for Small-scale Large Multimodal Models20 May
  8. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  9. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models13 Oct
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug

Other references · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  7. 2019Towards VQA Models That Can Readcited by 1,718
  8. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  9. 2018TallyQA: Answering Complex Counting Questionscited by 153
  10. 2021Towards artificial general intelligence via a multimodal foundation modelcited by 112
  11. 2017An Analysis of Visual Question Answering Algorithmscited by 90
  12. 2024Unified Hallucination Detection for Multimodal Large Language Modelscited by 75
7 more
  1. 2024Understanding the Limits of Vision Language Models Through the Lens of the Binding Problemcited by 75
  2. 2023Visual cognition in multimodal large language modelscited by 38
  3. 2024LLM Augmented LLMs: Expanding Capabilities through Compositioncited by 35
  4. 2025Visual Embodied Brain: Let Multimodal Large Language Models See, Think, and Control in Spacescited by 33
  5. 2025SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectoriescited by 30
  6. 2024CATCH: Complementary Adaptive Token-level Contrastive Decoding to Mitigate Hallucinations in LVLMscited by 12
  7. 2025Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Modelscited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion
  2. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge
  3. 2024MMFuser: Multimodal Multi-Layer Feature Fuser for Fine-Grained Vision-Language Understanding
  4. 2026Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attention
  5. 2024Instruction-Guided Fusion of Multi-Layer Visual Features in Large Vision-Language Models
  6. 2024DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception
  7. 2025Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Models
  8. 2025Unified Multimodal Understanding via Byte-Pair Visual Encoding