mlsanityData to 14 Oct 2026

Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practices

arXiv 2503.06063 ↗First posted 8 Mar 2025cs.CVcited by 22

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have made significant advancements in recent years, with visual features playing an increasingly critical role in enhancing model performance. However, the integration of multi-layer visual features in MLLMs remains underexplored, particularly with regard to optimal layer selection and fusion strategies. Existing methods often rely on arbitrary design choices, leading to suboptimal outcomes. In this paper, we systematically investigate two core aspects of multi-layer visual feature fusion: (1) selecting the most effective visual layers and (2) identifying the best fusion approach with the language model. Our experiments reveal that while combining visual features from multiple stages improves generalization, incorporating additional features from the same stage typically leads to diminished performance. Furthermore, we find that direct fusion of multi-layer visual features at the input stage consistently yields superior and more stable performance across various configurations. We make all our code publicly available: https://github.com/EIT-NLP/Layer_Select_Fuse_for_MLLM.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 58 of its 68 references that are in the map. First posted on arXiv 8 Mar 2025.

Builds on

58 of 68 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024MMFuser: Multimodal Multi-Layer Feature Fuser for Fine-Grained Vision-Language Understanding15 Oct
  2. 2024NVLM: Open Frontier-Class Multimodal LLMs17 Sep
  3. 2024EVLM: An Efficient Vision-Language Model for Visual Understanding19 Jul
  4. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  5. 2024Dense Connector for MLLMs22 May
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  7. 2023Osprey: Pixel Understanding with Visual Instruction Tuning15 Dec
  8. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge20 Nov
  9. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
  10. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  11. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models13 Oct
  12. 2023Improved Baselines with Visual Instruction Tuning5 Oct
2 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2023Mistral 7Bcited by 3,164
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
32 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  3. 2015VQA: Visual Question Answeringcited by 2,506
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  7. 2019Towards VQA Models That Can Readcited by 1,718
  8. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  9. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  10. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  11. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  12. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  13. 2016A Diagram Is Worth A Dozen Imagescited by 804
  14. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  15. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  16. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  17. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  18. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  19. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  20. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  21. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  22. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  23. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 340
  24. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  25. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  26. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  27. 2015Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Featurecited by 182
  28. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 122
  29. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  30. 2024DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMscited by 75
  31. 2024Transformer Layers as Painterscited by 54
  32. 2019Reflective Decoding Network for Image Captioningcited by 31

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 14.

22 papers.

Sort:

By area:

  1. 2025VisiPruner: Decoding Discontinuous Cross-Modal Dynamics for Efficient Multimodal LLMscited by 17
  2. 2026Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadoxcited by 7
  3. 2025Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Modelscited by 5
  4. 2025Recurrence Meets Transformers for Universal Multimodal Retrievalcited by 3
  5. 2025HERMAN: Hierarchical Representation Matching for CLIP-based Class-Incremental Learningcited by 3
  6. 2026RETLLM: Training and Data-Free MLLMs for Multimodal Information Retrievalcited by 3
  7. 2025Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Modelscited by 2
  8. 2026TGIF: Text-Guided Layer Fusion Mitigates Hallucination in Multimodal LLMscited by 2
  9. 2025VAT: Vision Action Transformer by Unlocking Full Representation of ViTcited by 1
  10. 2025Dynamic Image Prompt Adapter for Scalable Zero-shot Personalized Text-to-Image Generationcited by 1
  11. 2026From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusioncited by 1
  12. 2026E.M.Ground: A Temporal Grounding Vid-LLM with Holistic Event Perception and Matchingcited by 1
10 more
  1. 2026Boosting Visual Instruction Tuning with Self-Supervised Guidancecited by 1
  2. 2025InstructPLM-mu: 1-Hour Fine-Tuning of ESM2 Beats ESM3 in Protein Mutation Predictionscited by 0
  3. 2026Detached Skip-Links and R-Probe: Decoupling Feature Aggregation from Gradient Propagation for MLLM OCRcited by 0
  4. 2026CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Modelscited by 0
  5. 2026The Hidden Evolution of Disguised Visual Context inside the VLMcited by 0
  6. 2026Seeing Before Answering: Training-Free Visual Layer Profiling for Vision-Language Modelscited by 0
  7. 2026PACER: Progressive Availability-Conditioned Evidence Routing for Radiology Report Generation under Incomplete Clinical Contextcited by 0
  8. 2026eRLT: Efficient VLA Reinforcement Learning via Action-Relevant Token Routingcited by 0
  9. 2026Where-OPD: Spatially Guided On-Policy Self-Distillation of MLLMs with Synthetic Scenescited by 0
  10. 2026Sparse All-Layer Connector for Domain Generalised Semantic Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Multimodal Language Models See Better When They Look Shallower
  2. 2024The Revolution of Multimodal Large Language Models: A Survey
  3. 2026Sparse Shortcuts: Facilitating Efficient Fusion in Multimodal Large Language Models
  4. 2024Instruction-Guided Fusion of Multi-Layer Visual Features in Large Vision-Language Models
  5. 2025Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniques
  6. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  7. 2024Efficient Multimodal Large Language Models: A Survey
  8. 2024A Survey on Benchmarks of Multimodal Large Language Models