mlsanityData to 7 Oct 2026

Late-Layer Fusion is Enough: Dual-Path Vision Token Routing for Multimodal Large Language Models under Visual Saturation

arXiv 2606.09131 ↗First posted 8 Jun 2026cs.AI cs.CL cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Multimodal large language models (MLLMs) commonly inherit the deep, symmetric Transformer backbone designed for unimodal text modeling, and apply the same computation uniformly to image and language tokens. This design overlooks a key modality asymmetry: image and text tokens differ substantially in information density, redundancy, and required reasoning depth. Through a layer-wise analysis of LLaVA-1.5, we observe that vision tokens tend to saturate in the middle layers. Specifically, text-to-image attention decreases from 0.68 at layer 0 to 0.07 by layer 4, and stabilizes near 0.04 after layer 18, whereas text tokens continue to benefit from deep semantic processing. These findings suggest a mismatch between architectural symmetry and depth-asynchronous modality evolution, resulting in redundant visual computation and possible drift in perceptual representations during deep task-specific adaptation. Motivated by this, we propose Dual-Path Vision Token Routing (DPVR), a modality-asymmetric routing framework for efficient MLLMs. Its core instantiation, DPVR-LF (Late-Layer Fusion), routes vision tokens at the saturation point into a one-layer trainable side branch, runs a thirteen-layer text-only forward that skips image positions in the deep stack, and re-fuses the visual and textual streams only at the final layer. With approximately 3% trainable parameters, DPVR-LF preserves competitive multimodal performance on standard benchmarks while reducing visual computation in the deep Transformer stack. The results challenge the conventional assumption that vision tokens must traverse all deep language-model layers, and indicate that a single late fusion layer can be sufficient for maintaining strong perceptual competence in LLaVA-style MLLMs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 34 references that are in the map. First posted on arXiv 8 Jun 2026.

Builds on

24 of 34 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024TokenPacker: Efficient Visual Projector for Multimodal LLM2 Jul
  2. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  10. 2022Token Merging: Your ViT But Fastercited by 995
  11. 2016Deep Networks with Stochastic Depthcited by 973
  12. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
10 more
  1. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  2. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  3. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  4. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  5. 2024Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTScited by 52
  6. 2025AStar: Boosting Multimodal Reasoning with Automated Structured Thinkingcited by 25
  7. 2025RadialRouter: Structured Representation for Efficient and Robust Large Language Models Routingcited by 11
  8. 2026Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMscited by 9
  9. 2025DReSS: Data-driven Regularized Structured Streamlining for Large Language Modelscited by 3
  10. 2025Two-Stage Regularization-Based Structured Pruning for LLMscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference
  2. 2025Multimodal Language Models See Better When They Look Shallower
  3. 2026Do Vision Language Models Need to Process Image Tokens?
  4. 2025The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Loss
  5. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference
  6. 2024Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings
  7. 2026SPIDER: Multi-Layer Semantic Token Pruning and Adaptive Sub-Layer Skipping in Multimodal Large Language Models
  8. 2025Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment