mlsanityData to 7 Oct 2026

Seeing Before Answering: Training-Free Visual Layer Profiling for Vision-Language Models

arXiv 2608.16263 ↗First posted 17 Aug 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

LLaVA-style Vision-Language Models (VLMs) pass visual tokens from a fixed late layer of the vision backbone, typically the penultimate one, to the language model. We first show that this hidden convention is fragile: across 2 VLMs and 7 image and video benchmarks, the default layer is sub-optimal in 13 of 14 model-task pairs, and the best layer shifts with both task and visual backbone. Finding that layer by exhaustive layer-wise inference is prohibitively expensive, and no better fixed default exists. We therefore ask whether layer usefulness can instead be predicted from representation geometry. We study matrix-based entropy, introduced for unimodal layer analysis, which we compute over sample-level visual embeddings as Visual Dataset Entropy (VDE); and Gromov-Wasserstein (GW) distance, introduced for encoder-level VLM model selection, which we repurpose as a layer-wise visual--language alignment signal. Transferring these to LLaVA-based models is not obvious a priori: the vision tower is frozen while the multimodal projector is trained, so we profile both sides of the projector. We find that VDE transfers, and GW does not. Computed from 100 unlabeled task samples without downstream inference, pre-projector VDE tracks layer-wise accuracy and its top-ranked layers cover the oracle best layer on every task for the SigLIP-based LLaVA-Video, while giving region-level guidance for the CLIP-based Video-LLaVA. Post-projector profiles show that the projector reshapes visual geometry but does not erase the performance-relevant trend, leaving VDEpre the stronger signal. GW instead flattens after projection and is best read as an alignment diagnostic rather than a selector. VDE thus offers an interpretable, training-free policy that narrows the visual-layer search to a handful of candidates for limited downstream verification.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 32 references that are in the map. First posted on arXiv 17 Aug 2026.

Builds on

28 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance2 May
  2. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  9. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  10. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  11. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  12. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
14 more
  1. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  2. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  3. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  4. 2023The geometry of hidden representations of large transformer modelscited by 108
  5. 2025HD-EPIC: A Highly-Detailed Egocentric Video Datasetcited by 101
  6. 2024Emergence of a High-Dimensional Abstraction Phase in Language Transformerscited by 56
  7. 2024Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Modelscited by 35
  8. 2018Understanding Convolutional Neural Networks with Information Theory: An Initial Explorationcited by 31
  9. 2025Multimodal Language Models See Better When They Look Shallowercited by 22
  10. 2024MMFuser: Multimodal Multi-Layer Feature Fuser for Fine-Grained Vision-Language Understandingcited by 21
  11. 2025Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practicescited by 21
  12. 2023DiME: Maximizing Mutual Information by a Difference of Matrix-Based Entropiescited by 20
  13. 2025Visual Representations inside the Language Modelcited by 13
  14. 2019Information Plane Analysis of Deep Neural Networks via Matrix-Based Renyi's Entropy and Tensor Kernelscited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Prompt the Unseen: Evaluating Visual-Language Alignment Beyond Supervision
  2. 2025Adapting Vision-Language Models Without Labels: A Comprehensive Survey
  3. 2025Hidden in plain sight: VLMs overlook their visual representations
  4. 2025MAPS: Preserving Vision-Language Representations via Module-Wise Proximity Scheduling for Better Vision-Language-Action Generalization
  5. 2024Instruction-Guided Fusion of Multi-Layer Visual Features in Large Vision-Language Models
  6. 2026PRISM-VLM: A Multi-Axis Discriminative Benchmark for Compact Vision-Language Models
  7. 2026Large Vision-Language Models Get Lost in Attention
  8. 2024Rethinking VLMs and LLMs for Image Classification