# Hidden in plain sight: VLMs overlook their visual representations

arXiv: https://arxiv.org/abs/2506.08008 · HTML: https://mlsanity.com/paper/2506.08008/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jun 2025.

## Built on by (most cited first, 12 of 12)

- [Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens](https://mlsanity.com/paper/2511.19418/) (2025)
- [Position: Reasoning After Perception Means Reasoning Without Vision](https://mlsanity.com/paper/2507.16863/) (2025)
- [What does RL improve for Visual Reasoning? A Frankenstein-Style Analysis](https://mlsanity.com/paper/2602.12395/) (2026)
- [Vision Language Models Cannot Reason About Physical Transformation](https://mlsanity.com/paper/2603.07109/) (2026)
- [Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs](https://mlsanity.com/paper/2604.12896/) (2026)
- [Downscaling Intelligence: Exploring Perception and Reasoning Bottlenecks in Small Multimodal Models](https://mlsanity.com/paper/2511.17487/) (2025)
- [Visual Access Boundaries in Vision-Language Model Reasoning](https://mlsanity.com/paper/2607.12815/) (2026)
- [Do multimodal models imagine electric sheep?](https://mlsanity.com/paper/2605.09693/) (2026)
- [Teaching MLLMs to Say No: Generalized Referring Expression Comprehension via Refusal Calibrated GRPO](https://mlsanity.com/paper/2608.04698/) (2026)
- [Tinted Frames: Question Framing Blinds Vision-Language Models](https://mlsanity.com/paper/2603.19203/) (2026)
- [DiaVLo: Diagnosing Behaviours of Vision-Language Models](https://mlsanity.com/paper/2609.22008/) (2026)
- [Semantic Capability Acquisition and Specialization During Vision-Language Model Fine-Tuning](https://mlsanity.com/paper/2610.07385/) (2026)

## Abstract (arXiv)

Language provides a natural interface to specify and evaluate performance on visual tasks. To realize this possibility, vision language models (VLMs) must successfully integrate visual and linguistic information. Our work compares VLMs to a direct readout of their visual encoders to understand their ability to integrate across these modalities. Across a series of vision-centric benchmarks (e.g., depth estimation, correspondence), we find that VLMs perform substantially worse than their visual encoders, dropping to near-chance performance. We investigate these results through a series of analyses across the entire VLM: namely 1) the degradation of vision representations, 2) brittleness to task prompt, and 3) the language model's role in solving the task. We find that the bottleneck in performing these vision-centric tasks lies in this third category; VLMs are not effectively using visual information easily accessible throughout the entire model, and they inherit the language priors present in the LLM. Our work helps diagnose the failure modes of open-source VLMs, and presents a series of evaluations useful for future investigations into visual understanding within VLMs.