# VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchors

arXiv: https://arxiv.org/abs/2604.02486 · HTML: https://mlsanity.com/paper/2604.02486/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 2 Apr 2026.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 23 of 23)

- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?](https://mlsanity.com/paper/2403.14624/) (2024)
- [BLINK: Multimodal Large Language Models Can See but Not Perceive](https://mlsanity.com/paper/2404.12390/) (2024)
- [A Neural Algorithm of Artistic Style](https://mlsanity.com/paper/1508.06576/) (2015)
- [ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems](https://mlsanity.com/paper/2505.11831/) (2025)
- [SPair-71k: A Large-scale Benchmark for Semantic Correspondence](https://mlsanity.com/paper/1908.10543/) (2019)
- [Linearly Mapping from Image to Text Space](https://mlsanity.com/paper/2209.15162/) (2022)
- [Towards Interpreting Visual Information Processing in Vision-Language Models](https://mlsanity.com/paper/2410.07149/) (2024)
- [What's in the Image? A Deep-Dive into the Vision of Vision Language Models](https://mlsanity.com/paper/2411.17491/) (2024)
- [Vision Language Models are Biased](https://mlsanity.com/paper/2505.23941/) (2025)
- [GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Tasks](https://mlsanity.com/paper/2411.19325/) (2024)
- [VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledge](https://mlsanity.com/paper/2411.12915/) (2024)
- [Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs](https://mlsanity.com/paper/2506.09047/) (2025)
- [IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Models](https://mlsanity.com/paper/2403.15952/) (2024)
- [Visual Representations inside the Language Model](https://mlsanity.com/paper/2510.04819/) (2025)
- [LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs](https://mlsanity.com/paper/2602.00462/) (2026)
- [Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Models](https://mlsanity.com/paper/2412.14133/) (2024)
- [Towards Understanding How Knowledge Evolves in Large Vision-Language Models](https://mlsanity.com/paper/2504.02862/) (2025)
- [VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning?](https://mlsanity.com/paper/2603.07888/) (2026)
- [LLMs Can Compensate for Deficiencies in Visual Representations](https://mlsanity.com/paper/2506.05439/) (2025)
- [Anatomy-VLM: A Fine-grained Vision-Language Model for Medical Interpretation](https://mlsanity.com/paper/2511.08402/) (2025)
- [FLUXSynID: A Framework for Identity-Controlled Synthetic Face Generation with Document and Live Images](https://mlsanity.com/paper/2505.07530/) (2025)
- [Glo-VLMs: Leveraging Vision-Language Models for Fine-Grained Diseased Glomerulus Classification](https://mlsanity.com/paper/2508.15960/) (2025)

## Built on by (most cited first, 13 of 13)

- [3D-Anchored Lookahead Planning for Persistent Robotic Scene Memory via World-Model-Based MCTS](https://mlsanity.com/paper/2604.11302/) (2026)
- [When Does a Video-Language Model Stop Watching? Reward Strength Controls the Formation and Reversal of Visual Shortcuts in Multimodal RLVR](https://mlsanity.com/paper/2606.22043/) (2026)
- [Visual Access Boundaries in Vision-Language Model Reasoning](https://mlsanity.com/paper/2607.12815/) (2026)
- [DiffSpot: Can VLMs Spot Fine-Grained Visual Differences in Web Interfaces?](https://mlsanity.com/paper/2605.29615/) (2026)
- [The Cost of Language: Centroid Erasure Exposes and Exploits Modal Competition in Multimodal Language Models](https://mlsanity.com/paper/2604.14363/) (2026)
- [PragMatch: Separating Pragmatic Incongruity from Cross-Modal Mismatch in Large Vision-Language Models](https://mlsanity.com/paper/2608.09772/) (2026)
- [Is Visual Prompting All You Need? Studying VLM Spatial Reasoning under Progressive Visual Scaffolds](https://mlsanity.com/paper/2608.21170/) (2026)
- [From Vision to Language: Investigating Causal Information Flow in Multimodal Decision-Making](https://mlsanity.com/paper/2609.05149/) (2026)
- [Where Should Language Sit in a Multimodal Model? Lessons from What Language Does to Human Perception and Cognition](https://mlsanity.com/paper/2609.07474/) (2026)
- [Mechanism-Level Evaluation for Vision-Language Models: Controlled Activation-Replacement Diagnosis of Gender Bias](https://mlsanity.com/paper/2609.16651/) (2026)
- [Using OCR Heads to Verbalize Image Semantics](https://mlsanity.com/paper/2609.18823/) (2026)
- [Certified Interface Aliases: Exact Collisions in Vision-Language Preprocessing, and When They Exist](https://mlsanity.com/paper/2609.33003/) (2026)
- [Semantic Capability Acquisition and Specialization During Vision-Language Model Fine-Tuning](https://mlsanity.com/paper/2610.07385/) (2026)

## Abstract (arXiv)

Vision-language models (VLMs) have achieved impressive performance across a wide range of multimodal tasks. However, they often fail on tasks that require fine-grained visual perception, even when the required information is still present in their internal representations. Prior work has attributed this ``hidden-in-plain-sight'' gap to the language model, but the cause remains unexplained. In this work, we demonstrate that this gap arises from the language model's lack of semantic labels for fine-grained visual details: when visual entities can be mapped to known concepts, VLMs bypass visual comparison and reason through language; when they cannot, VLMs resort to brittle and hallucinated descriptions. We verify this across semantic correspondence, synthetic shape matching, and face matching, and find that VLMs perform much better when the relevant entities are nameable than when they are unnamable. Mechanistically, Logit Lens analysis confirms that VLMs explicitly recover semantic labels for nameable entities and surface more unique tokens compared to unnameable entities. Furthermore, we show that this limitation can be addressed: teaching completely arbitrary names for unknown entities improves performance. More importantly, task-specific finetuning yields even stronger generalization without relying on language priors, i.e., through real visual perception. Our findings suggest that current VLM failures on visual tasks reflect a learned shortcut rather than a fundamental limitation of multimodal reasoning. Code and datasets are available at https://github.com/Patchwork53/VLMs-Need-Words-COLM2026.