# Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?

arXiv: https://arxiv.org/abs/2605.22903 · HTML: https://mlsanity.com/paper/2605.22903/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 21 May 2026.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge](https://mlsanity.com/paper/2206.01718/) (2022)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding](https://mlsanity.com/paper/2403.00425/) (2024)
- [A Survey on Evaluation of Multimodal Large Language Models](https://mlsanity.com/paper/2408.15769/) (2024)
- [VLind-Bench: Measuring Language Priors in Large Vision-Language Models](https://mlsanity.com/paper/2406.08702/) (2024)
- [THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language Models](https://mlsanity.com/paper/2405.05256/) (2024)
- [Do Vision-Language Models Really Understand Visual Language?](https://mlsanity.com/paper/2410.00193/) (2024)
- [Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decoding](https://mlsanity.com/paper/2412.06474/) (2024)
- [HALP: Detecting Hallucinations in Vision-Language Models without Generating a Single Token](https://mlsanity.com/paper/2603.05465/) (2026)
- [Does Object Grounding Really Reduce Hallucination of Large Vision-Language Models?](https://mlsanity.com/paper/2406.14492/) (2024)
- [From Behavioral Performance to Internal Competence: Interpreting Vision-Language Models with VLM-Lens](https://mlsanity.com/paper/2510.02292/) (2025)
- [From Head to Tail: Towards Balanced Representation in Large Vision-Language Models through Adaptive Data Calibration](https://mlsanity.com/paper/2503.12821/) (2025)
- [On the Predictive Power of Representation Dispersion in Language Models](https://mlsanity.com/paper/2506.24106/) (2025)
- [H-POPE: Hierarchical Polling-based Probing Evaluation of Hallucinations in Large Vision-Language Models](https://mlsanity.com/paper/2411.04077/) (2024)
- [On Statistical Efficiency in Learning](https://mlsanity.com/paper/2012.13307/) (2020)
- [A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2505.01958/) (2025)
- [ISO-Bench: Benchmarking Multimodal Causal Reasoning in Visual-Language Models through Procedural Plans](https://mlsanity.com/paper/2507.23135/) (2025)

## Built on by (most cited first, 6 of 6)

- [Attention Without Grounding: Causal Evaluation of Visual Explanations in Medical VLMs](https://mlsanity.com/paper/2607.18577/) (2026)
- [Visual Credit Audit for Multimodal Spatial Reasoning](https://mlsanity.com/paper/2607.27069/) (2026)
- [Are We There Yet? Assessing Computer-Use Agents for Blind Users' Accessible Interaction with Desktop Applications](https://mlsanity.com/paper/2609.00524/) (2026)
- [One Perturbation Is Not Enough: Identifiability and Blind Baselines for Behavioral AI Evaluation](https://mlsanity.com/paper/2609.06190/) (2026)
- [What Looks Like a Capability Limit in Vision-Language Models Is a Readout Limit](https://mlsanity.com/paper/2609.27408/) (2026)
- [ConvCue: Complementary Visual Inductive Biases for Vision-Language Models](https://mlsanity.com/paper/2609.34196/) (2026)

## Abstract (arXiv)

Benchmark accuracy is often implicitly assumed to reflect grounded visual understanding in vision-language models (VLMs), yet it remains unclear to what extent such scores truly reflect reliance on visual evidence. Motivated by a surprising observation that removing a substantial fraction of image tokens only degrades model performance very slightly on a widely used hallucination benchmark, we systematically investigate this mismatch in a set of open-source VLMs. Our analysis spans multiple levels of granularity, spanning global visual degradation, localized occlusion, question reformulation, answer-space expansion, and decision-level analyses beyond standard accuracy. We further complement these behavioral results with a layer-wise analysis of vision-token geometry. Throughout the experiments, we find that although VLMs do incorporate visual input, their predictions are less sensitive to the loss of fine-grained visual evidence that standard accuracy should have suggested. Even when the final prediction remains unchanged, the model's internal support for the correct answer may already be weakened. We further complement a representation-level analysis, which shows increasing similarity among visual tokens in deeper layers, providing a possible explanation for our findings. Together, these results suggest that current benchmarks are not sufficient to reliably evaluate fine-grained visual grounding in VLMs.