mlsanityData to 7 Oct 2026

Can Vision-Language Models See Squares? Text-Recognition Mediates Spatial Reasoning Across Three Model Families

arXiv 2602.15950 ↗First posted 17 Feb 2026cs.CV cs.LGcited by 1

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

We present a simple experiment that exposes a fundamental limitation in vision-language models (VLMs): the inability to accurately localize filled cells in binary grids when those cells lack textual identity. We generate fifteen 15x15 grids with varying density (10.7%-41.8% filled cells) and render each as two image types -- text symbols (. and #) and filled squares without gridlines -- then ask three frontier VLMs (Claude Opus, ChatGPT 5.2, and Gemini 3 Thinking) to transcribe them. In the text-symbol condition, Claude and ChatGPT achieve approximately 91% cell accuracy and 84% F1, while Gemini achieves 84% accuracy and 63% F1. In the filled-squares condition, all three models collapse to 60-73% accuracy and 29-39% F1. Critically, all conditions pass through the same visual encoder -- the text symbols are images, not tokenized text. The text-vs-squares F1 gap ranges from 34 to 54 points across models, demonstrating that VLMs behave as if they possess a high-fidelity text-recognition pathway for spatial reasoning that dramatically outperforms their native visual pathway. Each model exhibits a distinct failure mode in the squares condition -- systematic under-counting (Claude), massive over-counting (ChatGPT), and template hallucination (Gemini) -- but all share the same underlying deficit: severely degraded spatial localization for non-textual visual elements.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 13 of its 13 references that are in the map. First posted on arXiv 17 Feb 2026.

Builds on

13 of 13 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMs5 Feb
  2. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
  3. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  4. 2022Visual Spatial Reasoning30 Apr

Other references · 9

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  3. 2023Teaching CLIP to Count to Tencited by 194
  4. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Modelscited by 96
  5. 2024Contrastive Localized Language-Image Pre-Trainingcited by 33
  6. 2024DocVLM: Make Your VLM an Efficient Readercited by 27
  7. 2025Your Vision-Language Model Can't Even Count to 20: Exposing the Failures of VLMs in Compositional Countingcited by 17
  8. 2024Can CLIP Count Stars? An Empirical Study on Quantity Bias in CLIPcited by 9
  9. 2025VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captionscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models
  2. 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models
  3. 2026WM-VLM: Probing Internal World Models for Interleaved Visual-Textual Reasoning
  4. 2026Spatial Reasoning is Not a Free Lunch: A Controlled Study on LLaVA
  5. 2025Vision language models are unreliable at trivial spatial cognition
  6. 2025Reading Images Like Texts: Sequential Image Understanding in Vision-Language Models
  7. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoning
  8. 2026Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models