mlsanityData to 7 Oct 2026

What's "up" with vision-language models? Investigating their struggle with spatial reasoning

arXiv 2310.19785 ↗First posted 30 Oct 2023cs.CL cs.CV cs.LGcited by 245

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Recent vision-language (VL) models are powerful, but can they reliably distinguish "right" from "left"? We curate three new corpora to quantify model comprehension of such basic spatial relations. These tests isolate spatial reasoning more precisely than existing datasets like VQAv2, e.g., our What'sUp benchmark contains sets of photographs varying only the spatial relations of objects, keeping their identity fixed (see Figure 1: models must comprehend not only the usual case of a dog under a table, but also, the same dog on top of the same table). We evaluate 18 VL models, finding that all perform poorly, e.g., BLIP finetuned on VQAv2, which nears human parity on VQAv2, achieves 56% accuracy on our benchmarks vs. humans at 99%. We conclude by studying causes of this surprising behavior, finding: 1) that popular vision-language pretraining corpora like LAION-2B contain little reliable data for learning spatial relationships; and 2) that basic modeling interventions like up-weighting preposition-containing instances or fine-tuning on our corpora are not sufficient to address the challenges our benchmarks pose. We are hopeful that these corpora will facilitate further research, and we release our data and code at https://github.com/amitakamath/whatsup_vlms.

Key facts. Cited by 245 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 34 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

24 of 34 references.

References · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  10. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  11. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  12. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
12 more
  1. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  2. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  3. 2018nocaps: novel object captioning at scalecited by 444
  4. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  5. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  6. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  7. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  8. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  9. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 113
  10. 2022Benchmarking Spatial Relationships in Text-to-Image Generationcited by 96
  11. 2017ShapeWorld - A new test methodology for multimodal language understandingcited by 37
  12. 2020Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting taskscited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 57; 2025: 100; 2026: 87.

245 papers.

Sort:

By area:

  1. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
  2. 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmarkcited by 133
  3. 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 130
  4. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Modelscited by 111
  5. 2025ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Modelscited by 110
  6. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 105
  7. 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Modelscited by 100
  8. 2025Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areascited by 93
  9. 2025SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Modelscited by 83
  10. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  11. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  12. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenescited by 62
38 more
  1. 2024Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguitiescited by 58
  2. 2024Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Modelscited by 54
  3. 2024An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Modelscited by 49
  4. 2025MindJourney: Test-Time Scaling with World Models for Spatial Reasoningcited by 49
  5. 2024Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMscited by 48
  6. 2024TopViewRS: Vision-Language Models as Top-View Spatial Reasonerscited by 48
  7. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  8. 2025SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoningcited by 45
  9. 2024VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Modelscited by 42
  10. 2025Visual Agentic AI for Spatial Reasoning with a Dynamic APIcited by 42
  11. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  12. 2025Visual Representation Alignment for Multimodal Large Language Modelscited by 40
  13. 2025Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learningcited by 38
  14. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 36
  15. 2025Beyond Semantics: Rediscovering Spatial Awareness in Vision-Language Modelscited by 35
  16. 2024Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoningcited by 33
  17. 2024Learning Action and Reasoning-Centric Image Editing from Videos and Simulationscited by 32
  18. 2024SPHERE: Unveiling Spatial Blind Spots in Vision-Language Models Through Hierarchical Evaluationcited by 31
  19. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 31
  20. 2025CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modallycited by 30
  21. 2025A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairscited by 30
  22. 2025Why Do MLLMs Struggle with Spatial Understanding? A Systematic Analysis from Data to Architecturecited by 30
  23. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  24. 2024MouSi: Poly-Visual-Expert Vision-Language Modelscited by 24
  25. 2025iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMscited by 24
  26. 2025SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Modelscited by 24
  27. 2024VisMin: Visual Minimal-Change Understandingcited by 23
  28. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  29. 2024KiVA: Kid-inspired Visual Analogies for Testing Large Multimodal Modelscited by 22
  30. 2024MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation modelscited by 22
  31. 2025SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligencecited by 22
  32. 2023GlitchBench: Can large multimodal models detect video game glitches?cited by 21
  33. 2024The Hard Positive Truth about Vision-Language Compositionalitycited by 21
  34. 2026Thinking with Blueprints: Assisting Vision-Language Models in Spatial Reasoning via Structured Object Representationcited by 21
  35. 2024Leveraging Hallucinations to Reduce Manual Prompt Dependency in Promptable Segmentationcited by 20
  36. 2025Is CLIP ideal? No. Can we fix it? Yes!cited by 20
  37. 2025Emergence and Evolution of Interpretable Concepts in Diffusion Modelscited by 20
  38. 2025SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligencecited by 19

50 most cited of 245 All 245 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data
  2. 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models
  3. 2022Visual Spatial Reasoning
  4. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models
  5. 2023Towards Grounded Visual Spatial Reasoning in Multi-Modal Vision Language Models
  6. 2025Exploring Spatial Language Grounding Through Referring Expressions
  7. 2026Consistent Yet Wrong: Evidence Insensitivity in Spatial Vision-Language Models
  8. 2026The Spatial Blindspot of Vision-Language Models