mlsanityData to 7 Oct 2026

Visual Spatial Reasoning

arXiv 2205.00363 ↗First posted 30 Apr 2022cs.CL cs.AI cs.CVcited by 288

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Spatial relations are a basic part of human cognition. However, they are expressed in natural language in a variety of ways, and previous work has suggested that current vision-and-language models (VLMs) struggle to capture relational information. In this paper, we present Visual Spatial Reasoning (VSR), a dataset containing more than 10k natural text-image pairs with 66 types of spatial relations in English (such as: under, in front of, and facing). While using a seemingly simple annotation format, we show how the dataset includes challenging linguistic phenomena, such as varying reference frames. We demonstrate a large gap between human and model performance: the human ceiling is above 95%, while state-of-the-art models only achieve around 70%. We observe that VLMs' by-relation performances have little correlation with the number of training examples and the tested models are in general incapable of recognising relations concerning the orientations of objects.

Key facts. Cited by 288 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 56 references that are in the map. First posted on arXiv 30 Apr 2022.

Builds on

36 of 56 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
24 more
  1. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  2. 2016Modeling Context in Referring Expressionscited by 1,054
  3. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  4. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  5. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  6. 2015Neural Module Networkscited by 375
  7. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  8. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  9. 2019TVQA+: Spatio-Temporal Grounding for Video Question Answeringcited by 153
  10. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  11. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 119
  12. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  13. 2021SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoningcited by 75
  14. 2019CLEVR-Ref+: Diagnosing Visual Reasoning with Referring Expressionscited by 73
  15. 2020Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTscited by 71
  16. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
  17. 2022Things not Written in Text: Exploring Spatial Commonsense from Visual Signalscited by 41
  18. 2020Words aren't enough, their order matters: On the Robustness of Grounding Visual Referring Expressionscited by 33
  19. 2018Visual Referring Expression Recognition: What Do Systems Actually Learn?cited by 20
  20. 2017Acquiring Common Sense Spatial Knowledge through Implicit Spatial Templatescited by 10
  21. 2016Resolving Language and Vision Ambiguities Together: Joint Segmentation & Prepositional Attachment Resolution in Captioned Scenescited by 6
  22. 2018How clever is the FiLM model, and how clever can it be?cited by 2
  23. 2017Deep learning evaluation using deep linguistic processingcited by 1
  24. 2022Learning Functional Distributional Semantics with Visual Datacited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 40; 2024: 71; 2025: 92; 2026: 82.

288 papers.

Sort:

By area:

  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  4. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  5. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  6. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  7. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  8. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  9. 2024What matters when building vision-language models?cited by 321
  10. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  11. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  12. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
38 more
  1. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  2. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  3. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  4. 2023RSGPT: A Remote Sensing Vision Language Model and Benchmarkcited by 204
  5. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  6. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  7. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  8. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 178
  9. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  10. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
  11. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  12. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 170
  13. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  14. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 169
  15. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  16. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  17. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
  18. 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 130
  19. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  20. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 124
  21. 2025GRIT: Teaching MLLMs to Think with Imagescited by 121
  22. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Modelscited by 111
  23. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 111
  24. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  25. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  26. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 105
  27. 2024SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understandingcited by 101
  28. 2022Benchmarking Spatial Relationships in Text-to-Image Generationcited by 96
  29. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 94
  30. 2025Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areascited by 93
  31. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  32. 2025Visual Planning: Let's Think Only with Imagescited by 85
  33. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  34. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledgecited by 80
  35. 2025Scaling Spatial Intelligence with Multimodal Foundation Modelscited by 75
  36. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  37. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  38. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenescited by 62

50 most cited of 288 All 288 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models
  2. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoning
  3. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models
  4. 2025SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data
  5. 2024Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models
  6. 2025Vision language models are unreliable at trivial spatial cognition
  7. 2024I Know About "Up"! Enhancing Spatial Reasoning in Visual Language Models Through 3D Reconstruction
  8. 2026SpatiaLab: Can Vision-Language Models Perform Spatial Reasoning in the Wild?