mlsanityData to 7 Oct 2026

GSR-BENCH: A Benchmark for Grounded Spatial Reasoning Evaluation via Multimodal LLMs

arXiv 2406.13246 ↗First posted 19 Jun 2024cs.CL cs.CV cs.LGcited by 33

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

The ability to understand and reason about spatial relationships between objects in images is an important component of visual reasoning. This skill rests on the ability to recognize and localize objects of interest and determine their spatial relation. Early vision and language models (VLMs) have been shown to struggle to recognize spatial relations. We extend the previously released What'sUp dataset and propose a novel comprehensive evaluation for spatial relationship understanding that highlights the strengths and weaknesses of 27 different models. In addition to the VLMs evaluated in What'sUp, our extensive evaluation encompasses 3 classes of Multimodal LLMs (MLLMs) that vary in their parameter sizes (ranging from 7B to 110B), training/instruction-tuning methods, and visual resolution to benchmark their performances and scrutinize the scaling laws in this task.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 16 of its 32 references that are in the map. First posted on arXiv 19 Jun 2024.

Builds on

16 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Towards Grounded Visual Spatial Reasoning in Multi-Modal Vision Language Models18 Aug
  2. 2022Visual Spatial Reasoning30 Apr

Other references · 14

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  7. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  8. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  10. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
  11. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  12. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
2 more
  1. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  2. 2024Look at the Text: Instruction-Tuned Language Models are More Robust Multiple Choice Selectors than You Thinkcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 19; 2026: 11.

33 papers.

Sort:

By area:

  1. 2025ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Modelscited by 110
  2. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 105
  3. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  4. 2025SpatialCoT: Advancing Spatial Reasoning through Coordinate Alignment and Chain-of-Thought for Embodied Task Planningcited by 62
  5. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  6. 2025SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Modelscited by 37
  7. 2025CAPTURe: Evaluating Spatial Reasoning in Vision Language Models via Occluded Object Countingcited by 36
  8. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 31
  9. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  10. 2025iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMscited by 24
  11. 2025When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMscited by 11
  12. 2025Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosiscited by 8
21 more
  1. 2025GamiBench: Evaluating Spatial Reasoning and 2D-to-3D Planning Capabilities of MLLMs with Origami Folding Taskscited by 4
  2. 2025CORDIAL: Can Multimodal Large Language Models Effectively Understand Coherence Relationships?cited by 3
  3. 2025MIRAGE: A Multi-modal Benchmark for Spatial Perception, Reasoning, and Intelligencecited by 3
  4. 2026Beyond Medical Diagnostics: How Medical Multimodal Large Language Models Think in Spacecited by 3
  5. 2025SNS-Bench-VL: Benchmarking Multimodal Large Language Models in Social Networking Servicescited by 2
  6. 2025MACEval: A Multi-Agent Continual Evaluation Network for Large Modelscited by 2
  7. 2026Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?cited by 2
  8. 2026SATURN: Symbolic Spatial Reasoning for Multi-Perspective Groundingcited by 2
  9. 2025Can Large Language Models Unveil the Mysteries? An Exploration of Their Ability to Unlock Information in Complex Scenarioscited by 1
  10. 2025CoSpace: Benchmarking Continuous Space Perception Ability for Vision-Language Modelscited by 1
  11. 2025VFSI: Validity First Spatial Intelligence for Constraint-Guided Traffic Diffusioncited by 1
  12. 2025GenDexHand: Generative Simulation for Dexterous Handscited by 1
  13. 2025From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Modelscited by 1
  14. 2026From Pixels to BFS: High Maze Accuracy Does Not Imply Visual Planningcited by 1
  15. 2026Uncovering and Shaping the Latent Representation of 3D Scene Topology in Vision-Language Modelscited by 1
  16. 2026SPACENUM: Revisiting Spatial Numerical Understanding in VLMscited by 1
  17. 2026Visual Credit Audit for Multimodal Spatial Reasoningcited by 1
  18. 2026SEER: A Self-Grounded Evidence Interface for Controlled Spatial Relation Classificationcited by 1
  19. 2026Left-Right Symmetry Breaking in CLIP-style Vision-Language Models Trained on Synthetic Spatial-Relation Datacited by 0
  20. 2026On the Strengths and Weaknesses of Data for Open-set Embodied Assistancecited by 0
  21. 2026OrientSAM: Mitigating Camera-Centric Shortcut in Multimodal Spatial Reasoning via Orientation-Aware Spatial Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models
  2. 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models
  3. 2024Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models
  4. 2026Trace, Verify, and Correct: A Training-Free Framework for Spatial Reasoning in Multimodal LLMs
  5. 2026Can LLMs See Without Pixels? Benchmarking Spatial Intelligence from Textual Descriptions
  6. 2024An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Models
  7. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence
  8. 2025Understanding Space Is Rocket Science -- Only Top Reasoning Models Can Solve Spatial Understanding Tasks