# GSR-BENCH: A Benchmark for Grounded Spatial Reasoning Evaluation via Multimodal LLMs

arXiv: https://arxiv.org/abs/2406.13246 · HTML: https://mlsanity.com/paper/2406.13246/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 19 Jun 2024.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 16 of 16)

- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [Visual Spatial Reasoning](https://mlsanity.com/paper/2205.00363/) (2022)
- [Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions](https://mlsanity.com/paper/2308.11483/) (2023)
- [Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts](https://mlsanity.com/paper/2111.08276/) (2021)
- [Towards Grounded Visual Spatial Reasoning in Multi-Modal Vision Language Models](https://mlsanity.com/paper/2308.09778/) (2023)
- [Look at the Text: Instruction-Tuned Language Models are More Robust Multiple Choice Selectors than You Think](https://mlsanity.com/paper/2404.08382/) (2024)

## Built on by (most cited first, 30 of 33)

- [ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models](https://mlsanity.com/paper/2505.21500/) (2025)
- [OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models](https://mlsanity.com/paper/2506.03135/) (2025)
- [A Survey on Benchmarks of Multimodal Large Language Models](https://mlsanity.com/paper/2408.08632/) (2024)
- [SpatialCoT: Advancing Spatial Reasoning through Coordinate Alignment and Chain-of-Thought for Embodied Task Planning](https://mlsanity.com/paper/2501.10074/) (2025)
- [A Survey on Evaluation of Multimodal Large Language Models](https://mlsanity.com/paper/2408.15769/) (2024)
- [CAPTURe: Evaluating Spatial Reasoning in Vision Language Models via Occluded Object Counting](https://mlsanity.com/paper/2504.15485/) (2025)
- [SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models](https://mlsanity.com/paper/2505.00788/) (2025)
- [Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks](https://mlsanity.com/paper/2510.25760/) (2025)
- [A Survey on Multimodal Benchmarks: In the Era of Large AI Models](https://mlsanity.com/paper/2409.18142/) (2024)
- [iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMs](https://mlsanity.com/paper/2502.03214/) (2025)
- [When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMs](https://mlsanity.com/paper/2512.07580/) (2025)
- [Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosis](https://mlsanity.com/paper/2510.08759/) (2025)
- [GamiBench: Evaluating Spatial Reasoning and 2D-to-3D Planning Capabilities of MLLMs with Origami Folding Tasks](https://mlsanity.com/paper/2512.22207/) (2025)
- [CORDIAL: Can Multimodal Large Language Models Effectively Understand Coherence Relationships?](https://mlsanity.com/paper/2502.11300/) (2025)
- [MIRAGE: A Multi-modal Benchmark for Spatial Perception, Reasoning, and Intelligence](https://mlsanity.com/paper/2505.10604/) (2025)
- [Beyond Medical Diagnostics: How Medical Multimodal Large Language Models Think in Space](https://mlsanity.com/paper/2603.13800/) (2026)
- [SPACENUM: Revisiting Spatial Numerical Understanding in VLMs](https://mlsanity.com/paper/2605.23898/) (2026)
- [CoSpace: Benchmarking Continuous Space Perception Ability for Vision-Language Models](https://mlsanity.com/paper/2503.14161/) (2025)
- [SNS-Bench-VL: Benchmarking Multimodal Large Language Models in Social Networking Services](https://mlsanity.com/paper/2505.23065/) (2025)
- [MACEval: A Multi-Agent Continual Evaluation Network for Large Models](https://mlsanity.com/paper/2511.09139/) (2025)
- [Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?](https://mlsanity.com/paper/2605.30557/) (2026)
- [SATURN: Symbolic Spatial Reasoning for Multi-Perspective Grounding](https://mlsanity.com/paper/2606.22694/) (2026)
- [Can Large Language Models Unveil the Mysteries? An Exploration of Their Ability to Unlock Information in Complex Scenarios](https://mlsanity.com/paper/2502.19973/) (2025)
- [VFSI: Validity First Spatial Intelligence for Constraint-Guided Traffic Diffusion](https://mlsanity.com/paper/2509.23971/) (2025)
- [GenDexHand: Generative Simulation for Dexterous Hands](https://mlsanity.com/paper/2511.01791/) (2025)
- [From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Models](https://mlsanity.com/paper/2511.12861/) (2025)
- [From Pixels to BFS: High Maze Accuracy Does Not Imply Visual Planning](https://mlsanity.com/paper/2603.26839/) (2026)
- [Uncovering and Shaping the Latent Representation of 3D Scene Topology in Vision-Language Models](https://mlsanity.com/paper/2605.07148/) (2026)
- [Visual Credit Audit for Multimodal Spatial Reasoning](https://mlsanity.com/paper/2607.27069/) (2026)
- [SEER: A Self-Grounded Evidence Interface for Controlled Spatial Relation Classification](https://mlsanity.com/paper/2608.03631/) (2026)

## Abstract (arXiv)

The ability to understand and reason about spatial relationships between objects in images is an important component of visual reasoning. This skill rests on the ability to recognize and localize objects of interest and determine their spatial relation. Early vision and language models (VLMs) have been shown to struggle to recognize spatial relations. We extend the previously released What'sUp dataset and propose a novel comprehensive evaluation for spatial relationship understanding that highlights the strengths and weaknesses of 27 different models. In addition to the VLMs evaluated in What'sUp, our extensive evaluation encompasses 3 classes of Multimodal LLMs (MLLMs) that vary in their parameter sizes (ranging from 7B to 110B), training/instruction-tuning methods, and visual resolution to benchmark their performances and scrutinize the scaling laws in this task.