# SpatialSense: An Adversarially Crowdsourced Benchmark for Spatial Relation Recognition

arXiv: https://arxiv.org/abs/1908.02660 · HTML: https://mlsanity.com/paper/1908.02660/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Aug 2019.

## Built on by (most cited first, 30 of 55)

- [SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models](https://mlsanity.com/paper/2412.07755/) (2024)
- [Baichuan-Omni-1.5 Technical Report](https://mlsanity.com/paper/2501.15368/) (2025)
- [Scene Graph Generation: A Comprehensive Survey](https://mlsanity.com/paper/2201.00443/) (2022)
- [MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs](https://mlsanity.com/paper/2503.13111/) (2025)
- [Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models](https://mlsanity.com/paper/2409.09788/) (2024)
- [FineVision: Open Data Is All You Need](https://mlsanity.com/paper/2510.17269/) (2025)
- [Finding and Fixing Spurious Patterns with Explanations](https://mlsanity.com/paper/2106.02112/) (2021)
- [LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding](https://mlsanity.com/paper/2605.27365/) (2026)
- [SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models](https://mlsanity.com/paper/2509.17664/) (2025)
- [SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence](https://mlsanity.com/paper/2505.17012/) (2025)
- [Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation](https://mlsanity.com/paper/2506.21876/) (2025)
- [FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension](https://mlsanity.com/paper/2409.14750/) (2024)
- [Can Multimodal Large Language Models Understand Spatial Relations?](https://mlsanity.com/paper/2505.19015/) (2025)
- [T5Gemma 2: Seeing, Reading, and Understanding Longer](https://mlsanity.com/paper/2512.14856/) (2025)
- [Generating Visual Spatial Description via Holistic 3D Scene Understanding](https://mlsanity.com/paper/2305.11768/) (2023)
- [Evaluation and Enhancement of Semantic Grounding in Large Vision-Language Models](https://mlsanity.com/paper/2309.04041/) (2023)
- [FloorplanQA: A Benchmark for Spatial Reasoning in LLMs using Structured Representations](https://mlsanity.com/paper/2507.07644/) (2025)
- [Face-Human-Bench: A Comprehensive Benchmark of Face and Human Understanding for Multi-modal Assistants](https://mlsanity.com/paper/2501.01243/) (2025)
- [Can Transformers Capture Spatial Relations between Objects?](https://mlsanity.com/paper/2403.00729/) (2024)
- [What is Right for Me is Not Yet Right for You: A Dataset for Grounding Relative Directions via Multi-Task Learning](https://mlsanity.com/paper/2205.02671/) (2022)
- [AGRO: Adversarial Discovery of Error-prone groups for Robust Optimization](https://mlsanity.com/paper/2212.00921/) (2022)
- [Adaptive Visual Scene Understanding: Incremental Scene Graph Generation](https://mlsanity.com/paper/2310.01636/) (2023)
- [In-Context Learning May Not Elicit Trustworthy Reasoning: A-Not-B Errors in Pretrained Language Models](https://mlsanity.com/paper/2409.15454/) (2024)
- [LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sight](https://mlsanity.com/paper/2511.20648/) (2025)
- [2.5D Visual Relationship Detection](https://mlsanity.com/paper/2104.12727/) (2021)
- [Mitigating Hallucinations in Multimodal Spatial Relations through Constraint-Aware Prompting](https://mlsanity.com/paper/2502.08317/) (2025)
- [Visual Spatial Description: Controlled Spatial-Oriented Image-to-Text Generation](https://mlsanity.com/paper/2210.11109/) (2022)
- [VisionKG: Unleashing the Power of Visual Datasets via Knowledge Graph](https://mlsanity.com/paper/2309.13610/) (2023)
- [STUPD: A Synthetic Dataset for Spatial and Temporal Relation Reasoning](https://mlsanity.com/paper/2309.06680/) (2023)
- [Q-GroundCAM: Quantifying Grounding in Vision Language Models via GradCAM](https://mlsanity.com/paper/2404.19128/) (2024)

## Abstract (arXiv)

Understanding the spatial relations between objects in images is a surprisingly challenging task. A chair may be "behind" a person even if it appears to the left of the person in the image (depending on which way the person is facing). Two students that appear close to each other in the image may not in fact be "next to" each other if there is a third student between them. We introduce SpatialSense, a dataset specializing in spatial relation recognition which captures a broad spectrum of such challenges, allowing for proper benchmarking of computer vision techniques. SpatialSense is constructed through adversarial crowdsourcing, in which human annotators are tasked with finding spatial relations that are difficult to predict using simple cues such as 2D spatial configuration or language priors. Adversarial crowdsourcing significantly reduces dataset bias and samples more interesting relations in the long tail compared to existing datasets. On SpatialSense, state-of-the-art recognition models perform comparably to simple baselines, suggesting that they rely on straightforward cues instead of fully reasoning about this complex task. The SpatialSense benchmark provides a path forward to advancing the spatial reasoning capabilities of computer vision systems. The dataset and code are available at https://github.com/princeton-vl/SpatialSense.