# SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoning

arXiv: https://arxiv.org/abs/2104.05832 · HTML: https://mlsanity.com/paper/2104.05832/

## Key facts

- Cited by 77 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 56 references that are in the map.
- First posted on arXiv 12 Apr 2021.

## Builds on (most cited first, 30 of 33)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [A Corpus for Reasoning About Natural Language Grounded in Photographs](https://mlsanity.com/paper/1811.00491/) (2018)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments](https://mlsanity.com/paper/1811.12354/) (2018)
- [Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks](https://mlsanity.com/paper/1811.01088/) (2018)
- [Learning to Ask: Neural Question Generation for Reading Comprehension](https://mlsanity.com/paper/1705.00106/) (2017)
- [Event Extraction by Answering (Almost) Natural Questions](https://mlsanity.com/paper/2004.13625/) (2020)
- [Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning](https://mlsanity.com/paper/1908.05803/) (2019)
- [TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions](https://mlsanity.com/paper/2005.00242/) (2020)
- [What do Models Learn from Question Answering Datasets?](https://mlsanity.com/paper/2004.03490/) (2020)
- [Temporal Common Sense Acquisition with Minimal Supervision](https://mlsanity.com/paper/2005.04304/) (2020)
- [RMM: A Recursive Mental Model for Dialog Navigation](https://mlsanity.com/paper/2005.00728/) (2020)
- [Question Answering is a Format; When is it Useful?](https://mlsanity.com/paper/1909.11291/) (2019)
- [Crowdsourcing Question-Answer Meaning Representations](https://mlsanity.com/paper/1711.05885/) (2017)
- [GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level](https://mlsanity.com/paper/1908.07855/) (2019)
- [ArraMon: A Joint Navigation-Assembly Instruction Interpretation Task in Dynamic Environments](https://mlsanity.com/paper/2011.07660/) (2020)

## Built on by (most cited first, 30 of 77)

- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Visual Spatial Reasoning](https://mlsanity.com/paper/2205.00363/) (2022)
- [Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing](https://mlsanity.com/paper/2506.09965/) (2025)
- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [BIG-Bench Extra Hard](https://mlsanity.com/paper/2502.19187/) (2025)
- [Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models](https://mlsanity.com/paper/2406.14852/) (2024)
- [QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension](https://mlsanity.com/paper/2107.12708/) (2021)
- [Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluation](https://mlsanity.com/paper/2402.11443/) (2024)
- [StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts](https://mlsanity.com/paper/2204.08292/) (2022)
- [Evaluating Spatial Understanding of Large Language Models](https://mlsanity.com/paper/2310.14540/) (2023)
- [Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models](https://mlsanity.com/paper/2404.03622/) (2024)
- [Instruction Pre-Training: Language Models are Supervised Multitask Learners](https://mlsanity.com/paper/2406.14491/) (2024)
- [Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguities](https://mlsanity.com/paper/2410.17385/) (2024)
- [TopViewRS: Vision-Language Models as Top-View Spatial Reasoners](https://mlsanity.com/paper/2406.02537/) (2024)
- [Things not Written in Text: Exploring Spatial Commonsense from Visual Signals](https://mlsanity.com/paper/2203.08075/) (2022)
- [Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models](https://mlsanity.com/paper/2310.00836/) (2023)
- [Transfer Learning with Synthetic Corpora for Spatial Role Labeling and Reasoning](https://mlsanity.com/paper/2210.16952/) (2022)
- [Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoning](https://mlsanity.com/paper/2410.16162/) (2024)
- [Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmark](https://mlsanity.com/paper/2401.03991/) (2024)
- [iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMs](https://mlsanity.com/paper/2502.03214/) (2025)
- [Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation](https://mlsanity.com/paper/2506.21876/) (2025)
- [Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipes](https://mlsanity.com/paper/2504.15037/) (2025)
- [Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding](https://mlsanity.com/paper/2109.04947/) (2021)
- [Eliminating Reasoning via Inferring with Planning: A New Framework to Guide LLMs' Non-linear Thinking](https://mlsanity.com/paper/2310.12342/) (2023)
- [Quantifying the Capabilities of LLMs across Scale and Precision](https://mlsanity.com/paper/2405.03146/) (2024)
- [Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration?](https://mlsanity.com/paper/2602.07055/) (2026)
- [Reframing Spatial Reasoning Evaluation in Language Models: A Real-World Simulation Benchmark for Qualitative Reasoning](https://mlsanity.com/paper/2405.15064/) (2024)
- [Space3D-Bench: Spatial 3D Question Answering Benchmark](https://mlsanity.com/paper/2408.16662/) (2024)
- [SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewards](https://mlsanity.com/paper/2511.07403/) (2025)
- [A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practices](https://mlsanity.com/paper/2406.08787/) (2024)

## Abstract (arXiv)

This paper proposes a question-answering (QA) benchmark for spatial reasoning on natural language text which contains more realistic spatial phenomena not covered by prior work and is challenging for state-of-the-art language models (LM). We propose a distant supervision method to improve on this task. Specifically, we design grammar and reasoning rules to automatically generate a spatial description of visual scenes and corresponding QA pairs. Experiments show that further pretraining LMs on these automatically generated data significantly improves LMs' capability on spatial understanding, which in turn helps to better solve two external datasets, bAbI, and boolQ. We hope that this work can foster investigations into more sophisticated models for spatial reasoning over text.