mlsanityData to 7 Oct 2026

SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoning

arXiv 2104.05832 ↗First posted 12 Apr 2021cs.CL cs.AIcited by 75

Not on the map. Why?

Cited by 41 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper proposes a question-answering (QA) benchmark for spatial reasoning on natural language text which contains more realistic spatial phenomena not covered by prior work and is challenging for state-of-the-art language models (LM). We propose a distant supervision method to improve on this task. Specifically, we design grammar and reasoning rules to automatically generate a spatial description of visual scenes and corresponding QA pairs. Experiments show that further pretraining LMs on these automatically generated data significantly improves LMs' capability on spatial understanding, which in turn helps to better solve two external datasets, bAbI, and boolQ. We hope that this work can foster investigations into more sophisticated models for spatial reasoning over text.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Builds on 33 of its 56 references that are in the map. First posted on arXiv 12 Apr 2021.

Builds on

33 of 56 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  10. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  11. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  12. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
21 more
  1. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  2. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  3. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  4. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  5. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  6. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  7. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  8. 2017Learning to Ask: Neural Question Generation for Reading Comprehensioncited by 186
  9. 2020Event Extraction by Answering (Almost) Natural Questionscited by 177
  10. 2019Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoningcited by 100
  11. 2020TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questionscited by 65
  12. 2020What do Models Learn from Question Answering Datasets?cited by 45
  13. 2020Temporal Common Sense Acquisition with Minimal Supervisioncited by 38
  14. 2020RMM: A Recursive Mental Model for Dialog Navigationcited by 32
  15. 2019Question Answering is a Format; When is it Useful?cited by 25
  16. 2017Crowdsourcing Question-Answer Meaning Representationscited by 21
  17. 2019GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Levelcited by 17
  18. 2020ArraMon: A Joint Navigation-Assembly Instruction Interpretation Task in Dynamic Environmentscited by 14
  19. 2020Syn-QG: Syntactic and Shallow Semantic Rules for Question Generationcited by 12
  20. 2020A Linguistic Analysis of Visually Grounded Dialogues Based on Spatial Expressionscited by 6
  21. 2020From Spatial Relations to Spatial Configurationscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 8; 2023: 12; 2024: 19; 2025: 18; 2026: 14.

75 papers.

Sort:

By area:

  1. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  2. 2022Visual Spatial Reasoningcited by 288
  3. 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 130
  4. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  5. 2025BIG-Bench Extra Hardcited by 123
  6. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Modelscited by 111
  7. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  8. 2024Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluationcited by 73
  9. 2022StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Textscited by 65
  10. 2023Evaluating Spatial Understanding of Large Language Modelscited by 65
  11. 2024Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Modelscited by 65
  12. 2024Instruction Pre-Training: Language Models are Supervised Multitask Learnerscited by 63
38 more
  1. 2024Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguitiescited by 58
  2. 2024TopViewRS: Vision-Language Models as Top-View Spatial Reasonerscited by 48
  3. 2022Things not Written in Text: Exploring Spatial Commonsense from Visual Signalscited by 41
  4. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Modelscited by 35
  5. 2024Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoningcited by 33
  6. 2022Transfer Learning with Synthetic Corpora for Spatial Role Labeling and Reasoningcited by 32
  7. 2024Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmarkcited by 26
  8. 2025iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMscited by 24
  9. 2025Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluationcited by 23
  10. 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipescited by 22
  11. 2021Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understandingcited by 19
  12. 2023Eliminating Reasoning via Inferring with Planning: A New Framework to Guide LLMs' Non-linear Thinkingcited by 19
  13. 2026Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration?cited by 18
  14. 2024Quantifying the Capabilities of LLMs across Scale and Precisioncited by 17
  15. 2024Space3D-Bench: Spatial 3D Question Answering Benchmarkcited by 17
  16. 2024Reframing Spatial Reasoning Evaluation in Language Models: A Real-World Simulation Benchmark for Qualitative Reasoningcited by 16
  17. 2024A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practicescited by 15
  18. 2025SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewardscited by 14
  19. 2021DomiKnowS: A Library for Integration of Symbolic Domain Knowledge in Deep Learningcited by 12
  20. 2024STBench: Assessing the Ability of Large Language Models in Spatio-Temporal Analysiscited by 11
  21. 2023GLUECons: A Generic Benchmark for Learning Under Constraintscited by 9
  22. 2025HoT: Highlighted Chain of Thought for Referencing Supporting Facts from Inputscited by 9
  23. 2024Enhancing Logical Reasoning in Large Language Models through Graph-based Synthetic Datacited by 8
  24. 2022PromptDA: Label-guided Data Augmentation for Prompt-based Few-shot Learnerscited by 7
  25. 2023Are LLMs the Master of All Trades? : Exploring Domain-Agnostic Reasoning Skills of LLMscited by 7
  26. 2024GRASP: A Grid-Based Benchmark for Evaluating Commonsense Spatial Reasoningcited by 7
  27. 2026VoxelCodeBench: Benchmarking 3D World Modeling Through Code Generationcited by 7
  28. 2022ShortcutLens: A Visual Analytics Approach for Exploring Shortcuts in Natural Language Understanding Datasetcited by 6
  29. 2023Chain-of-Symbol Prompting Elicits Planning in Large Langauge Modelscited by 6
  30. 2023Statistical Depth for Ranking and Characterizing Transformer-Based Text Embeddingscited by 5
  31. 2023What's left can't be right -- The remaining positional incompetence of contrastive vision-language modelscited by 5
  32. 2024SpaRC and SpaRP: Spatial Reasoning Characterization and Path Generation for Understanding Spatial Reasoning Capability of Large Language Modelscited by 5
  33. 2025USTBench: Benchmarking and Dissecting Spatiotemporal Reasoning of LLMs as Urban Agentscited by 5
  34. 2025IMoRe: Implicit Program-Guided Reasoning for Human Motion Q&Acited by 5
  35. 2021TraVLR: Now You See It, Now You Don't! A Bimodal Dataset for Evaluating Visio-Linguistic Reasoningcited by 4
  36. 2023Unifying Structure Reasoning and Language Model Pre-training for Complex Reasoningcited by 4
  37. 2023SpaceNLI: Evaluating the Consistency of Predicting Inferences in Spacecited by 4
  38. 2022Generalizable Neuro-symbolic Systems for Commonsense Question Answeringcited by 3

50 most cited of 75 All 75 (JSON)