mlsanityData to 14 Oct 2026

StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts

arXiv 2204.08292 ↗First posted 18 Apr 2022cs.CL cs.AIcited by 68

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Inferring spatial relations in natural language is a crucial ability an intelligent system should possess. The bAbI dataset tries to capture tasks relevant to this domain (task 17 and 19). However, these tasks have several limitations. Most importantly, they are limited to fixed expressions, they are limited in the number of reasoning steps required to solve them, and they fail to test the robustness of models to input that contains irrelevant or redundant information. In this paper, we present a new Question-Answering dataset called StepGame for robust multi-hop spatial reasoning in texts. Our experiments demonstrate that state-of-the-art models on the bAbI dataset struggle on the StepGame dataset. Moreover, we propose a Tensor-Product based Memory-Augmented Neural Network (TP-MANN) specialized for spatial reasoning tasks. Experimental results on both datasets show that our model outperforms all the baselines with superior generalization and robustness performance.

Key facts. Cited by 68 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 44 references that are in the map. First posted on arXiv 18 Apr 2022.

Builds on

24 of 44 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  4. 2017Graph Attention Networkscited by 8,923
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  6. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
  7. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  8. 2018Universal Transformerscited by 647
  9. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 410
  10. 2017A simple neural network module for relational reasoningcited by 346
  11. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 314
  12. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 239
12 more
  1. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  2. 2021SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoningcited by 77
  3. 2017Recurrent Relational Networkscited by 72
  4. 2019How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representationscited by 63
  5. 2020Learning Associative Inference Using Fast Weight Memorycited by 51
  6. 2020Self-Attentive Associative Memorycited by 33
  7. 2018Learning to Reason with Third-Order Tensor Productscited by 27
  8. 2017Representation Learning for Grounded Spatial Reasoningcited by 23
  9. 2017Learning Interpretable Spatial Operations in a Rich 3D Blocks Worldcited by 15
  10. 2020Robust and Interpretable Grounding of Spatial References with Relation Networkscited by 8
  11. 2016Gated End-to-End Memory Networkscited by 6
  12. 2017Source-Target Inference Models for Spatial Instruction Understandingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 14; 2024: 21; 2025: 10; 2026: 20.

68 papers.

Sort:

By area:

  1. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  2. 2022FOLIO: Natural Language Reasoning with First-Order Logiccited by 235
  3. 2025BIG-Bench Extra Hardcited by 125
  4. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 97
  5. 2024Stable LM 2 1.6B Technical Reportcited by 90
  6. 2023Evaluating Spatial Understanding of Large Language Modelscited by 66
  7. 2024Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Modelscited by 66
  8. 2023Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Textcited by 59
  9. 2024TopViewRS: Vision-Language Models as Top-View Spatial Reasonerscited by 49
  10. 2024Instruction Tuning With Loss Over Instructionscited by 40
  11. 2023DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningcited by 35
  12. 2022Transfer Learning with Synthetic Corpora for Spatial Role Labeling and Reasoningcited by 34
38 more
  1. 2024Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoningcited by 34
  2. 2024CityGPT: Empowering Urban Spatial Cognition of Large Language Modelscited by 30
  3. 2024Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmarkcited by 27
  4. 2023Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learnercited by 23
  5. 2023Eliminating Reasoning via Inferring with Planning: A New Framework to Guide LLMs' Non-linear Thinkingcited by 19
  6. 2024Quantifying the Capabilities of LLMs across Scale and Precisioncited by 19
  7. 2026Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration?cited by 19
  8. 2023Rethinking Semi-supervised Learning with Language Modelscited by 17
  9. 2024Reframing Spatial Reasoning Evaluation in Language Models: A Real-World Simulation Benchmark for Qualitative Reasoningcited by 17
  10. 2024Understanding the Language Model to Solve the Symbolic Multi-Step Reasoning Problem from the Perspective of Buffer Mechanismcited by 15
  11. 2024STBench: Assessing the Ability of Large Language Models in Spatio-Temporal Analysiscited by 11
  12. 2024Consecutive Batch Model Editing with HooK Layerscited by 10
  13. 2024Enhancing Logical Reasoning in Large Language Models through Graph-based Synthetic Datacited by 8
  14. 2023Teaching Probabilistic Logical Reasoning to Transformerscited by 7
  15. 2024GRASP: A Grid-Based Benchmark for Evaluating Commonsense Spatial Reasoningcited by 7
  16. 2025ORIGAMISPACE: Benchmarking Multimodal LLMs in Multi-Step Spatial Reasoning with Mathematical Constraintscited by 7
  17. 2024Dspy-based Neural-Symbolic Pipeline to Enhance Spatial Reasoning in LLMscited by 6
  18. 2024SpaRC and SpaRP: Spatial Reasoning Characterization and Path Generation for Understanding Spatial Reasoning Capability of Large Language Modelscited by 5
  19. 2025USTBench: Benchmarking and Dissecting Spatiotemporal Reasoning of LLMs as Urban Agentscited by 5
  20. 2024A LLM Benchmark based on the Minecraft Builder Dialog Agent Taskcited by 4
  21. 2025A Comparative Study of Neurosymbolic AI Approaches to Interpretable Logical Reasoningcited by 4
  22. 2026STAR: Mitigating Cascading Errors in Spatial Reasoning via Turn-point Alignment and Segment-level DPOcited by 4
  23. 2023Self Contrastive Learning for Session-based Recommendationcited by 3
  24. 2023DepWiGNN: A Depth-wise Graph Neural Network for Multi-hop Spatial Reasoning in Textcited by 3
  25. 2024Using Game Play to Investigate Multimodal and Conversational Grounding in Large Multimodal Modelscited by 3
  26. 2025An Empirical Study of Conformal Prediction in LLM with ASP Scaffolds for Robust Reasoningcited by 3
  27. 2025SCoRE: Benchmarking Long-Chain Reasoning in Commonsense Scenarioscited by 3
  28. 2023Rethink the Effectiveness of Text Data Augmentation: An Empirical Analysiscited by 2
  29. 2024Attention-based Iterative Decomposition for Tensor Product Representationcited by 2
  30. 2026SpatialText: A Pure-Text Cognitive Benchmark for Spatial Understanding in Large Language Modelscited by 2
  31. 2025Evaluating the Ability of Large Language Models to Reason about Cardinal Directions, Revisitedcited by 1
  32. 20253DFroMLLM: 3D Prototype Generation only from Pretrained Multimodal LLMscited by 1
  33. 2025FoR-SALE: Frame of Reference-guided Spatial Adjustment in LLM-based Diffusion Editingcited by 1
  34. 2025DecompSR: A dataset for decomposed analyses of compositional multihop spatial reasoningcited by 1
  35. 2026Mind the Gap Between Spatial Reasoning and Acting! Step-by-Step Evaluation of Agents With Spatial-Gymcited by 1
  36. 2026How Do LLMs and VLMs Understand Viewpoint Rotation Without Vision? An Interpretability Studycited by 1
  37. 2026Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalizationcited by 1
  38. 2026Spatial Reasoning via Modality Switching Between Language and Symbolic Representationscited by 1

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Neuro-symbolic Training for Reasoning over Spatial Language
  2. 2023Disentangling Extraction and Reasoning in Multi-hop Spatial Reasoning
  3. 2025Stuck in the Matrix: Probing Spatial Reasoning in Large Language Models
  4. 2026ChainSpace: A Chained-Reasoning Paradigm for Spatial Intelligence
  5. 2025Foundation Models for Geospatial Reasoning: Assessing Capabilities of Large Language Models in Understanding Geometries and Topological Spatial Relations
  6. 2023SpaceNLI: Evaluating the Consistency of Predicting Inferences in Space
  7. 2026Graph-Enhanced Large Language Models for Spatial Search
  8. 2026Mind the Spatial Gap: A Benchmark for Geospatial Reasoning and Constrained Spatial Editing