# StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts

arXiv: https://arxiv.org/abs/2204.08292 · HTML: https://mlsanity.com/paper/2204.08292/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 24 of its 44 references that are in the map.
- First posted on arXiv 18 Apr 2022.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [The Web as a Knowledge-base for Answering Complex Questions](https://mlsanity.com/paper/1803.06643/) (2018)
- [A simple neural network module for relational reasoning](https://mlsanity.com/paper/1706.01427/) (2017)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments](https://mlsanity.com/paper/1811.12354/) (2018)
- [Constructing Datasets for Multi-hop Reading Comprehension Across Documents](https://mlsanity.com/paper/1710.06481/) (2017)
- [SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoning](https://mlsanity.com/paper/2104.05832/) (2021)
- [Recurrent Relational Networks](https://mlsanity.com/paper/1711.08028/) (2017)
- [How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representations](https://mlsanity.com/paper/1909.04925/) (2019)
- [Learning Associative Inference Using Fast Weight Memory](https://mlsanity.com/paper/2011.07831/) (2020)
- [Self-Attentive Associative Memory](https://mlsanity.com/paper/2002.03519/) (2020)
- [Learning to Reason with Third-Order Tensor Products](https://mlsanity.com/paper/1811.12143/) (2018)
- [Representation Learning for Grounded Spatial Reasoning](https://mlsanity.com/paper/1707.03938/) (2017)
- [Learning Interpretable Spatial Operations in a Rich 3D Blocks World](https://mlsanity.com/paper/1712.03463/) (2017)
- [Robust and Interpretable Grounding of Spatial References with Relation Networks](https://mlsanity.com/paper/2005.00696/) (2020)
- [Gated End-to-End Memory Networks](https://mlsanity.com/paper/1610.04211/) (2016)
- [Source-Target Inference Models for Spatial Instruction Understanding](https://mlsanity.com/paper/1707.03804/) (2017)

## Built on by (most cited first, 30 of 68)

- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [FOLIO: Natural Language Reasoning with First-Order Logic](https://mlsanity.com/paper/2209.00840/) (2022)
- [BIG-Bench Extra Hard](https://mlsanity.com/paper/2502.19187/) (2025)
- [Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluations](https://mlsanity.com/paper/2306.04618/) (2023)
- [Stable LM 2 1.6B Technical Report](https://mlsanity.com/paper/2402.17834/) (2024)
- [Evaluating Spatial Understanding of Large Language Models](https://mlsanity.com/paper/2310.14540/) (2023)
- [Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models](https://mlsanity.com/paper/2404.03622/) (2024)
- [Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Text](https://mlsanity.com/paper/2307.07696/) (2023)
- [TopViewRS: Vision-Language Models as Top-View Spatial Reasoners](https://mlsanity.com/paper/2406.02537/) (2024)
- [Instruction Tuning With Loss Over Instructions](https://mlsanity.com/paper/2405.14394/) (2024)
- [DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning](https://mlsanity.com/paper/2309.05173/) (2023)
- [Transfer Learning with Synthetic Corpora for Spatial Role Labeling and Reasoning](https://mlsanity.com/paper/2210.16952/) (2022)
- [Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoning](https://mlsanity.com/paper/2410.16162/) (2024)
- [CityGPT: Empowering Urban Spatial Cognition of Large Language Models](https://mlsanity.com/paper/2406.13948/) (2024)
- [Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmark](https://mlsanity.com/paper/2401.03991/) (2024)
- [Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner](https://mlsanity.com/paper/2305.01711/) (2023)
- [Eliminating Reasoning via Inferring with Planning: A New Framework to Guide LLMs' Non-linear Thinking](https://mlsanity.com/paper/2310.12342/) (2023)
- [Quantifying the Capabilities of LLMs across Scale and Precision](https://mlsanity.com/paper/2405.03146/) (2024)
- [Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration?](https://mlsanity.com/paper/2602.07055/) (2026)
- [Rethinking Semi-supervised Learning with Language Models](https://mlsanity.com/paper/2305.13002/) (2023)
- [Reframing Spatial Reasoning Evaluation in Language Models: A Real-World Simulation Benchmark for Qualitative Reasoning](https://mlsanity.com/paper/2405.15064/) (2024)
- [Understanding the Language Model to Solve the Symbolic Multi-Step Reasoning Problem from the Perspective of Buffer Mechanism](https://mlsanity.com/paper/2405.15302/) (2024)
- [STBench: Assessing the Ability of Large Language Models in Spatio-Temporal Analysis](https://mlsanity.com/paper/2406.19065/) (2024)
- [Consecutive Batch Model Editing with HooK Layers](https://mlsanity.com/paper/2403.05330/) (2024)
- [Enhancing Logical Reasoning in Large Language Models through Graph-based Synthetic Data](https://mlsanity.com/paper/2409.12437/) (2024)
- [Teaching Probabilistic Logical Reasoning to Transformers](https://mlsanity.com/paper/2305.13179/) (2023)
- [GRASP: A Grid-Based Benchmark for Evaluating Commonsense Spatial Reasoning](https://mlsanity.com/paper/2407.01892/) (2024)
- [ORIGAMISPACE: Benchmarking Multimodal LLMs in Multi-Step Spatial Reasoning with Mathematical Constraints](https://mlsanity.com/paper/2511.18450/) (2025)
- [Dspy-based Neural-Symbolic Pipeline to Enhance Spatial Reasoning in LLMs](https://mlsanity.com/paper/2411.18564/) (2024)
- [SpaRC and SpaRP: Spatial Reasoning Characterization and Path Generation for Understanding Spatial Reasoning Capability of Large Language Models](https://mlsanity.com/paper/2406.04566/) (2024)

## Abstract (arXiv)

Inferring spatial relations in natural language is a crucial ability an intelligent system should possess. The bAbI dataset tries to capture tasks relevant to this domain (task 17 and 19). However, these tasks have several limitations. Most importantly, they are limited to fixed expressions, they are limited in the number of reasoning steps required to solve them, and they fail to test the robustness of models to input that contains irrelevant or redundant information. In this paper, we present a new Question-Answering dataset called StepGame for robust multi-hop spatial reasoning in texts. Our experiments demonstrate that state-of-the-art models on the bAbI dataset struggle on the StepGame dataset. Moreover, we propose a Tensor-Product based Memory-Augmented Neural Network (TP-MANN) specialized for spatial reasoning tasks. Experimental results on both datasets show that our model outperforms all the baselines with superior generalization and robustness performance.