# Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmark

arXiv: https://arxiv.org/abs/2401.03991 · HTML: https://mlsanity.com/paper/2401.03991/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 9 of its 24 references that are in the map.
- First posted on arXiv 8 Jan 2024.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 9 of 9)

- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning](https://mlsanity.com/paper/2205.09712/) (2022)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models](https://mlsanity.com/paper/2303.10420/) (2023)
- [SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoning](https://mlsanity.com/paper/2104.05832/) (2021)
- [StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts](https://mlsanity.com/paper/2204.08292/) (2022)
- [Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Text](https://mlsanity.com/paper/2307.07696/) (2023)
- [Transfer Learning with Synthetic Corpora for Spatial Role Labeling and Reasoning](https://mlsanity.com/paper/2210.16952/) (2022)
- [Dialectical language model evaluation: An initial appraisal of the commonsense spatial reasoning abilities of LLMs](https://mlsanity.com/paper/2304.11164/) (2023)

## Built on by (most cited first, 27 of 27)

- [How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective](https://mlsanity.com/paper/2509.18905/) (2025)
- [Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoning](https://mlsanity.com/paper/2410.16162/) (2024)
- [Reframing Spatial Reasoning Evaluation in Language Models: A Real-World Simulation Benchmark for Qualitative Reasoning](https://mlsanity.com/paper/2405.15064/) (2024)
- [Spatial-RAG: Spatial Retrieval Augmented Generation for Real-World Geospatial Reasoning Questions](https://mlsanity.com/paper/2502.18470/) (2025)
- [A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Science](https://mlsanity.com/paper/2504.09848/) (2025)
- [STBench: Assessing the Ability of Large Language Models in Spatio-Temporal Analysis](https://mlsanity.com/paper/2406.19065/) (2024)
- [MazeEval: A Benchmark for Testing Sequential Decision-Making in Language Models](https://mlsanity.com/paper/2507.20395/) (2025)
- [How language models extrapolate outside the training data: A case study in Textualized Gridworld](https://mlsanity.com/paper/2406.15275/) (2024)
- [Large Language Model Powered Intelligent Urban Agents: Concepts, Capabilities, and Applications](https://mlsanity.com/paper/2507.00914/) (2025)
- [An Empirical Study of Conformal Prediction in LLM with ASP Scaffolds for Robust Reasoning](https://mlsanity.com/paper/2503.05439/) (2025)
- [On the Performance of LLMs for Real Estate Appraisal](https://mlsanity.com/paper/2506.11812/) (2025)
- [Geospatial Mechanistic Interpretability of Large Language Models](https://mlsanity.com/paper/2505.03368/) (2025)
- [Leveraging LLMs for Mission Planning in Precision Agriculture](https://mlsanity.com/paper/2506.10093/) (2025)
- [CONSCIENTIA: Can LLM Agents Learn to Strategize? Emergent Deception and Trust in a Multi-Agent NYC Simulation](https://mlsanity.com/paper/2604.09746/) (2026)
- [DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving](https://mlsanity.com/paper/2605.23176/) (2026)
- [Evaluating the Ability of Large Language Models to Reason about Cardinal Directions, Revisited](https://mlsanity.com/paper/2507.12059/) (2025)
- [3DFroMLLM: 3D Prototype Generation only from Pretrained Multimodal LLMs](https://mlsanity.com/paper/2508.08821/) (2025)
- [DecompSR: A dataset for decomposed analyses of compositional multihop spatial reasoning](https://mlsanity.com/paper/2511.02627/) (2025)
- [Spatially-Enhanced Retrieval-Augmented Generation for Walkability and Urban Discovery](https://mlsanity.com/paper/2512.04790/) (2025)
- [How Do LLMs and VLMs Understand Viewpoint Rotation Without Vision? An Interpretability Study](https://mlsanity.com/paper/2604.15294/) (2026)
- [Spatial Reasoning via Modality Switching Between Language and Symbolic Representations](https://mlsanity.com/paper/2606.31285/) (2026)
- [Influence of Solution Efficiency and Valence of Instruction on Additive and Subtractive Solution Strategies in Humans and GPT-4](https://mlsanity.com/paper/2404.16692/) (2024)
- [From Actions to Words: Towards Abstractive-Textual Policy Summarization in RL](https://mlsanity.com/paper/2503.10509/) (2025)
- [AlphaSpace: Enabling Robotic Actions through Semantic Tokenization and Symbolic Reasoning](https://mlsanity.com/paper/2503.18769/) (2025)
- [Grid Spatial Understanding: A Dataset for Textual Spatial Reasoning over Grids, Embodied Settings, and Coordinate Structures](https://mlsanity.com/paper/2603.17333/) (2026)
- [StateVLM: A State-Aware Vision-Language Model for Robotic Affordance Reasoning](https://mlsanity.com/paper/2605.03927/) (2026)
- [Towards a Data-Centric Understanding of Spatial Reasoning in Multimodal Large Language Models](https://mlsanity.com/paper/2610.19108/) (2026)

## Abstract (arXiv)

Artificial intelligence (AI) has made remarkable progress across various domains, with large language models like ChatGPT gaining substantial attention for their human-like text-generation capabilities. Despite these achievements, spatial reasoning remains a significant challenge for these models. Benchmarks like StepGame evaluate AI spatial reasoning, where ChatGPT has shown unsatisfactory performance. However, the presence of template errors in the benchmark has an impact on the evaluation results. Thus there is potential for ChatGPT to perform better if these template errors are addressed, leading to more accurate assessments of its spatial reasoning capabilities. In this study, we refine the StepGame benchmark, providing a more accurate dataset for model evaluation. We analyze GPT's spatial reasoning performance on the rectified benchmark, identifying proficiency in mapping natural language text to spatial relations but limitations in multi-hop reasoning. We provide a flawless solution to the benchmark by combining template-to-relation mapping with logic-based reasoning. This combination demonstrates proficiency in performing qualitative reasoning on StepGame without encountering any errors. We then address the limitations of GPT models in spatial reasoning. We deploy Chain-of-thought and Tree-of-thoughts prompting strategies, offering insights into GPT's ``cognitive process", and achieving remarkable improvements in accuracy. Our investigation not only sheds light on model deficiencies but also proposes enhancements, contributing to the advancement of AI with more robust spatial reasoning capabilities.