PLUGH: A Benchmark for Spatial Understanding and Reasoning in Large Language Models
On the map: LLM reasoning → Spatial and embodied reasoning
Abstract (arXiv)
We present PLUGH (https://www.urbandictionary.com/define.php?term=plugh), a modern benchmark that currently consists of 5 tasks, each with 125 input texts extracted from 48 different games and representing 61 different (non-isomorphic) spatial graphs to assess the abilities of Large Language Models (LLMs) for spatial understanding and reasoning. Our evaluation of API-based and open-sourced LLMs shows that while some commercial LLMs exhibit strong reasoning abilities, open-sourced competitors can demonstrate almost the same level of quality; however, all models still have significant room for improvement. We identify typical reasons for LLM failures and discuss possible ways to deal with them. Datasets and evaluation code are released (https://github.com/altsoph/PLUGH).
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 11 of its 25 references that are in the map. First posted on arXiv 3 Aug 2024.
Builds on
11 of 25 references.
Closest ancestors · 1
Same direction, newest first
Other references · 10
- 2023GPT-4 Technical Reportcited by 18,396
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
- 2024Mixtral of Expertscited by 1,823
- 2023Cognitive Architectures for Language Agentscited by 429
- 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 312
- 2018TextWorld: A Learning Environment for Text-based Gamescited by 310
- 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 182
- 2019Interactive Fiction Games: A Colossal Adventurecited by 168
- 2023Are NLP Models Good at Tracing Thoughts: An Overview of Narrative Understandingcited by 9
- 2021Actionable Entities Recognition Benchmark for Interactive Fictioncited by 2
Built on by
Citing papers per year: 2025: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025Stuck in the Matrix: Probing Spatial Reasoning in Large Language Models
- 2024SpaRC and SpaRP: Spatial Reasoning Characterization and Path Generation for Understanding Spatial Reasoning Capability of Large Language Models
- 2026Do LLMs Build Spatial World Models? Evidence from Grid-World Maze Tasks
- 2026Do LLMs Build World Models From Text? A Multilingual Diagnostic of Spatial Reasoning
- 2024Can Large Language Models Create New Knowledge for Spatial Reasoning Tasks?
- 2025LTD-Bench: Evaluating Large Language Models by Letting Them Draw
- 2024Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmark
- 2026From Symbolic to Geometric: Enabling Spatial Reasoning in Large Language Models