mlsanityData to 7 Oct 2026

Evaluating Spatial Understanding of Large Language Models

arXiv 2310.14540 ↗First posted 23 Oct 2023cs.CL cs.AIcited by 65

Not on the map. Why?

Cited by 34 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) show remarkable capabilities across a variety of tasks. Despite the models only seeing text in training, several recent studies suggest that LLM representations implicitly capture aspects of the underlying grounded concepts. Here, we explore LLM representations of a particularly salient kind of grounded knowledge -- spatial relationships. We design natural-language navigation tasks and evaluate the ability of LLMs, in particular GPT-3.5-turbo, GPT-4, and Llama2 series models, to represent and reason about spatial structures. These tasks reveal substantial variability in LLM performance across different spatial structures, including square, hexagonal, and triangular grids, rings, and trees. In extensive error analysis, we find that LLMs' mistakes reflect both spatial and non-spatial factors. These findings suggest that LLMs appear to capture certain aspects of spatial structure implicitly, but room for improvement remains.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Builds on 10 of its 21 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

10 of 21 references.

References · 10

  1. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  3. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  4. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  5. 2021Implicit Representations of Meaning in Neural Language Modelscited by 179
  6. 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 118
  7. 2023Evaluating Cognitive Maps and Planning in Large Language Models with CogEvalcited by 77
  8. 2021SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoningcited by 75
  9. 2022StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Textscited by 65
  10. 2023SayNav: Grounding Large Language Models for Dynamic Planning to Navigation in New Environmentscited by 53

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 31; 2026: 16.

65 papers.

Sort:

By area:

  1. 2025Gemma 3 Technical Reportcited by 1,686
  2. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  3. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 141
  4. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
  5. 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 130
  6. 2025BIG-Bench Extra Hardcited by 123
  7. 2024Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Modelscited by 65
  8. 2025SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognitioncited by 35
  9. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  10. 2024CityGPT: Empowering Urban Spatial Cognition of Large Language Modelscited by 29
  11. 2024VideoINSTA: Zero-shot Long Video Understanding via Informative Spatial-Temporal Reasoning with LLMscited by 29
  12. 2025iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMscited by 24
38 more
  1. 2025Foundation Models for Geospatial Reasoning: Assessing Capabilities of Large Language Models in Understanding Geometries and Topological Spatial Relationscited by 20
  2. 2024Whiteboard-of-Thought: Thinking Step-by-Step Across Modalitiescited by 15
  3. 2025SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewardscited by 14
  4. 2025FloorplanQA: A Benchmark for Spatial Reasoning in LLMs using Structured Representationscited by 13
  5. 2024Evaluating Large Language Models on Spatial Tasks: A Multi-Task Benchmarking Studycited by 12
  6. 2024Scaffolding Language Learning via Multi-modal Tutoring Systems with Pedagogical Instructionscited by 11
  7. 2024STBench: Assessing the Ability of Large Language Models in Spatio-Temporal Analysiscited by 11
  8. 2025A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Sciencecited by 11
  9. 2024Evaluating the Ability of Large Language Models to Reason about Cardinal Directionscited by 10
  10. 2025From Text to Space: Mapping Abstract Spatial Models in LLMs during a Grid-World Navigation Taskcited by 10
  11. 2024How language models extrapolate outside the training data: A case study in Textualized Gridworldcited by 7
  12. 2024GRASP: A Grid-Based Benchmark for Evaluating Commonsense Spatial Reasoningcited by 7
  13. 2024Grammar-based Game Description Generation using Large Language Modelscited by 7
  14. 2025Evaluating Large Language Models for Real-World Engineering Taskscited by 7
  15. 2026El Agente Quntur: A research collaborator agent for quantum chemistrycited by 7
  16. 2024Evaluating the Generation of Spatial Relations in Text and Image Generative Modelscited by 6
  17. 2024Evaluating Vision-Language Models as Evaluators in Path Planningcited by 6
  18. 2025Factorio Learning Environmentcited by 6
  19. 2024Can Large Language Models Reason about the Region Connection Calculus?cited by 5
  20. 2025USTBench: Benchmarking and Dissecting Spatiotemporal Reasoning of LLMs as Urban Agentscited by 5
  21. 2025An LLM-enabled Multi-Agent Autonomous Mechatronics Design Frameworkcited by 4
  22. 2025Scene Graph-Guided Proactive Replanning for Failure-Resilient Embodied Agentcited by 4
  23. 2025GraphicBench: A Planning Benchmark for Graphic Design with Language Agentscited by 3
  24. 2025MapTrace: Scalable Data Generation for Route Tracing on Mapscited by 3
  25. 2026See, Remember, Explore: A Benchmark and Baselines for Streaming Spatial Reasoningcited by 3
  26. 2026MultiView-Bench: A Diagnostic Benchmark for World-Centric Multi-View Integration in VLMscited by 3
  27. 2025SCoRE: Benchmarking Long-Chain Reasoning in Commonsense Scenarioscited by 2
  28. 2025From Reasoning to Generalization: Knowledge-Augmented LLMs for ARC Benchmarkcited by 2
  29. 2025GIQ: Benchmarking 3D Geometric Reasoning of Vision Foundation Models with Simulated and Real Polyhedracited by 2
  30. 2026MosaicThinker: On-Device Visual Spatial Reasoning for Embodied AI via Iterative Construction of Space Representationcited by 2
  31. 2026GPSBench: Do Large Language Models Understand GPS Coordinates?cited by 2
  32. 2026STAR: Mitigating Cascading Errors in Spatial Reasoning via Turn-point Alignment and Segment-level DPOcited by 2
  33. 2025The World As Large Language Models See It: Exploring the reliability of LLMs in representing geographical featurescited by 1
  34. 2025A Study on Individual Spatiotemporal Activity Generation Method Using MCP-Enhanced Chain-of-Thought Large Language Modelscited by 1
  35. 2025Evaluating the Ability of Large Language Models to Reason about Cardinal Directions, Revisitedcited by 1
  36. 2025GOPLA: Generalizable Object Placement Learning via Synthetic Augmentation of Human Arrangementcited by 1
  37. 2026Language-based Trial and Error Falls Behind in the Era of Experiencecited by 1
  38. 2026From Human Cognition to Neural Activations: Probing the Computational Primitives of Spatial Reasoning in LLMscited by 1

50 most cited of 65 All 65 (JSON)