mlsanityData to 7 Oct 2026

Inherent limitations of LLMs regarding spatial information

arXiv 2312.03042 ↗First posted 5 Dec 2023cs.CL cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Despite the significant advancements in natural language processing capabilities demonstrated by large language models such as ChatGPT, their proficiency in comprehending and processing spatial information, especially within the domains of 2D and 3D route planning, remains notably underdeveloped. This paper investigates the inherent limitations of ChatGPT and similar models in spatial reasoning and navigation-related tasks, an area critical for applications ranging from autonomous vehicle guidance to assistive technologies for the visually impaired. In this paper, we introduce a novel evaluation framework complemented by a baseline dataset, meticulously crafted for this study. This dataset is structured around three key tasks: plotting spatial points, planning routes in two-dimensional (2D) spaces, and devising pathways in three-dimensional (3D) environments. We specifically developed this dataset to assess the spatial reasoning abilities of ChatGPT. Our evaluation reveals key insights into the model's capabilities and limitations in spatial understanding.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 21 of its 27 references that are in the map. First posted on arXiv 5 Dec 2023.

Builds on

21 of 27 references.

References · 21

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  8. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  9. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  10. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  11. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 414
  12. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
9 more
  1. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  2. 2023Understanding Social Reasoning in Language Models with Language Modelscited by 166
  3. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  4. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  5. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  6. 2023The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domaincited by 97
  7. 2021NaturalProofs: Mathematical Theorem Proving in Natural Languagecited by 80
  8. 2023Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Taskscited by 54
  9. 2023FIND: A Function Description Benchmark for Evaluating Interpretability Methodscited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025LLM Meets Scene Graph: Can Large Language Models Understand and Generate Scene Graphs? A Benchmark and Empirical Studycited by 9
  2. 2025CrashAgent: Crash Scenario Generation via Multi-modal Reasoningcited by 5
  3. 2026Spatial-Agent: Agentic Geo-spatial Reasoning with Scientific Core Conceptscited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2026Do LLMs Build Spatial World Models? Evidence from Grid-World Maze Tasks
  2. 2025A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Science
  3. 2026The Sword, Shield, and Achilles' Heel: Characterizing the Linguistic Inductive Bias of Large Language Models for Spatial Reasoning in Navigation Planning
  4. 2024Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmark
  5. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models
  6. 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipes
  7. 2024Evaluating Large Language Models on Spatial Tasks: A Multi-Task Benchmarking Study
  8. 2025Stuck in the Matrix: Probing Spatial Reasoning in Large Language Models