mlsanityData to 7 Oct 2026

Evaluating the Ability of Large Language Models to Reason about Cardinal Directions

arXiv 2406.16528 ↗First posted 24 Jun 2024cs.CLcited by 10

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We investigate the abilities of a representative set of Large language Models (LLMs) to reason about cardinal directions (CDs). To do so, we create two datasets: the first, co-created with ChatGPT, focuses largely on recall of world knowledge about CDs; the second is generated from a set of templates, comprehensively testing an LLM's ability to determine the correct CD given a particular scenario. The templates allow for a number of degrees of variation such as means of locomotion of the agent involved, and whether set in the first , second or third person. Even with a temperature setting of zero, Our experiments show that although LLMs are able to perform well in the simpler dataset, in the second more complex dataset no LLM is able to reliably determine the correct CD, even with a temperature setting of zero.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 11 of its 19 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

11 of 19 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models23 Oct
  2. 2022Towards Reasoning in Large Language Models: A Survey20 Dec
  3. 2022Faithful Reasoning Using Large Language Models30 Aug

Other references · 8

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  3. 2023Evaluating Spatial Understanding of Large Language Modelscited by 65
  4. 2024Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Modelscited by 65
  5. 2023GeoLM: Empowering Language Models for Geospatially Grounded Language Understandingcited by 37
  6. 2024STEER: Assessing the Economic Rationality of Large Language Modelscited by 26
  7. 2023An Evaluation of ChatGPT-4's Qualitative Spatial Reasoning Capabilities in RCC-8cited by 9
  8. 2023A Survey on Prompting Techniques in LLMscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 2.

10 papers.

  1. 2024Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scorescited by 25
  2. 2025Foundation Models for Geospatial Reasoning: Assessing Capabilities of Large Language Models in Understanding Geometries and Topological Spatial Relationscited by 20
  3. 2024Can Large Language Models Reason about the Region Connection Calculus?cited by 5
  4. 2025Geospatial Mechanistic Interpretability of Large Language Modelscited by 2
  5. 2025Evaluating the Ability of Large Language Models to Reason about Cardinal Directions, Revisitedcited by 1
  6. 20253DFroMLLM: 3D Prototype Generation only from Pretrained Multimodal LLMscited by 1
  7. 2025GeoResponder: Towards Building Geospatial LLMs for Time-Critical Disaster Responsecited by 1
  8. 2025CartoMapQA: A Fundamental Benchmark Dataset Evaluating Vision-Language Models on Cartographic Map Understandingcited by 0
  9. 2026LMM Modality Transfer: A Pre-requisite for Autonomous GIS Agentscited by 0
  10. 2026Recovering Temporal and Geographic Signals from Language Model Embeddingscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Can Large Language Models put 2 and 2 together? Probing for Entailed Arithmetical Relationships
  2. 2026Opening the Black Box: A Survey on the Mechanisms of Multi-Step Reasoning in Large Language Models
  3. 2025Reasoning Capabilities and Invariability of Large Language Models
  4. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
  5. 2024Conditional and Modal Reasoning in Large Language Models
  6. 2024Reasoning in Large Language Models: A Geometric Perspective
  7. 2025Beyond Chains of Thought: Benchmarking Latent-Space Reasoning Abilities in Large Language Models
  8. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance