mlsanityData to 14 Oct 2026

Evaluating the Ability of Large Language Models to Reason about Cardinal Directions, Revisited

arXiv 2507.12059 ↗First posted 16 Jul 2025cs.CLcited by 1

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We investigate the abilities of 28 Large language Models (LLMs) to reason about cardinal directions (CDs) using a benchmark generated from a set of templates, extensively testing an LLM's ability to determine the correct CD given a particular scenario. The templates allow for a number of degrees of variation such as means of locomotion of the agent involved, and whether set in the first, second or third person. Even the newer Large Reasoning Models are unable to reliably determine the correct CD for all questions. This paper summarises and extends earlier work presented at COSIT-24.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 20 of its 27 references that are in the map. First posted on arXiv 16 Jul 2025.

Builds on

20 of 27 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models16 Jan
  2. 2024Evaluating the Ability of Large Language Models to Reason about Cardinal Directions24 Jun
  3. 2022Towards Reasoning in Large Language Models: A Survey20 Dec
  4. 2022Faithful Reasoning Using Large Language Models30 Aug

Other references · 16

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  3. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 314
  4. 2021SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoningcited by 77
  5. 2022StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Textscited by 68
  6. 2023Evaluating Spatial Understanding of Large Language Modelscited by 66
  7. 2023GeoLM: Empowering Language Models for Geospatially Grounded Language Understandingcited by 37
  8. 2024Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmarkcited by 27
  9. 2024STEER: Assessing the Economic Rationality of Large Language Modelscited by 27
  10. 2024Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scorescited by 25
  11. 2024Reframing Spatial Reasoning Evaluation in Language Models: A Real-World Simulation Benchmark for Qualitative Reasoningcited by 17
  12. 2023An Evaluation of ChatGPT-4's Qualitative Spatial Reasoning Capabilities in RCC-8cited by 9
4 more
  1. 2023A Survey on Prompting Techniques in LLMscited by 9
  2. 2024Distortions in Judged Spatial Relations in Large Language Modelscited by 5
  3. 2024Can Large Language Models Reason about the Region Connection Calculus?cited by 5
  4. 2024Do Multimodal Language Models Really Understand Direction? A Benchmark for Compass Direction Reasoningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026QSTRBench: a New Benchmark to Evaluate the Ability of Language Models to Reason with Qualitative Spatial and Temporal Calculicited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Opening the Black Box: A Survey on the Mechanisms of Multi-Step Reasoning in Large Language Models
  2. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  3. 2024Conditional and Modal Reasoning in Large Language Models
  4. 2024Reasoning in Large Language Models: A Geometric Perspective
  5. 2025Large Language and Reasoning Models are Shallow Disjunctive Reasoners
  6. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
  7. 2025Reasoning Capabilities and Invariability of Large Language Models
  8. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey