# Evaluating the Ability of Large Language Models to Reason about Cardinal Directions, Revisited

arXiv: https://arxiv.org/abs/2507.12059 · HTML: https://mlsanity.com/paper/2507.12059/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 16 Jul 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 20 of 20)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models](https://mlsanity.com/paper/2501.09686/) (2025)
- [Faithful Reasoning Using Large Language Models](https://mlsanity.com/paper/2208.14271/) (2022)
- [SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoning](https://mlsanity.com/paper/2104.05832/) (2021)
- [StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts](https://mlsanity.com/paper/2204.08292/) (2022)
- [Evaluating Spatial Understanding of Large Language Models](https://mlsanity.com/paper/2310.14540/) (2023)
- [GeoLM: Empowering Language Models for Geospatially Grounded Language Understanding](https://mlsanity.com/paper/2310.14478/) (2023)
- [Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmark](https://mlsanity.com/paper/2401.03991/) (2024)
- [STEER: Assessing the Economic Rationality of Large Language Models](https://mlsanity.com/paper/2402.09552/) (2024)
- [Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores](https://mlsanity.com/paper/2410.03492/) (2024)
- [Reframing Spatial Reasoning Evaluation in Language Models: A Real-World Simulation Benchmark for Qualitative Reasoning](https://mlsanity.com/paper/2405.15064/) (2024)
- [Evaluating the Ability of Large Language Models to Reason about Cardinal Directions](https://mlsanity.com/paper/2406.16528/) (2024)
- [An Evaluation of ChatGPT-4's Qualitative Spatial Reasoning Capabilities in RCC-8](https://mlsanity.com/paper/2309.15577/) (2023)
- [A Survey on Prompting Techniques in LLMs](https://mlsanity.com/paper/2312.03740/) (2023)
- [Distortions in Judged Spatial Relations in Large Language Models](https://mlsanity.com/paper/2401.04218/) (2024)
- [Can Large Language Models Reason about the Region Connection Calculus?](https://mlsanity.com/paper/2411.19589/) (2024)
- [Do Multimodal Language Models Really Understand Direction? A Benchmark for Compass Direction Reasoning](https://mlsanity.com/paper/2412.16599/) (2024)

## Built on by (most cited first, 1 of 1)

- [QSTRBench: a New Benchmark to Evaluate the Ability of Language Models to Reason with Qualitative Spatial and Temporal Calculi](https://mlsanity.com/paper/2605.18380/) (2026)

## Abstract (arXiv)

We investigate the abilities of 28 Large language Models (LLMs) to reason about cardinal directions (CDs) using a benchmark generated from a set of templates, extensively testing an LLM's ability to determine the correct CD given a particular scenario. The templates allow for a number of degrees of variation such as means of locomotion of the agent involved, and whether set in the first, second or third person. Even the newer Large Reasoning Models are unable to reliably determine the correct CD for all questions. This paper summarises and extends earlier work presented at COSIT-24.