# Evaluating the Ability of Large Language Models to Reason about Cardinal Directions

arXiv: https://arxiv.org/abs/2406.16528 · HTML: https://mlsanity.com/paper/2406.16528/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 11 of its 19 references that are in the map.
- First posted on arXiv 24 Jun 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 11 of 11)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [Faithful Reasoning Using Large Language Models](https://mlsanity.com/paper/2208.14271/) (2022)
- [Evaluating Spatial Understanding of Large Language Models](https://mlsanity.com/paper/2310.14540/) (2023)
- [Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models](https://mlsanity.com/paper/2404.03622/) (2024)
- [Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models](https://mlsanity.com/paper/2310.14491/) (2023)
- [GeoLM: Empowering Language Models for Geospatially Grounded Language Understanding](https://mlsanity.com/paper/2310.14478/) (2023)
- [STEER: Assessing the Economic Rationality of Large Language Models](https://mlsanity.com/paper/2402.09552/) (2024)
- [An Evaluation of ChatGPT-4's Qualitative Spatial Reasoning Capabilities in RCC-8](https://mlsanity.com/paper/2309.15577/) (2023)
- [A Survey on Prompting Techniques in LLMs](https://mlsanity.com/paper/2312.03740/) (2023)

## Built on by (most cited first, 10 of 10)

- [Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores](https://mlsanity.com/paper/2410.03492/) (2024)
- [Foundation Models for Geospatial Reasoning: Assessing Capabilities of Large Language Models in Understanding Geometries and Topological Spatial Relations](https://mlsanity.com/paper/2505.17136/) (2025)
- [Can Large Language Models Reason about the Region Connection Calculus?](https://mlsanity.com/paper/2411.19589/) (2024)
- [Geospatial Mechanistic Interpretability of Large Language Models](https://mlsanity.com/paper/2505.03368/) (2025)
- [Evaluating the Ability of Large Language Models to Reason about Cardinal Directions, Revisited](https://mlsanity.com/paper/2507.12059/) (2025)
- [3DFroMLLM: 3D Prototype Generation only from Pretrained Multimodal LLMs](https://mlsanity.com/paper/2508.08821/) (2025)
- [GeoResponder: Towards Building Geospatial LLMs for Time-Critical Disaster Response](https://mlsanity.com/paper/2509.19354/) (2025)
- [CartoMapQA: A Fundamental Benchmark Dataset Evaluating Vision-Language Models on Cartographic Map Understanding](https://mlsanity.com/paper/2512.03558/) (2025)
- [LMM Modality Transfer: A Pre-requisite for Autonomous GIS Agents](https://mlsanity.com/paper/2608.06948/) (2026)
- [Recovering Temporal and Geographic Signals from Language Model Embeddings](https://mlsanity.com/paper/2609.05721/) (2026)

## Abstract (arXiv)

We investigate the abilities of a representative set of Large language Models (LLMs) to reason about cardinal directions (CDs). To do so, we create two datasets: the first, co-created with ChatGPT, focuses largely on recall of world knowledge about CDs; the second is generated from a set of templates, comprehensively testing an LLM's ability to determine the correct CD given a particular scenario. The templates allow for a number of degrees of variation such as means of locomotion of the agent involved, and whether set in the first , second or third person. Even with a temperature setting of zero, Our experiments show that although LLMs are able to perform well in the simpler dataset, in the second more complex dataset no LLM is able to reliably determine the correct CD, even with a temperature setting of zero.