# Distortions in Judged Spatial Relations in Large Language Models

arXiv: https://arxiv.org/abs/2401.04218 · HTML: https://mlsanity.com/paper/2401.04218/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 30 references that are in the map.
- First posted on arXiv 8 Jan 2024.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"](https://mlsanity.com/paper/2309.12288/) (2023)
- [Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4](https://mlsanity.com/paper/2304.03439/) (2023)
- [How well do Large Language Models perform in Arithmetic tasks?](https://mlsanity.com/paper/2304.02015/) (2023)
- [Capturing Failures of Large Language Models via Human Cognitive Biases](https://mlsanity.com/paper/2202.12299/) (2022)
- [GPT4GEO: How a Language Model Sees the World's Geography](https://mlsanity.com/paper/2306.00020/) (2023)
- [Autonomous GIS: the next-generation AI-powered GIS](https://mlsanity.com/paper/2305.06453/) (2023)
- [GeoGPT: Understanding and Processing Geospatial Tasks through An Autonomous GPT](https://mlsanity.com/paper/2307.07930/) (2023)
- [Evaluating the Effectiveness of Large Language Models in Representing Textual Descriptions of Geometry and Spatial Relations](https://mlsanity.com/paper/2307.03678/) (2023)
- [An Evaluation of ChatGPT-4's Qualitative Spatial Reasoning Capabilities in RCC-8](https://mlsanity.com/paper/2309.15577/) (2023)

## Built on by (most cited first, 5 of 5)

- [Foundation Models for Geospatial Reasoning: Assessing Capabilities of Large Language Models in Understanding Geometries and Topological Spatial Relations](https://mlsanity.com/paper/2505.17136/) (2025)
- [GPSBench: Do Large Language Models Understand GPS Coordinates?](https://mlsanity.com/paper/2602.16105/) (2026)
- [Geospatial Mechanistic Interpretability of Large Language Models](https://mlsanity.com/paper/2505.03368/) (2025)
- [Evaluating the Ability of Large Language Models to Reason about Cardinal Directions, Revisited](https://mlsanity.com/paper/2507.12059/) (2025)
- [Assessing the Geographic Diversity of AI's Platial Representations in Image Generation](https://mlsanity.com/paper/2606.05188/) (2026)

## Abstract (arXiv)

We present a benchmark for assessing the capability of Large Language Models (LLMs) to discern intercardinal directions between geographic locations and apply it to three prominent LLMs: GPT-3.5, GPT-4, and Llama-2. This benchmark specifically evaluates whether LLMs exhibit a hierarchical spatial bias similar to humans, where judgments about individual locations' spatial relationships are influenced by the perceived relationships of the larger groups that contain them. To investigate this, we formulated 14 questions focusing on well-known American cities. Seven questions were designed to challenge the LLMs with scenarios potentially influenced by the orientation of larger geographical units, such as states or countries, while the remaining seven targeted locations were less susceptible to such hierarchical categorization. Among the tested models, GPT-4 exhibited superior performance with 55 percent accuracy, followed by GPT-3.5 at 47 percent, and Llama-2 at 45 percent. The models showed significantly reduced accuracy on tasks with suspected hierarchical bias. For example, GPT-4's accuracy dropped to 33 percent on these tasks, compared to 86 percent on others. However, the models identified the nearest cardinal direction in most cases, reflecting their associative learning mechanism, thereby embodying human-like misconceptions. We discuss avenues for improving the spatial reasoning capabilities of LLMs.