# TurnBack: A Geospatial Route Cognition Benchmark for Large Language Models through Reverse Route

arXiv: https://arxiv.org/abs/2509.18173 · HTML: https://mlsanity.com/paper/2509.18173/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 51 references that are in the map.
- First posted on arXiv 17 Sep 2025.

## Builds on (most cited first, 19 of 19)

- [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](https://mlsanity.com/paper/2412.14171/) (2024)
- [Large Language Model Alignment: A Survey](https://mlsanity.com/paper/2309.15025/) (2023)
- [Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning](https://mlsanity.com/paper/2403.11401/) (2024)
- [Does Prompt Formatting Have Any Impact on LLM Performance?](https://mlsanity.com/paper/2411.10541/) (2024)
- [ST-LLM: Large Language Models Are Effective Temporal Learners](https://mlsanity.com/paper/2404.00308/) (2024)
- [GeomVerse: A Systematic Evaluation of Large Models for Geometric Reasoning](https://mlsanity.com/paper/2312.12241/) (2023)
- [Evaluating Cognitive Maps and Planning in Large Language Models with CogEval](https://mlsanity.com/paper/2309.15129/) (2023)
- [Exploring the Encoding Layer and Loss Function in End-to-End Speaker and Language Recognition System](https://mlsanity.com/paper/1804.05160/) (2018)
- [Number Cookbook: Number Understanding of Language Models and How to Improve It](https://mlsanity.com/paper/2411.03766/) (2024)
- [Are Large Language Models Geospatially Knowledgeable?](https://mlsanity.com/paper/2310.13002/) (2023)
- [CityBench: Evaluating the Capabilities of Large Language Models for Urban Tasks](https://mlsanity.com/paper/2406.13945/) (2024)
- [Route Planning in Transportation Networks](https://mlsanity.com/paper/1504.05140/) (2015)
- [Generating Landmark Navigation Instructions from Maps as a Graph-to-Text Problem](https://mlsanity.com/paper/2012.15329/) (2020)
- [Visual Memory for Robust Path Following](https://mlsanity.com/paper/1812.00940/) (2018)
- [Semantic Trajectory Data Mining with LLM-Informed POI Classification](https://mlsanity.com/paper/2405.11715/) (2024)
- [Benchmarking Large Language Models for Math Reasoning Tasks](https://mlsanity.com/paper/2408.10839/) (2024)
- [MANGO: A Benchmark for Evaluating Mapping and Navigation Abilities of Large Language Models](https://mlsanity.com/paper/2403.19913/) (2024)
- [Geode: A Zero-shot Geospatial Question-Answering Agent with Explicit Reasoning and Precise Spatio-Temporal Retrieval](https://mlsanity.com/paper/2407.11014/) (2024)
- [Trajectory Representation Learning on Road Networks and Grids with Spatio-Temporal Dynamics](https://mlsanity.com/paper/2411.14014/) (2024)

## Abstract (arXiv)

Humans can interpret geospatial information through natural language, while the geospatial cognition capabilities of Large Language Models (LLMs) remain underexplored. Prior research in this domain has been constrained by non-quantifiable metrics, limited evaluation datasets and unclear research hierarchies. Therefore, we propose a large-scale benchmark and conduct a comprehensive evaluation of the geospatial route cognition of LLMs. We create a large-scale evaluation dataset comprised of 36000 routes from 12 metropolises worldwide. Then, we introduce PathBuilder, a novel tool for converting natural language instructions into navigation routes, and vice versa, bridging the gap between geospatial information and natural language. Finally, we propose a new evaluation framework and metrics to rigorously assess 11 state-of-the-art (SOTA) LLMs on the task of route reversal. The benchmark reveals that LLMs exhibit limitation to reverse routes: most reverse routes neither return to the starting point nor are similar to the optimal route. Additionally, LLMs face challenges such as low robustness in route generation and high confidence for their incorrect answers. Code & Data available here: hrefhttps://github.com/bghjmn32/EMNLP2025TurnbackTurnBack.