mlsanityData to 14 Oct 2026

TurnBack: A Geospatial Route Cognition Benchmark for Large Language Models through Reverse Route

arXiv 2509.18173 ↗First posted 17 Sep 2025cs.LG cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Humans can interpret geospatial information through natural language, while the geospatial cognition capabilities of Large Language Models (LLMs) remain underexplored. Prior research in this domain has been constrained by non-quantifiable metrics, limited evaluation datasets and unclear research hierarchies. Therefore, we propose a large-scale benchmark and conduct a comprehensive evaluation of the geospatial route cognition of LLMs. We create a large-scale evaluation dataset comprised of 36000 routes from 12 metropolises worldwide. Then, we introduce PathBuilder, a novel tool for converting natural language instructions into navigation routes, and vice versa, bridging the gap between geospatial information and natural language. Finally, we propose a new evaluation framework and metrics to rigorously assess 11 state-of-the-art (SOTA) LLMs on the task of route reversal. The benchmark reveals that LLMs exhibit limitation to reverse routes: most reverse routes neither return to the starting point nor are similar to the optimal route. Additionally, LLMs face challenges such as low robustness in route generation and high confidence for their incorrect answers. Code & Data available here: hrefhttps://github.com/bghjmn32/EMNLP2025TurnbackTurnBack.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 51 references that are in the map. First posted on arXiv 17 Sep 2025.

Builds on

19 of 51 references.

References · 19

  1. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 611
  2. 2023Large Language Model Alignment: A Surveycited by 243
  3. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 175
  4. 2024Does Prompt Formatting Have Any Impact on LLM Performance?cited by 141
  5. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  6. 2023GeomVerse: A Systematic Evaluation of Large Models for Geometric Reasoningcited by 108
  7. 2023Evaluating Cognitive Maps and Planning in Large Language Models with CogEvalcited by 77
  8. 2018Exploring the Encoding Layer and Loss Function in End-to-End Speaker and Language Recognition Systemcited by 47
  9. 2024Number Cookbook: Number Understanding of Language Models and How to Improve Itcited by 42
  10. 2023Are Large Language Models Geospatially Knowledgeable?cited by 30
  11. 2024CityBench: Evaluating the Capabilities of Large Language Models for Urban Taskscited by 27
  12. 2015Route Planning in Transportation Networkscited by 23
7 more
  1. 2020Generating Landmark Navigation Instructions from Maps as a Graph-to-Text Problemcited by 22
  2. 2018Visual Memory for Robust Path Followingcited by 16
  3. 2024Semantic Trajectory Data Mining with LLM-Informed POI Classificationcited by 13
  4. 2024Benchmarking Large Language Models for Math Reasoning Taskscited by 12
  5. 2024MANGO: A Benchmark for Evaluating Mapping and Navigation Abilities of Large Language Modelscited by 9
  6. 2024Geode: A Zero-shot Geospatial Question-Answering Agent with Explicit Reasoning and Precise Spatio-Temporal Retrievalcited by 5
  7. 2024Trajectory Representation Learning on Road Networks and Grids with Spatio-Temporal Dynamicscited by 2

Built on by

Not cited yet.