mlsanityData to 7 Oct 2026

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models

arXiv 2501.00316 ↗First posted 31 Dec 2024cs.CLcited by 18

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Recent advancements in foundation models have improved autonomous tool usage and reasoning, but their capabilities in map-based reasoning remain underexplored. To address this, we introduce MapEval, a benchmark designed to assess foundation models across three distinct tasks - textual, API-based, and visual reasoning - through 700 multiple-choice questions spanning 180 cities and 54 countries, covering spatial relationships, navigation, travel planning, and real-world map interactions. Unlike prior benchmarks that focus on simple location queries, MapEval requires models to handle long-context reasoning, API interactions, and visual map analysis, making it the most comprehensive evaluation framework for geospatial AI. On evaluation of 30 foundation models, including Claude-3.5-Sonnet, GPT-4o, and Gemini-1.5-Pro, none surpass 67% accuracy, with open-source models performing significantly worse and all models lagging over 20% behind human performance. These results expose critical gaps in spatial inference, as models struggle with distances, directions, route planning, and place-specific reasoning, highlighting the need for better geospatial AI to bridge the gap between foundation models and real-world navigation. All the resources are available at: https://mapeval.github.io/.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 12 of its 54 references that are in the map. First posted on arXiv 31 Dec 2024.

Builds on

12 of 54 references.

References · 12

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  6. 2024Mixtral of Expertscited by 1,785
  7. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  8. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  9. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  10. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  11. 2024TraveLLM: Could you plan my new public transit route in face of a network disruption?cited by 5
  12. 2024MapQaTor: An Extensible Framework for Efficient Annotation of Map-Based QA Datasetscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 13.

18 papers.

Sort:

By area:

  1. 2025ReasonMap: Towards Fine-Grained Visual Reasoning from Transit Mapscited by 20
  2. 2026MLLM-4D: Towards Visual-based Spatial-Temporal Intelligencecited by 8
  3. 2025Think in Games: Learning to Reason in Games via Reinforcement Learning with Large Language Modelscited by 7
  4. 2025GeoArena: Evaluating Open-World Geographic Reasoning in Large Vision-Language Modelscited by 5
  5. 2026Spatial-Agent: Agentic Geo-spatial Reasoning with Scientific Core Conceptscited by 5
  6. 2026MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphscited by 3
  7. 2026GPSBench: Do Large Language Models Understand GPS Coordinates?cited by 2
  8. 2025Understanding the Geospatial Reasoning Capabilities of LLMs: A Trajectory Recovery Perspectivecited by 1
  9. 2026Mil-SCORE: Benchmarking Long-Context Geospatial Reasoning and Planning in Large Language Modelscited by 1
  10. 2026Do Vision-Language Models Respect Contextual Integrity in Location Disclosure?cited by 1
  11. 2025CartoMapQA: A Fundamental Benchmark Dataset Evaluating Vision-Language Models on Cartographic Map Understandingcited by 0
  12. 2026TimeSpot: Benchmarking Geo-Temporal Understanding in Vision-Language Models in Real-World Settingscited by 0
6 more
  1. 2026The Global-Local loop: what is missing in bridging the gap between geospatial data from numerous communities?cited by 0
  2. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modescited by 0
  3. 2026GeoNatureAgent Benchmark: Benchmarking LLM Agents for Environmental Geospatial Analysis Across Frontier and Open-Weight Foundation Modelscited by 0
  4. 2026Perceive, Interact, Reason: Building Tool-Augmented Visual Agents for Spatial Reasoningcited by 0
  5. 2026Running the Gauntlet: Challenging Agentic Taskscited by 0
  6. 2026Geo-Spatial Concept Probing of Large Language Models: Abstraction, Compositionality, and Groundingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MapVerse: A Benchmark for Geospatial Question Answering on Diverse Real-World Maps
  2. 2026MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning
  3. 2026Network-based Spatial Context Retrieval for Open-weight LLMs: A Faithfulness Benchmark for Grounded Geographic Reasoning
  4. 2025Foundation Models for Geospatial Reasoning: Assessing Capabilities of Large Language Models in Understanding Geometries and Topological Spatial Relations
  5. 2025GeoChain: Multimodal Chain-of-Thought for Geographic Reasoning
  6. 2025SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence
  7. 2025Thinking on Maps: How Foundation Model Agents Explore, Remember, and Reason Map Environments
  8. 2025Where on Earth? A Vision-Language Benchmark for Probing Model Geolocation Skills Across Scales