MapReason-OSM: Can Vision-Language Models Make Graph-Verifiable Mobility Decisions from Street Maps ?
On the map: LLM reasoning → Spatial and embodied reasoning
Abstract (arXiv)
Vision-language models (VLMs) are increasingly used to read maps for logistics, delivery, and accessible navigation, where the output is an actionable decision (a route, a pin, a parking choice) that must respect the road network. Yet most map benchmarks grade free text or multiple-choice answers that cannot be verified against the underlying graph. We present MapReason-OSM, a benchmark and evaluation harness for graph-verifiable mobility decisions on self-rendered OpenStreetMap panels. We render fixed-style maps for ten U.S. downtowns at two aligned zoom scales, overlay a consistent marker grammar, and pair each panel with a hidden street graph and exact oracles, yielding 6,000 instances (12,000 panels across the two zooms) over 12 routing, facility-location, and visual disambiguation tasks. Models return structured decisions that we snap back to the graph and score for validity, legality, optimality, and constraint satisfaction, plus cross-zoom consistency. Across seven VLMs, models read maps and route simply but fail at graph cost reasoning (single-facility pin placement is near chance even for frontier reasoning models), and are frequently scale-inconsistent. We release the benchmark, harness, and deterministic generator. Code and data: https://github.com/Vi-Sri/mapreason-osm
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 8 of its 37 references that are in the map. First posted on arXiv 21 Jun 2026.
Builds on
8 of 37 references.
Closest ancestors · 1
Same direction, newest first
Other references · 7
- 2023GPT-4 Technical Reportcited by 18,396
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
- 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
- 2023GPT4GEO: How a Language Model Sees the World's Geographycited by 68
- 2025Can Large Vision Language Models Read Maps Like a Human?cited by 30
- 2024Evaluating Large Language Models on Spatial Tasks: A Multi-Task Benchmarking Studycited by 12
- 2026MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarioscited by 8
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Understanding the Geospatial Reasoning Capabilities of LLMs: A Trajectory Recovery Perspective
- 2026GROKE: Vision-Free Navigation Instruction Evaluation via Graph Reasoning on OpenStreetMap
- 2026PlanBench-V: A Spatial Planning Map Benchmark for Vision-Language Models
- 2024MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
- 2026m2sv: A Scalable Benchmark for Map-to-Street-View Spatial Reasoning
- 2025RoadBench: Benchmarking MLLMs on Fine-Grained Spatial Understanding and Reasoning under Urban Road Scenarios
- 2025Eyes Will Shut: A Vision-Based Next GPS Location Prediction Model by Reinforcement Learning from Visual Map Feed Back
- 2026TRIP-Evaluate: An Open Multimodal Benchmark for Evaluating Large Models in Transportation