# Can Large Vision Language Models Read Maps Like a Human?

arXiv: https://arxiv.org/abs/2503.14607 · HTML: https://mlsanity.com/paper/2503.14607/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 38 of its 73 references that are in the map.
- First posted on arXiv 18 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 38)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [Multimodal Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2302.00923/) (2023)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models](https://mlsanity.com/paper/2407.07895/) (2024)
- [DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](https://mlsanity.com/paper/2412.10302/) (2024)
- [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models](https://mlsanity.com/paper/2402.12289/) (2024)
- [Hallucination of Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2404.18930/) (2024)
- [SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning](https://mlsanity.com/paper/2307.06135/) (2023)
- [NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models](https://mlsanity.com/paper/2305.16986/) (2023)
- [EMMA: End-to-End Multimodal Model for Autonomous Driving](https://mlsanity.com/paper/2410.23262/) (2024)
- [GRAPE: Generalizing Robot Policy via Preference Alignment](https://mlsanity.com/paper/2411.19309/) (2024)
- [Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillings](https://mlsanity.com/paper/2305.02317/) (2023)
- [MapNav: A Novel Memory Representation via Annotated Semantic Maps for Vision-and-Language Navigation](https://mlsanity.com/paper/2502.13451/) (2025)
- [End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answering](https://mlsanity.com/paper/2411.05755/) (2024)
- [AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Driving](https://mlsanity.com/paper/2412.15206/) (2024)
- [Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization](https://mlsanity.com/paper/2502.13146/) (2025)
- [Diffusion of Thoughts: Chain-of-Thought Reasoning in Diffusion Language Models](https://mlsanity.com/paper/2402.07754/) (2024)

## Built on by (most cited first, 30 of 30)

- [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.16419/) (2025)
- [UniOcc: A Unified Benchmark for Occupancy Forecasting and Prediction in Autonomous Driving](https://mlsanity.com/paper/2503.24381/) (2025)
- [SAFEFLOW: A Principled Protocol for Trustworthy and Transactional Autonomous Agent Systems](https://mlsanity.com/paper/2506.07564/) (2025)
- [ReasonMap: Towards Fine-Grained Visual Reasoning from Transit Maps](https://mlsanity.com/paper/2505.18675/) (2025)
- [RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning](https://mlsanity.com/paper/2510.02240/) (2025)
- [Embodied Web Agents: Bridging Physical-Digital Realms for Integrated Agent Intelligence](https://mlsanity.com/paper/2506.15677/) (2025)
- [Generative AI for Autonomous Driving: Frontiers and Opportunities](https://mlsanity.com/paper/2505.08854/) (2025)
- [MapIQ: Evaluating Multimodal Large Language Models for Map Question Answering](https://mlsanity.com/paper/2507.11625/) (2025)
- [Demystifying the Visual Quality Paradox in Multimodal Large Language Models](https://mlsanity.com/paper/2506.15645/) (2025)
- [RDD: Retrieval-Based Demonstration Decomposer for Planner Alignment in Long-Horizon Tasks](https://mlsanity.com/paper/2510.14968/) (2025)
- [FRIEDA: Benchmarking Multi-Step Cartographic Reasoning in Vision-Language Models](https://mlsanity.com/paper/2512.08016/) (2025)
- [MapTrace: Scalable Data Generation for Route Tracing on Maps](https://mlsanity.com/paper/2512.19609/) (2025)
- [MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphs](https://mlsanity.com/paper/2602.18600/) (2026)
- [The Role of Open-Source LLMs in Shaping the Future of GeoAI](https://mlsanity.com/paper/2504.17833/) (2025)
- [VLM@school -- Evaluation of AI image understanding on German middle school knowledge](https://mlsanity.com/paper/2506.11604/) (2025)
- [Q-Router: Agentic Video Quality Assessment with Expert Model Routing and Artifact Localization](https://mlsanity.com/paper/2510.08789/) (2025)
- [OmniMapBench: Benchmarking Visual-Centric Reasoning on Diverse Map Documents](https://mlsanity.com/paper/2607.09068/) (2026)
- [Does RLVR Extend Reasoning Boundaries? Investigating Capability Expansion in Vision-Language Models](https://mlsanity.com/paper/2511.00710/) (2025)
- [Mil-SCORE: Benchmarking Long-Context Geospatial Reasoning and Planning in Large Language Models](https://mlsanity.com/paper/2601.21826/) (2026)
- [TraversalBench: Challenging Paths to Follow for Vision Language Models](https://mlsanity.com/paper/2604.10999/) (2026)
- [360CityArena: A Realistic Virtual Urban Navigation Benchmark for Embodied Agents](https://mlsanity.com/paper/2608.08814/) (2026)
- [ICDAR2026 Competition on Multimodal Reasoning over Documents in Multiple Domains](https://mlsanity.com/paper/2609.25055/) (2026)
- [SVG Decomposition for Enhancing Large Multimodal Models Visualization Comprehension: A Study with Floor Plans](https://mlsanity.com/paper/2511.03478/) (2025)
- [LAVQA: A Latency-Aware Visual Question Answering Framework for Shared Autonomy in Self-Driving Vehicles](https://mlsanity.com/paper/2511.11840/) (2025)
- [TransitLM: A Large-Scale Dataset and Benchmark for Map-Free Transit Route Generation](https://mlsanity.com/paper/2605.22355/) (2026)
- [PGT: Procedurally Generated Tasks for improving visual grounding in MLLMs](https://mlsanity.com/paper/2605.23883/) (2026)
- [Perceive, Interact, Reason: Building Tool-Augmented Visual Agents for Spatial Reasoning](https://mlsanity.com/paper/2606.12830/) (2026)
- [MapReason-OSM: Can Vision-Language Models Make Graph-Verifiable Mobility Decisions from Street Maps ?](https://mlsanity.com/paper/2606.22597/) (2026)
- [Toward AI-Friendly Cartography: Understanding How Color Design Influences Foundation Model Spatial Reasoning on Sequential Choropleth Maps](https://mlsanity.com/paper/2608.15736/) (2026)
- [AirLog: Store-Level Indoor Life Logging Made Easy](https://mlsanity.com/paper/2609.31864/) (2026)

## Abstract (arXiv)

In this paper, we introduce MapBench-the first dataset specifically designed for human-readable, pixel-based map-based outdoor navigation, curated from complex path finding scenarios. MapBench comprises over 1600 pixel space map path finding problems from 100 diverse maps. In MapBench, LVLMs generate language-based navigation instructions given a map image and a query with beginning and end landmarks. For each map, MapBench provides Map Space Scene Graph (MSSG) as an indexing data structure to convert between natural language and evaluate LVLM-generated results. We demonstrate that MapBench significantly challenges state-of-the-art LVLMs both zero-shot prompting and a Chain-of-Thought (CoT) augmented reasoning framework that decomposes map navigation into sequential cognitive processes. Our evaluation of both open-source and closed-source LVLMs underscores the substantial difficulty posed by MapBench, revealing critical limitations in their spatial reasoning and structured decision-making capabilities. We release all the code and dataset in https://github.com/taco-group/MapBench.