mlsanityData to 14 Oct 2026

CartoMapQA: A Fundamental Benchmark Dataset Evaluating Vision-Language Models on Cartographic Map Understanding

arXiv 2512.03558 ↗First posted 3 Dec 2025cs.CV cs.CLcited by 1

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

The rise of Visual-Language Models (LVLMs) has unlocked new possibilities for seamlessly integrating visual and textual information. However, their ability to interpret cartographic maps remains largely unexplored. In this paper, we introduce CartoMapQA, a benchmark specifically designed to evaluate LVLMs' understanding of cartographic maps through question-answering tasks. The dataset includes over 2000 samples, each composed of a cartographic map, a question (with open-ended or multiple-choice answers), and a ground-truth answer. These tasks span key low-, mid- and high-level map interpretation skills, including symbol recognition, embedded information extraction, scale interpretation, and route-based reasoning. Our evaluation of both open-source and proprietary LVLMs reveals persistent challenges: models frequently struggle with map-specific semantics, exhibit limited geospatial reasoning, and are prone to Optical Character Recognition (OCR)-related errors. By isolating these weaknesses, CartoMapQA offers a valuable tool for guiding future improvements in LVLM architectures. Ultimately, it supports the development of models better equipped for real-world applications that depend on robust and reliable map understanding, such as navigation, geographic search, and urban planning. Our source code and data are openly available to the research community at: https://github.com/ungquanghuy-kddi/CartoMapQA.git

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 45 references that are in the map. First posted on arXiv 3 Dec 2025.

Builds on

26 of 45 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023GeoChat: Grounded Large Vision-Language Model for Remote Sensing24 Nov

Other references · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  8. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  9. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  10. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  11. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  12. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
13 more
  1. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  2. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  3. 2020RSVQA: Visual Question Answering for Remote Sensing Datacited by 203
  4. 2023UrbanCLIP: Learning Text-enhanced Urban Region Profiling with Contrastive Language-Image Pretraining from the Webcited by 78
  5. 2023K2: A Foundation Language Model for Geoscience Knowledge Understanding and Utilizationcited by 69
  6. 2023GPT4GEO: How a Language Model Sees the World's Geographycited by 68
  7. 2023Evaluating Spatial Understanding of Large Language Modelscited by 66
  8. 2024V-IRL: Grounding Virtual Intelligence in Real Lifecited by 35
  9. 2023Are Large Language Models Geospatially Knowledgeable?cited by 30
  10. 2025Modeling and Analyzing Urban Networks and Amenities with OSMnxcited by 25
  11. 2024MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Modelscited by 21
  12. 2024Evaluating the Ability of Large Language Models to Reason about Cardinal Directionscited by 10
  13. 2024GRASP: A Grid-Based Benchmark for Evaluating Commonsense Spatial Reasoningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Mind the Spatial Gap: A Benchmark for Geospatial Reasoning and Constrained Spatial Editingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MAPWise: Evaluating Vision-Language Models for Advanced Map Queries
  2. 2025FRIEDA: Benchmarking Multi-Step Cartographic Reasoning in Vision-Language Models
  3. 2026MapVerse: A Benchmark for Geospatial Question Answering on Diverse Real-World Maps
  4. 2025MapIQ: Evaluating Multimodal Large Language Models for Map Question Answering
  5. 2025Can Large Vision Language Models Read Maps Like a Human?
  6. 2025GeoChain: Multimodal Chain-of-Thought for Geographic Reasoning
  7. 2025ReasonMap: Towards Fine-Grained Visual Reasoning from Transit Maps
  8. 2025How Well Do Vision--Language Models Understand Cities? A Comparative Study on Spatial Reasoning from Street-View Images