mlsanityData to 7 Oct 2026

Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMs

arXiv 2311.14656 ↗First posted 24 Nov 2023cs.CV cs.AIcited by 34

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Multimodal large language models (MLLMs) have shown remarkable capabilities across a broad range of tasks but their knowledge and abilities in the geographic and geospatial domains are yet to be explored, despite potential wide-ranging benefits to navigation, environmental research, urban development, and disaster response. We conduct a series of experiments exploring various vision capabilities of MLLMs within these domains, particularly focusing on the frontier model GPT-4V, and benchmark its performance against open-source counterparts. Our methodology involves challenging these models with a small-scale geographic benchmark consisting of a suite of visual tasks, testing their abilities across a spectrum of complexity. The analysis uncovers not only where such models excel, including instances where they outperform humans, but also where they falter, providing a balanced view of their capabilities in the geographic domain. To enable the comparison and evaluation of future models, our benchmark will be publicly released.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 38 references that are in the map. First posted on arXiv 24 Nov 2023.

Builds on

27 of 38 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyond3 Oct
  2. 2023SATIN: A Multi-Task Metadataset for Classifying Satellite Imagery using Vision-Language Models23 Apr
  3. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2022Emergent Abilities of Large Language Modelscited by 2,483
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  12. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
12 more
  1. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  2. 2021LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentationcited by 259
  3. 2022Data Contamination: From Memorization to Exploitationcited by 234
  4. 2023On the Opportunities and Challenges of Foundation Models for Geospatial Artificial Intelligencecited by 88
  5. 2022SSL4EO-S12: A Large-Scale Multi-Modal, Multi-Temporal Dataset for Self-Supervised Learning in Earth Observationcited by 82
  6. 2023On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Drivingcited by 79
  7. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  8. 2023GPT4GEO: How a Language Model Sees the World's Geographycited by 68
  9. 2023GeoLLM: Extracting Geospatial Knowledge from Large Language Modelscited by 62
  10. 2023CXR-LLAVA: a multimodal large language model for interpreting chest X-ray imagescited by 37
  11. 2023An Early Evaluation of GPT-4V(ision)cited by 31
  12. 2023ChatGPT as a mapping assistant: A novel method to enrich maps with generative AI and content derived from street-level photographscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 10; 2026: 5.

34 papers.

Sort:

By area:

  1. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  2. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  3. 2025ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Modelscited by 56
  4. 2024SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretationcited by 47
  5. 2024DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Expertscited by 42
  6. 2024Good at captioning, bad at counting: Benchmarking GPT-4V on Earth observation datacited by 39
  7. 2024Towards Vision-Language Geo-Foundation Model: A Surveycited by 36
  8. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  9. 2024GeoLLM-Engine: A Realistic Environment for Building Geospatial Copilotscited by 27
  10. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  11. 2024Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Modelscited by 23
  12. 2024From Text to Transformation: A Comprehensive Review of Large Language Models' Versatilitycited by 19
22 more
  1. 2024CHOICE: Benchmarking the Remote Sensing Capabilities of Large Vision-Language Modelscited by 19
  2. 2024Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directionscited by 17
  3. 2024Survey of Large Multimodal Model Datasets, Application Categories and Taxonomycited by 17
  4. 2024Combining Observational Data and Language for Species Range Estimationcited by 12
  5. 2024Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks?cited by 12
  6. 2024STBench: Assessing the Ability of Large Language Models in Spatio-Temporal Analysiscited by 11
  7. 2024ActiView: Evaluating Active Perception Ability for Multimodal Large Language Modelscited by 11
  8. 2025A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Sciencecited by 11
  9. 2024INS-MMBench: A Comprehensive Benchmark for Evaluating LVLMs' Performance in Insurancecited by 6
  10. 2025PEACE: Empowering Geologic Map Holistic Understanding with MLLMscited by 6
  11. 2025MapIQ: Evaluating Multimodal Large Language Models for Map Question Answeringcited by 5
  12. 2024GRAB: A Challenging GRaph Analysis Benchmark for Large Multimodal Modelscited by 3
  13. 2025Geospatial Mechanistic Interpretability of Large Language Modelscited by 2
  14. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Modelscited by 2
  15. 2025Argus Inspection: Do Multimodal Large Language Models Possess the Eye of Panoptes?cited by 1
  16. 2025AutoMalDesc: Large-Scale Script Analysis for Cyber Threat Researchcited by 1
  17. 2025From Pixels to Urban Policy-Intelligence: Recovering Legacy Effects of Redlining with a Multimodal LLMcited by 0
  18. 2026Where Do Vision-Language Models Fail? World Scale Analysis for Image Geolocalizationcited by 0
  19. 2026PlanBench-V: A Spatial Planning Map Benchmark for Vision-Language Modelscited by 0
  20. 2026Textual Supervision Enhances Geospatial Representations in Vision-Language Modelscited by 0
  21. 2026Toward Governance-Aware Autonomous GIS: A Narrative Review of Ethical and Privacy Risks in LLM-Enabled GeoAIcited by 0
  22. 2026KilometerVision: A New Frontier for Large-Scale Spatial Intelligence in VLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Benchmarking Large and Small MLLMs
  2. 2025OmniGeo: Towards a Multimodal Large Language Models for Geospatial Artificial Intelligence
  3. 2025SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence
  4. 2023Are Large Language Models Geospatially Knowledgeable?
  5. 2026What We are Missing in Multimodal LLM Evaluation?
  6. 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipes
  7. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  8. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Survey