mlsanityData to 14 Oct 2026

GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Tasks

arXiv 2411.19325 ↗First posted 28 Nov 2024cs.CVcited by 51

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

While numerous recent benchmarks focus on evaluating generic Vision-Language Models (VLMs), they do not effectively address the specific challenges of geospatial applications. Generic VLM benchmarks are not designed to handle the complexities of geospatial data, an essential component for applications such as environmental monitoring, urban planning, and disaster management. Key challenges in the geospatial domain include temporal change detection, large-scale object counting, tiny object detection, and understanding relationships between entities in remote sensing imagery. To bridge this gap, we present GEOBench-VLM, a comprehensive benchmark specifically designed to evaluate VLMs on geospatial tasks, including scene understanding, object counting, localization, fine-grained categorization, segmentation, and temporal analysis. Our benchmark features over 10,000 manually verified instructions and spanning diverse visual conditions, object types, and scales. We evaluate several state-of-the-art VLMs to assess performance on geospatial-specific challenges. The results indicate that although existing VLMs demonstrate potential, they face challenges when dealing with geospatial-specific tasks, highlighting the room for further improvements. Notably, the best-performing LLaVa-OneVision achieves only 41.7% accuracy on MCQs, slightly more than GPT-4o, which is approximately double the random guess performance. Our benchmark is publicly available at https://github.com/The-AI-Alliance/GEO-Bench-VLM .

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 60 of its 77 references that are in the map. First posted on arXiv 28 Nov 2024.

Builds on

60 of 77 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding18 Jun
  2. 2024SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understanding14 Jun
  3. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model4 Feb
  4. 2024Good at captioning, bad at counting: Benchmarking GPT-4V on Earth observation data31 Jan
  5. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain30 Jan
  6. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  7. 2024SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Model18 Jan
  8. 2023GeoChat: Grounded Large Vision-Language Model for Remote Sensing24 Nov
  9. 2023RSGPT: A Remote Sensing Vision Language Model and Benchmark28 Jul
  10. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensing19 Jun

Other references · 50

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  7. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  8. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  11. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
38 more
  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  2. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  4. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  5. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  6. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  7. 2017DOTA: A Large-scale Dataset for Object Detection in Aerial Imagescited by 623
  8. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  9. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  10. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 410
  11. 2019Object Detection in Optical Remote Sensing Images: A Survey and A New Benchmarkcited by 340
  12. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 292
  13. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  14. 2018DeepGlobe 2018: A Challenge to Parse the Earth through Satellite Imagescited by 277
  15. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  16. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 146
  17. 2021FAIR1M: A Benchmark Dataset for Fine-grained Object Recognition in High-Resolution Remote Sensing Imagerycited by 143
  18. 2021Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networkscited by 143
  19. 2019xBD: A Dataset for Assessing Building Damage from Satellite Imagerycited by 139
  20. 2021Anchor-free Oriented Proposal Generator for Object Detectioncited by 105
  21. 2024Benchmarking Vision Language Models for Cultural Understandingcited by 101
  22. 2017PatternNet: A Benchmark Dataset for Performance Evaluation of Remote Sensing Image Retrievalcited by 97
  23. 2016A Large Contextual Dataset for Classification, Detection and Counting of Cars with Deep Learningcited by 86
  24. 2023Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentationcited by 76
  25. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
  26. 2021Broaden the Vision: Geo-Diverse Visual Commonsense Reasoningcited by 59
  27. 2023RRSIS: Referring Remote Sensing Image Segmentationcited by 59
  28. 2023EarthVQA: Towards Queryable Earth via Relational Reasoning-Based Remote Sensing Visual Question Answeringcited by 52
  29. 2023A New Learning Paradigm for Foundation Model-based Remote Sensing Change Detectioncited by 47
  30. 2020Meta-Learning for Few-Shot Land Cover Classificationcited by 34
  31. 2020Synthesizing Optical and SAR Imagery From Land Cover Maps and Auxiliary Raster Datacited by 14
  32. 2023FireRisk: A Remote Sensing Dataset for Fire Risk Assessment with Benchmarks Using Supervised and Self-supervised Learningcited by 14
  33. 2024M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Modelscited by 13
  34. 2020AiRound and CV-BrCT: Novel Multi-View Datasets for Scene Classificationcited by 10
  35. 2024QuakeSet: A Dataset and Low-Resource Models to Monitor Earthquakes through Sentinel-1cited by 8
  36. 2024Panoptic Perception: A Novel Task and Fine-grained Dataset for Universal Remote Sensing Image Interpretationcited by 8
  37. 2023FPCD: An Open Aerial VHR Dataset for Farm Pond Change Detectioncited by 3
  38. 2024Estimating Earthquake Magnitude in Sentinel-1 Imagery via Rankingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 16; 2026: 35.

51 papers.

Sort:

By area:

  1. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 114
  2. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 43
  3. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  4. 2025Vision-Language Modeling Meets Remote Sensing: Models, Datasets and Perspectivescited by 18
  5. 2025EarthMind: Leveraging Cross-Sensor Data for Advanced Earth Observation Interpretation with a Unified Multimodal LLMcited by 18
  6. 2025Towards LLM Agents for Earth Observationcited by 17
  7. 2026VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchorscited by 13
  8. 2025A Benchmark for Ultra-High-Resolution Remote Sensing MLLMscited by 11
  9. 2025GeoAnalystBench: A GeoAI benchmark for assessing large language models for spatial analysis workflow and code generationcited by 10
  10. 2025SegEarth-R2: Towards Comprehensive Language-guided Segmentation for Remote Sensing Imagescited by 10
  11. 2026OpenEarth-Agent: From Tool Calling to Tool Creation for Open-Environment Earth Observationcited by 9
  12. 2026TerraScope: Pixel-Grounded Visual Reasoning for Earth Observationcited by 7
38 more
  1. 2026VLRS-Bench: A Vision-Language Reasoning Benchmark for Remote Sensingcited by 6
  2. 2026GeoMMBench and GeoMMAgent: Toward Expert-Level Multimodal Intelligence in Geoscience and Remote Sensingcited by 6
  3. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Modelscited by 3
  4. 2026OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Taskscited by 3
  5. 2026OSMDA: OpenStreetMap-based Domain Adaptation for Remote Sensing VLMscited by 3
  6. 2025MOAT: Evaluating LMMs for Capability Integration and Instruction Groundingcited by 2
  7. 2026Agentic AI for Remote Sensing: Technical Challenges and Research Directionscited by 2
  8. 2026Evaluating and Enhancing Negation Comprehension in Remote Sensing MLLMscited by 2
  9. 2025Leveraging ChatGPT's Multimodal Vision Capabilities to Rank Satellite Images by Poverty Level: Advancing Tools for Social Science Researchcited by 1
  10. 2025SAR Object Detection with Self-Supervised Pretraining and Curriculum-Aware Samplingcited by 1
  11. 2025GeoGrid-Bench: Can Foundation Models Understand Multimodal Gridded Geo-Spatial Data?cited by 1
  12. 2025Remote Sensing Image Intelligent Interpretation with the Language-Centered Perspective: Principles, Methods and Challengescited by 1
  13. 2025Geo3DVQA: Evaluating Vision-Language Models for 3D Geospatial Reasoning from Aerial Imagerycited by 1
  14. 2026Reasoning with Pixel-level Precision: QVLM Architecture and SQuID Dataset for Quantitative Geospatial Analyticscited by 1
  15. 2026DisasterVQA: A Visual Question Answering Benchmark Dataset for Disaster Scenescited by 1
  16. 2026Seeing Clearly, Reasoning Confidently: Plug-and-Play Remedies for Vision Language Model Blindnesscited by 1
  17. 2026UHR-Micro: Diagnosing and Mitigating the Resolution Illusion in Earth Observation VLMscited by 1
  18. 2026GeoX: Mastering Geospatial Reasoning Through Self-Play and Verifiable Rewardscited by 1
  19. 2026An Open-Source Benchmark and Baseline for Multi-temporal Referring Segmentationcited by 1
  20. 2026SIGNPOST-Bench: Benchmarking Text-Vision Conflict Resolution in Multimodal Large Language Modelscited by 1
  21. 2026VIVAS: Vitalizing Visual Perception in VLM Pre-training via Vision-language Unified Autoregressive Supervisioncited by 1
  22. 2026From Steering to Pedalling: Do Autonomous Driving VLMs Generalize to Cyclist-Assistive Spatial Perception and Planning?cited by 0
  23. 2026S2SServiceBench: A Multimodal Benchmark for Last-Mile S2S Climate Servicescited by 0
  24. 2026ThermEval: A Structured Benchmark for Evaluation of Vision-Language Models on Thermal Imagerycited by 0
  25. 2026TimeSpot: Benchmarking Geo-Temporal Understanding in Vision-Language Models in Real-World Settingscited by 0
  26. 2026The Global-Local loop: what is missing in bridging the gap between geospatial data from numerous communities?cited by 0
  27. 2026SDGBiasBench: Benchmarking and Mitigating Vision--Language Models' Biases in Sustainable Development Goalscited by 0
  28. 2026Beyond Appearance: Can Multimodal Large Language Models Exploit Vertical Structure for Remote Sensing Natural Scene Understanding?cited by 0
  29. 2026Dive into the Scene: Breaking the Perceptual Bottleneck in Vision-Language Decision Making via Focus Plan Generationcited by 0
  30. 2026MonoIR-RS: Infrared Remote Sensing Vision-Language Learning with CLIP and VLM Adaptationcited by 0
  31. 2026More with Less: a Large Scale Remote Sensing VLM with a Simple Recipecited by 0
  32. 2026RRS-10K: A Multitask Vision-Language Model Benchmark for Rare Remote Sensing Image Interpretationcited by 0
  33. 2026EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazardscited by 0
  34. 2026GeBDA: Building Damage Assessment as Text-Based Sequence Predictioncited by 0
  35. 2026LogiScope-VQA: Benchmarking Vision-Language Models for Logistics Hazard Identification in Industrial Scenarioscited by 0
  36. 2026Earth-Agent-Pro: Towards Real-World Full-Chain Earth Observation with Agentscited by 0
  37. 2026One Domain, Many Tongues: Composing Domain and Language LoRAs for Cross-Lingual Remote-Sensing MLLMs without Paired Datacited by 0
  38. 2026USAI-Quant: A Quantitative Reasoning Benchmark for Vision-Language Models in Built Environmentscited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025GeoRSMLLM: A Multimodal Large Language Model for Vision-Language Tasks in Geoscience and Remote Sensing
  2. 2026EarthSpatialBench: Benchmarking Spatial Reasoning Capabilities of Multimodal LLMs on Earth Imagery
  3. 2025Foundation Models for Remote Sensing: An Analysis of MLLMs for Object Localization
  4. 2026GeoRC: A Benchmark for Geolocation Reasoning Chains
  5. 2024Remote Sensing SpatioTemporal Vision-Language Models: A Comprehensive Survey
  6. 2025Where on Earth? A Vision-Language Benchmark for Probing Model Geolocation Skills Across Scales
  7. 2026Where Do Vision-Language Models Fail? World Scale Analysis for Image Geolocalization
  8. 2025Efficient Few-Shot Learning in Remote Sensing: Fusing Vision and Vision-Language Models