mlsanityData to 7 Oct 2026

Good at captioning, bad at counting: Benchmarking GPT-4V on Earth observation data

arXiv 2401.17600 ↗First posted 31 Jan 2024cs.CL cs.AI cs.CVcited by 39

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Large Vision-Language Models (VLMs) have demonstrated impressive performance on complex tasks involving visual input with natural language instructions. However, it remains unclear to what extent capabilities on natural images transfer to Earth observation (EO) data, which are predominantly satellite and aerial images less common in VLM training data. In this work, we propose a comprehensive benchmark to gauge the progress of VLMs toward being useful tools for EO data by assessing their abilities on scene understanding, localization and counting, and change detection tasks. Motivated by real-world applications, our benchmark includes scenarios like urban monitoring, disaster relief, land use, and conservation. We discover that, although state-of-the-art VLMs like GPT-4V possess extensive world knowledge that leads to strong performance on open-ended tasks like location understanding and image captioning, their poor spatial reasoning limits usefulness on object localization and counting tasks. Our benchmark will be made publicly available at https://vleo.danielz.ch/ and on Hugging Face at https://huggingface.co/collections/mit-ei/vleo-benchmark-datasets-65b789b0466555489cce0d70 for easy model evaluation.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 60 references that are in the map. First posted on arXiv 31 Jan 2024.

Builds on

44 of 60 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain30 Jan
  2. 2024SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Model18 Jan
  3. 2023GeoChat: Grounded Large Vision-Language Model for Remote Sensing24 Nov
  4. 2023Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMs24 Nov
  5. 2023RSGPT: A Remote Sensing Vision Language Model and Benchmark28 Jul

Other references · 39

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  9. 2018Datasheets for Datasetscited by 1,371
  10. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  12. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
27 more
  1. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  2. 2021Robust fine-tuning of zero-shot modelscited by 785
  3. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  4. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  5. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  6. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  7. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  8. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  9. 2019Object Detection in Optical Remote Sensing Images: A Survey and A New Benchmarkcited by 339
  10. 2017Functional Map of the Worldcited by 338
  11. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 305
  12. 2019BigEarthNet: A Large-Scale Benchmark Archive For Remote Sensing Image Understandingcited by 231
  13. 2017Exploring Models and Data for Remote Sensing Image Caption Generationcited by 195
  14. 2019xBD: A Dataset for Assessing Building Damage from Satellite Imagerycited by 138
  15. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  16. 2022RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Datacited by 121
  17. 2022MMRotate: A Rotated Object Detection Benchmark using PyTorchcited by 115
  18. 2017PatternNet: A Benchmark Dataset for Performance Evaluation of Remote Sensing Image Retrievalcited by 96
  19. 2016A Large Contextual Dataset for Classification, Detection and Counting of Cars with Deep Learningcited by 86
  20. 2021Extending the WILDS Benchmark for Unsupervised Adaptationcited by 85
  21. 2021Continental-Scale Building Detection from High Resolution Satellite Imagerycited by 50
  22. 2023Text2Seg: Remote Sensing Image Semantic Segmentation via Text-Guided Visual Foundation Modelscited by 44
  23. 2020Meta-Learning for Few-Shot Land Cover Classificationcited by 34
  24. 2022xView3-SAR: Detecting Dark Fishing Activity Using Synthetic Aperture Radar Imagerycited by 25
  25. 2023On the Promises and Challenges of Multimodal Foundation Models for Geographical, Environmental, Agricultural, and Urban Planning Applicationscited by 15
  26. 2023Multimodal Foundation Models for Zero-shot Animal Species Recognition in Camera Trap Imagescited by 13
  27. 2020Counting Cows: Tracking Illegal Cattle Ranching From High-Resolution Satellite Imagerycited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 17; 2026: 12.

39 papers.

Sort:

By area:

  1. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  2. 2024VHM: Versatile and Honest Vision Language Model for Remote Sensing Image Analysiscited by 84
  3. 2024EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialoguescited by 81
  4. 2024Understanding the Limits of Vision Language Models Through the Lens of the Binding Problemcited by 71
  5. 2024PANGAEA: A Global and Inclusive Benchmark for Geospatial Foundation Modelscited by 62
  6. 2024GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Taskscited by 50
  7. 2024Towards Vision-Language Geo-Foundation Model: A Surveycited by 36
  8. 2024LHRS-Bot-Nova: Improved Multimodal Large Language Model for Remote Sensing Vision-Language Interpretationcited by 25
  9. 2025Vision-Language Modeling Meets Remote Sensing: Models, Datasets and Perspectivescited by 18
  10. 2025EarthMind: Leveraging Cross-Sensor Data for Advanced Earth Observation Interpretation with a Unified Multimodal LLMcited by 18
  11. 2025Visual symbolic mechanisms: Emergent symbol processing in vision language modelscited by 12
  12. 2025A Benchmark for Ultra-High-Resolution Remote Sensing MLLMscited by 11
27 more
  1. 2025InstructSAM: A Training-Free Framework for Instruction-Oriented Remote Sensing Object Recognitioncited by 8
  2. 2025GeoRSMLLM: A Multimodal Large Language Model for Vision-Language Tasks in Geoscience and Remote Sensingcited by 6
  3. 2026GeoMMBench and GeoMMAgent: Toward Expert-Level Multimodal Intelligence in Geoscience and Remote Sensingcited by 6
  4. 2025Pedestrian Intention Prediction via Vision-Language Foundation Modelscited by 5
  5. 2025[De|Re]constructing VLMs' Reasoning in Countingcited by 5
  6. 2025Foundation Models for Remote Sensing: An Analysis of MLLMs for Object Localizationcited by 3
  7. 2025Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuningcited by 3
  8. 2024Multimodal LLMs Struggle with Basic Visual Network Analysis: a VNA Benchmarkcited by 2
  9. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Modelscited by 2
  10. 2026SITUATE -- Synthetic Object Counting Dataset for VLM trainingcited by 2
  11. 2024Meaningless is better: hashing bias-inducing words in LLM prompts improves performance in logical reasoning and statistical learningcited by 1
  12. 2025Leveraging ChatGPT's Multimodal Vision Capabilities to Rank Satellite Images by Poverty Level: Advancing Tools for Social Science Researchcited by 1
  13. 2025BYO-Eval: Build Your Own Dataset for Fine-Grained Visual Assessment of Multimodal Language Modelscited by 1
  14. 2025Zero-Shot Multi-Spectral Learning: Reimagining a Generalist Multimodal Gemini 2.5 Model for Remote Sensing Applicationscited by 1
  15. 2026Reasoning with Pixel-level Precision: QVLM Architecture and SQuID Dataset for Quantitative Geospatial Analyticscited by 1
  16. 2026From Horizontal to Rotated: Cross-View Object Geo-Localization with Orientation Awarenesscited by 1
  17. 2026Beyond 2D Matching: A Unified Single-Stage Framework for Geometry-Aware Cross-View Object Geo-Localizationcited by 1
  18. 2024Can GPT-4 Help Detect Quit Vaping Intentions? An Exploration of Automatic Data Annotation Approachcited by 0
  19. 2025Decoding Neighborhood Environments with Large Language Modelscited by 0
  20. 2025GeoGrid-Bench: Can Foundation Models Understand Multimodal Gridded Geo-Spatial Data?cited by 0
  21. 2026Vision-Language Agents for Interactive Forest Change Analysiscited by 0
  22. 2026Forest-Chat: Adapting Vision-Language Agents for Interactive Forest Change Analysiscited by 0
  23. 2026TimeSpot: Benchmarking Geo-Temporal Understanding in Vision-Language Models in Real-World Settingscited by 0
  24. 2026Unlocking Multi-Spectral Data for Multi-Modal Models with Guided Inputs and Chain-of-Thought Reasoningcited by 0
  25. 2026Multimodal Large Language Models for Remote Sensing Image Understanding: Domain-Specific or General-Purpose?cited by 0
  26. 2026Earth-Agent-Pro: Towards Real-World Full-Chain Earth Observation with Agentscited by 0
  27. 2026Hob-VL: A Benchmark for Visually Grounded Boolean Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Tasks
  2. 2024VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding
  3. 2025Where on Earth? A Vision-Language Benchmark for Probing Model Geolocation Skills Across Scales
  4. 2026More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe
  5. 2026VLRS-Bench: A Vision-Language Reasoning Benchmark for Remote Sensing
  6. 2026LongEarth-R1: Benchmarking and Aligning Vision-Language Models for Long-Horizon Earth Observation Reasoning
  7. 2024TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Data
  8. 2024CHOICE: Benchmarking the Remote Sensing Capabilities of Large Vision-Language Models