mlsanityData to 14 Oct 2026

CHOICE: Benchmarking the Remote Sensing Capabilities of Large Vision-Language Models

arXiv 2411.18145 ↗First posted 27 Nov 2024cs.CVcited by 22

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

The rapid advancement of Large Vision-Language Models (VLMs), both general-domain models and those specifically tailored for remote sensing, has demonstrated exceptional perception and reasoning capabilities in Earth observation tasks. However, a benchmark for systematically evaluating their capabilities in this domain is still lacking. To bridge this gap, we propose CHOICE, an extensive benchmark designed to objectively evaluate the hierarchical remote sensing capabilities of VLMs. Focusing on 2 primary capability dimensions essential to remote sensing: perception and reasoning, we further categorize 6 secondary dimensions and 23 leaf tasks to ensure a well-rounded assessment coverage. CHOICE guarantees the quality of all 10,507 problems through a rigorous process of data collection from 50 globally distributed cities, question construction and quality control. The newly curated data and the format of multiple-choice questions with definitive answers allow for an objective and straightforward performance assessment. Our evaluation of 3 proprietary and 21 open-source VLMs highlights their critical limitations within this specialized context. We hope that CHOICE will serve as a valuable resource and offer deeper insights into the challenges and potential of VLMs in the field of remote sensing. We will release CHOICE at [this https URL](https://github.com/ShawnAn-WHU/CHOICE).

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 39 of its 58 references that are in the map. First posted on arXiv 27 Nov 2024.

Builds on

39 of 58 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  8. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  9. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  10. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  11. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  12. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
27 more
  1. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  2. 2017DOTA: A Large-scale Dataset for Object Detection in Aerial Imagescited by 623
  3. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 363
  4. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 355
  5. 2016AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classificationcited by 347
  6. 2019Object Detection in Optical Remote Sensing Images: A Survey and A New Benchmarkcited by 340
  7. 2023GeoChat: Grounded Large Vision-Language Model for Remote Sensingcited by 328
  8. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  9. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 222
  10. 2021Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Datacited by 215
  11. 2023RSGPT: A Remote Sensing Vision Language Model and Benchmarkcited by 207
  12. 2020RSVQA: Visual Question Answering for Remote Sensing Datacited by 203
  13. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 193
  14. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 151
  15. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Modelcited by 138
  16. 2022RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Datacited by 123
  17. 2024VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understandingcited by 121
  18. 2021Anchor-free Oriented Proposal Generator for Object Detectioncited by 105
  19. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93
  20. 2020Remote Sensing Image Scene Classification Meets Deep Learning: Challenges, Methods, Benchmarks, and Opportunitiescited by 71
  21. 2023GPT4GEO: How a Language Model Sees the World's Geographycited by 68
  22. 2024MTP: Advancing Remote Sensing Foundation Model via Multi-Task Pretrainingcited by 66
  23. 2023EarthVQA: Towards Queryable Earth via Relational Reasoning-Based Remote Sensing Visual Question Answeringcited by 52
  24. 2024Towards Vision-Language Geo-Foundation Model: A Surveycited by 36
  25. 2023Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMscited by 34
  26. 2024LHRS-Bot-Nova: Improved Multimodal Large Language Model for Remote Sensing Vision-Language Interpretationcited by 25
  27. 2024Learning without Exact Guidance: Updating Large-scale High-resolution Land Cover Maps from Low-resolution Historical Labelscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 18.

22 papers.

Sort:

By area:

  1. 2025Geo-R1: Improving Few-Shot Geospatial Referring Expression Understanding with Reinforcement Fine-Tuningcited by 20
  2. 2025GeoZero: Incentivizing Reasoning from Scratch on Geospatial Scenescited by 14
  3. 2025SegEarth-R2: Towards Comprehensive Language-guided Segmentation for Remote Sensing Imagescited by 10
  4. 2026OpenEarth-Agent: From Tool Calling to Tool Creation for Open-Environment Earth Observationcited by 9
  5. 2026VLRS-Bench: A Vision-Language Reasoning Benchmark for Remote Sensingcited by 6
  6. 2026GeoMMBench and GeoMMAgent: Toward Expert-Level Multimodal Intelligence in Geoscience and Remote Sensingcited by 6
  7. 2026GeoAgentBench: A Dynamic Execution Benchmark for Tool-Augmented Agents in Spatial Analysiscited by 5
  8. 2026GeoAlignCLIP: Enhancing Fine-Grained Vision-Language Alignment in Remote Sensing via Multi-Granular Consistency Learningcited by 2
  9. 2025Remote Sensing Image Intelligent Interpretation with the Language-Centered Perspective: Principles, Methods and Challengescited by 1
  10. 2026RSHallu: Dual-Mode Hallucination Evaluation for Remote-Sensing Multimodal Large Language Models with Domain-Tailored Mitigationcited by 1
  11. 2026UHR-Micro: Diagnosing and Mitigating the Resolution Illusion in Earth Observation VLMscited by 1
  12. 2026Geometric Consistency Protocol for Foundation Model Features in Multi-View Satellite Imagerycited by 1
10 more
  1. 2026Is One-Shot In-Context Learning Helpful for Data Selection in Task-Specific Fine-Tuning of Multimodal LLMs?cited by 0
  2. 2026Can Multimodal Large Language Models Truly Understand Small Objects?cited by 0
  3. 2026SenseBench: A Benchmark for Remote Sensing Low-Level Visual Perception and Description in Large Vision-Language Modelscited by 0
  4. 2026Beyond Appearance: Can Multimodal Large Language Models Exploit Vertical Structure for Remote Sensing Natural Scene Understanding?cited by 0
  5. 2026RRS-10K: A Multitask Vision-Language Model Benchmark for Rare Remote Sensing Image Interpretationcited by 0
  6. 2026GeBDA: Building Damage Assessment as Text-Based Sequence Predictioncited by 0
  7. 2026Earth-Agent-Pro: Towards Real-World Full-Chain Earth Observation with Agentscited by 0
  8. 2026RSure-Agent: Reliable Use of Tool Observations for Remote Sensing Agentscited by 0
  9. 2026Hallucination-aware Faithful Reasoning in Remote Sensingcited by 0
  10. 2026ReDiS: Reasoning-Diagnosis Self-Evolution for Multimodal Reasoning in Remote Sensingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Tasks
  2. 2026More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe
  3. 2025Vision-Language Modeling Meets Remote Sensing: Models, Datasets and Perspectives
  4. 2024Advancements in Visual Language Models for Remote Sensing: Datasets, Capabilities, and Enhancement Techniques
  5. 2025A Benchmark for Ultra-High-Resolution Remote Sensing MLLMs
  6. 2025Towards Faithful Reasoning in Remote Sensing: A Perceptually-Grounded GeoSpatial Chain-of-Thought for Vision-Language Models
  7. 2026Multimodal Large Language Models for Remote Sensing Image Understanding: Domain-Specific or General-Purpose?
  8. 2025Foundation Models for Remote Sensing: An Analysis of MLLMs for Object Localization