mlsanityData to 7 Oct 2026

Towards Vision-Language Geo-Foundation Model: A Survey

arXiv 2406.09385 ↗First posted 13 Jun 2024cs.CVcited by 36

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision-Language Foundation Models (VLFMs) have made remarkable progress on various multimodal tasks, such as image captioning, image-text retrieval, visual question answering, and visual grounding. However, most methods rely on training with general image datasets, and the lack of geospatial data leads to poor performance on earth observation. Numerous geospatial image-text pair datasets and VLFMs fine-tuned on them have been proposed recently. These new approaches aim to leverage large-scale, multimodal geospatial data to build versatile intelligent models with diverse geo-perceptive capabilities, which we refer to as Vision-Language Geo-Foundation Models (VLGFMs). This paper thoroughly reviews VLGFMs, summarizing and analyzing recent developments in the field. In particular, we introduce the background and motivation behind the rise of VLGFMs, highlighting their unique research significance. Then, we systematically summarize the core technologies employed in VLGFMs, including data construction, model architectures, and applications of various multimodal geospatial tasks. Finally, we conclude with insights, issues, and discussions regarding future research directions. To the best of our knowledge, this is the first comprehensive literature review of VLGFMs. We keep tracing related works at https://github.com/zytx121/Awesome-VLGFM.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 75 of its 140 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

75 of 140 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024VHM: Versatile and Honest Vision Language Model for Remote Sensing Image Analysis29 Mar
  2. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model4 Feb
  3. 2024Good at captioning, bad at counting: Benchmarking GPT-4V on Earth observation data31 Jan
  4. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain30 Jan
  5. 2024SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Model18 Jan
  6. 2023SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing20 Dec
  7. 2023Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignment12 Dec
  8. 2023GeoChat: Grounded Large Vision-Language Model for Remote Sensing24 Nov
  9. 2023Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMs24 Nov
  10. 2023RSGPT: A Remote Sensing Vision Language Model and Benchmark28 Jul
  11. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing20 Jun
  12. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensing19 Jun
2 more
  1. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trends9 May
  2. 2023SATIN: A Multi-Task Metadataset for Classifying Satellite Imagery using Vision-Language Models23 Apr

Other references · 61

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2023Segment Anythingcited by 8,158
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
38 more
  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2022A ConvNet for the 2020scited by 3,545
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  7. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  8. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  10. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  11. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  12. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  13. 2017DOTA: A Large-scale Dataset for Object Detection in Aerial Imagescited by 622
  14. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  15. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  16. 2016AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classificationcited by 343
  17. 2017Functional Map of the Worldcited by 338
  18. 2017Road Extraction by Deep Residual U-Netcited by 333
  19. 2023SpectralGPT: Spectral Remote Sensing Foundation Modelcited by 250
  20. 2023SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagerycited by 202
  21. 2020RSVQA: Visual Question Answering for Remote Sensing Datacited by 197
  22. 2017Exploring Models and Data for Remote Sensing Image Caption Generationcited by 195
  23. 2019R3Det: Refined Single-Stage Detector with Feature Refinement for Rotating Objectcited by 193
  24. 2023GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localizationcited by 185
  25. 2020FloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understandingcited by 181
  26. 2021Rethinking Rotated Object Detection with Gaussian Wasserstein Distance Losscited by 156
  27. 2022Advancing Plain Vision Transformer Towards Remote Sensing Foundation Modelcited by 149
  28. 2023DiffusionSat: A Generative Foundation Model for Satellite Imagerycited by 130
  29. 2019IoU Loss for 2D/3D Object Detectioncited by 128
  30. 2022RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Datacited by 121
  31. 2022MMRotate: A Rotated Object Detection Benchmark using PyTorchcited by 115
  32. 2020Graph Convolutional Networks for Hyperspectral Image Classificationcited by 113
  33. 2017PatternNet: A Benchmark Dataset for Performance Evaluation of Remote Sensing Image Retrievalcited by 96
  34. 2023InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Languagecited by 96
  35. 2022Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrievalcited by 77
  36. 2023Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentationcited by 76
  37. 2023GPT4GEO: How a Language Model Sees the World's Geographycited by 68
  38. 2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 17; 2026: 11.

36 papers.

Sort:

By area:

  1. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 100
  2. 2025Text2Earth: Unlocking Text-driven Remote Sensing Image Generation with a Global-Scale Dataset and a Foundation Modelcited by 46
  3. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  4. 2024LHRS-Bot-Nova: Improved Multimodal Large Language Model for Remote Sensing Vision-Language Interpretationcited by 25
  5. 2024CHOICE: Benchmarking the Remote Sensing Capabilities of Large Vision-Language Modelscited by 19
  6. 2025Geo-R1: Improving Few-Shot Geospatial Referring Expression Understanding with Reinforcement Fine-Tuningcited by 19
  7. 2025A Survey on Remote Sensing Foundation Models: From Vision to Multimodalitycited by 18
  8. 2025Vision-Language Modeling Meets Remote Sensing: Models, Datasets and Perspectivescited by 18
  9. 2025ATRNet-STAR: A Large Dataset and Benchmark Towards Remote Sensing Object Recognition in the Wildcited by 12
  10. 2026Foundation Models in Remote Sensing: Evolving from Unimodality to Multimodalitycited by 12
  11. 2024REO-VLM: Transforming VLM to Meet Regression Challenges in Earth Observationcited by 9
  12. 2025A Simple Aerial Detection Baseline of Multimodal Language Modelscited by 9
24 more
  1. 2025AgriFM: A Multi-source Temporal Remote Sensing Foundation Model for Agriculture Mappingcited by 8
  2. 2026RemoteAgent: Bridging Vague Human Intents and Earth Observation with RL-based Agentic MLLMscited by 7
  3. 2024ImageRAG: Enhancing Ultra High Resolution Remote Sensing Imagery Analysis with ImageRAGcited by 6
  4. 2025GeoRSMLLM: A Multimodal Large Language Model for Vision-Language Tasks in Geoscience and Remote Sensingcited by 6
  5. 2026GRASP: Guided Region-Aware Sparse Prompting for Adapting MLLMs to Remote Sensingcited by 5
  6. 2024Harnessing Language for Coordination: A Framework and Benchmark for LLM-Driven Multi-Agent Controlcited by 4
  7. 2025DGTRSD & DGTRS-CLIP: A Dual-Granularity Remote Sensing Image-Text Dataset and Vision Language Foundation Model for Alignmentcited by 4
  8. 2025S2Vec: Self-Supervised Geospatial Embeddings for the Built Environmentcited by 4
  9. 2025A Genealogy of Foundation Models in Remote Sensingcited by 4
  10. 2025Urban-R1: Reinforced MLLMs Mitigate Geospatial Biases for Urban General Intelligencecited by 3
  11. 2025Co-Training Vision Language Models for Remote Sensing Multi-task Learningcited by 3
  12. 2025Satellites Reveal Mobility: A Commuting Origin-destination Flow Generator for Global Citiescited by 2
  13. 2025GeoProg3D: Compositional Visual Reasoning for City-Scale 3D Language Fieldscited by 2
  14. 2025FUSAR-KLIP: Towards Multimodal Foundation Models for Remote Sensingcited by 2
  15. 2026GeoAlignCLIP: Enhancing Fine-Grained Vision-Language Alignment in Remote Sensing via Multi-Granular Consistency Learningcited by 2
  16. 2026RemoteShield: Enable Robust Multimodal Large Language Models for Earth Observationcited by 2
  17. 2024General Vision Encoder Features as Guidance in Medical Image Registrationcited by 1
  18. 2024Aquila: A Hierarchically Aligned Visual-Language Model for Enhanced Remote Sensing Image Comprehensioncited by 1
  19. 2026AirSpatialBot: A Spatially-Aware Aerial Agent for Fine-Grained Vehicle Attribute Recognization and Retrievalcited by 1
  20. 2026FUSAR-GPT : A Spatiotemporal Feature-Embedded and Two-Stage Decoupled Visual Language Model for SAR Imagerycited by 1
  21. 2026RemoteZero: Geospatial Reasoning with Zero Labelscited by 1
  22. 2026Cryo-Bench: Benchmarking Foundation Models for Cryosphere Mappingcited by 0
  23. 2026Earth Science Foundation Models: From Perception to Reasoning and Discoverycited by 0
  24. 2026Multimodal Large Language Models for Remote Sensing Image Understanding: Domain-Specific or General-Purpose?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Tasks
  2. 2024VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding
  3. 2024Pushing the Limits of Vision-Language Models in Remote Sensing without Human Annotations
  4. 2024GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Tasks
  5. 2025Foundation Models for Remote Sensing: An Analysis of MLLMs for Object Localization
  6. 2026GeoPID: Decomposing and Steering Visual Information in Vision-Language Models
  7. 2024Advancements in Visual Language Models for Remote Sensing: Datasets, Capabilities, and Enhancement Techniques
  8. 2025A Vision Centric Remote Sensing Benchmark