mlsanityData to 7 Oct 2026

From Pixels to Prose: Advancing Multi-Modal Language Models for Remote Sensing

arXiv 2411.05826 ↗First posted 5 Nov 2024cs.CV cs.AI cs.LGcited by 3

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Remote sensing has evolved from simple image acquisition to complex systems capable of integrating and processing visual and textual data. This review examines the development and application of multi-modal language models (MLLMs) in remote sensing, focusing on their ability to interpret and describe satellite imagery using natural language. We cover the technical underpinnings of MLLMs, including dual-encoder architectures, Transformer models, self-supervised and contrastive learning, and cross-modal integration. The unique challenges of remote sensing data--varying spatial resolutions, spectral richness, and temporal changes--are analyzed for their impact on MLLM performance. Key applications such as scene description, object detection, change detection, text-to-image retrieval, image-to-text generation, and visual question answering are discussed to demonstrate their relevance in environmental monitoring, urban planning, and disaster response. We review significant datasets and resources supporting the training and evaluation of these models. Challenges related to computational demands, scalability, data quality, and domain adaptation are highlighted. We conclude by proposing future research directions and technological advancements to further enhance MLLM utility in remote sensing.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 68 references that are in the map. First posted on arXiv 5 Nov 2024.

Builds on

36 of 68 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024PriorCLIP: Visual Prior Guided Vision-Language Model for Remote Sensing Image-Text Retrieval16 May
  2. 2024Mind the Modality Gap: Towards a Remote Sensing Vision-Language Model via Cross-modal Alignment15 Feb
  3. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model4 Feb
  4. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain30 Jan
  5. 2024SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Model18 Jan
  6. 2023RSGPT: A Remote Sensing Vision Language Model and Benchmark28 Jul

Other references · 30

  1. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  2. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  3. 2023SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagerycited by 202
  4. 2020RSVQA: Visual Question Answering for Remote Sensing Datacited by 197
  5. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
  6. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  7. 2022RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Datacited by 121
  8. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
  9. 2024Foundation Models for Remote Sensing and Earth Observation: A Surveycited by 66
  10. 2022Multimodal Fusion Transformer for Remote Sensing Image Classificationcited by 61
  11. 2024Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Drivingcited by 52
  12. 2024Understanding Information Storage and Transfer in Multi-modal Large Language Modelscited by 48
18 more
  1. 2022Remote Sensing Cross-Modal Text-Image Retrieval Based on Global and Local Informationcited by 40
  2. 2024Changen2: Multi-Temporal Remote Sensing Generative Change Foundation Modelcited by 37
  3. 2022ResFormer: Scaling ViTs with Multi-Resolution Trainingcited by 34
  4. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 33
  5. 2024Large Language Models for Captioning and Retrieving Remote Sensing Imagescited by 30
  6. 2024Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attackscited by 24
  7. 2023MarineGPT: Unlocking Secrets of Ocean to the Publiccited by 23
  8. 2024ChatEarthNet: A Global-Scale Image-Text Dataset Empowering Vision-Language Geo-Foundation Modelscited by 23
  9. 2022Multi-Modal Fusion Transformer for Visual Question Answering in Remote Sensingcited by 13
  10. 2022Deep Unsupervised Contrastive Hashing for Large-Scale Cross-Modal Text-Image Retrieval in Remote Sensingcited by 10
  11. 2023LiT-4-RSVQA: Lightweight Transformer-based Visual Question Answering in Remote Sensingcited by 10
  12. 2024Knowledge-aware Text-Image Retrieval for Remote Sensing Imagescited by 9
  13. 2024LLM-Optic: Unveiling the Capabilities of Large Language Models for Universal Visual Groundingcited by 7
  14. 2024Towards a multimodal framework for remote sensing image change retrieval and captioningcited by 5
  15. 2024Transcending Fusion: A Multi-Scale Alignment Method for Remote Sensing Image-Text Retrievalcited by 4
  16. 2020Unsupervised Denoising for Satellite Imagery using Wavelet Subband CycleGANcited by 1
  17. 2023Joint multi-modal Self-Supervised pre-training in Remote Sensing: Application to Methane Source Classificationcited by 1
  18. 2023Context-PEFT: Efficient Multi-Modal, Multi-Task Fine-Tuningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026GRASP: Guided Region-Aware Sparse Prompting for Adapting MLLMs to Remote Sensingcited by 5
  2. 2025Remote Sensing Image Intelligent Interpretation with the Language-Centered Perspective: Principles, Methods and Challengescited by 1
  3. 2026Group Contrastive Learning for Weakly Paired Multimodal Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Multimodal Large Language Models for Remote Sensing Image Understanding: Domain-Specific or General-Purpose?
  2. 2025Vision-Language Modeling Meets Remote Sensing: Models, Datasets and Perspectives
  3. 2024Advancements in Visual Language Models for Remote Sensing: Datasets, Capabilities, and Enhancement Techniques
  4. 2024Remote Sensing SpatioTemporal Vision-Language Models: A Comprehensive Survey
  5. 2026SkyNative: A Native Multimodal Architecture for Remote Sensing Vision-Language Understanding
  6. 2025GeoRSMLLM: A Multimodal Large Language Model for Vision-Language Tasks in Geoscience and Remote Sensing
  7. 2025Enhancing Remote Sensing Vision-Language Models Through MLLM and LLM-Based High-Quality Image-Text Dataset Generation
  8. 2026OSMDA: OpenStreetMap-based Domain Adaptation for Remote Sensing VLMs