# EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialogues

arXiv: https://arxiv.org/abs/2412.15190 · HTML: https://mlsanity.com/paper/2412.15190/

## Key facts

- Cited by 85 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 53 of its 73 references that are in the map.
- First posted on arXiv 19 Dec 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 53)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models](https://mlsanity.com/paper/2311.17043/) (2023)
- [Language Is Not All You Need: Aligning Perception with Language Models](https://mlsanity.com/paper/2302.14045/) (2023)
- [RemoteCLIP: A Vision Language Foundation Model for Remote Sensing](https://mlsanity.com/paper/2306.11029/) (2023)
- [AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classification](https://mlsanity.com/paper/1608.05167/) (2016)
- [Object Detection in Optical Remote Sensing Images: A Survey and A New Benchmark](https://mlsanity.com/paper/1909.00133/) (2019)
- [Functional Map of the World](https://mlsanity.com/paper/1711.07846/) (2017)
- [GeoChat: Grounded Large Vision-Language Model for Remote Sensing](https://mlsanity.com/paper/2311.15826/) (2023)
- [InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model](https://mlsanity.com/paper/2401.16420/) (2024)
- [BigEarthNet: A Large-Scale Benchmark Archive For Remote Sensing Image Understanding](https://mlsanity.com/paper/1902.06148/) (2019)
- [RSGPT: A Remote Sensing Vision Language Model and Benchmark](https://mlsanity.com/paper/2307.15266/) (2023)
- [Object Detection in Aerial Images: A Large-Scale Benchmark and Challenges](https://mlsanity.com/paper/2102.12219/) (2021)
- [RSVQA: Visual Question Answering for Remote Sensing Data](https://mlsanity.com/paper/2003.07333/) (2020)
- [Exploring Models and Data for Remote Sensing Image Caption Generation](https://mlsanity.com/paper/1712.07835/) (2017)
- [VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning](https://mlsanity.com/paper/2102.10407/) (2021)
- [EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain](https://mlsanity.com/paper/2401.16822/) (2024)
- [FloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understanding](https://mlsanity.com/paper/2012.02951/) (2020)
- [SatlasPretrain: A Large-Scale Dataset for Remote Sensing Image Understanding](https://mlsanity.com/paper/2211.15660/) (2022)
- [Physically Grounded Vision-Language Models for Robotic Manipulation](https://mlsanity.com/paper/2309.02561/) (2023)
- [RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing](https://mlsanity.com/paper/2306.11300/) (2023)

## Built on by (most cited first, 30 of 85)

- [GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing](https://mlsanity.com/paper/2501.13925/) (2025)
- [DisasterM3: A Remote Sensing Vision-Language Dataset for Disaster Damage Assessment and Response](https://mlsanity.com/paper/2505.21089/) (2025)
- [When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruning](https://mlsanity.com/paper/2503.07588/) (2025)
- [Geo-R1: Improving Few-Shot Geospatial Referring Expression Understanding with Reinforcement Fine-Tuning](https://mlsanity.com/paper/2509.21976/) (2025)
- [DynamicVL: Benchmarking Multimodal Large Language Models for Dynamic City Understanding](https://mlsanity.com/paper/2505.21076/) (2025)
- [EarthMind: Leveraging Cross-Sensor Data for Advanced Earth Observation Interpretation with a Unified Multimodal LLM](https://mlsanity.com/paper/2506.01667/) (2025)
- [REOBench: Benchmarking Robustness of Earth Observation Foundation Models](https://mlsanity.com/paper/2505.16793/) (2025)
- [GeoZero: Incentivizing Reasoning from Scratch on Geospatial Scenes](https://mlsanity.com/paper/2511.22645/) (2025)
- [OpenEarthAgent: A Unified Framework for Tool-Augmented Geospatial Agents](https://mlsanity.com/paper/2602.17665/) (2026)
- [When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding](https://mlsanity.com/paper/2506.05551/) (2025)
- [Foundation Models in Remote Sensing: Evolving from Unimodality to Multimodality](https://mlsanity.com/paper/2603.00988/) (2026)
- [No One Knows the State of the Art in Geospatial Foundation Models](https://mlsanity.com/paper/2605.12678/) (2026)
- [EarthGPT-X: A Spatial MLLM for Multi-level Multi-Source Remote Sensing Imagery Understanding with Visual Prompting](https://mlsanity.com/paper/2504.12795/) (2025)
- [A Benchmark for Ultra-High-Resolution Remote Sensing MLLMs](https://mlsanity.com/paper/2512.17319/) (2025)
- [UniGeoSeg: Towards Unified Open-World Segmentation for Geospatial Scenes](https://mlsanity.com/paper/2511.23332/) (2025)
- [SegEarth-R2: Towards Comprehensive Language-guided Segmentation for Remote Sensing Images](https://mlsanity.com/paper/2512.20013/) (2025)
- [GeoEyes: On-Demand Visual Focusing for Evidence-Grounded Understanding of Ultra-High-Resolution Remote Sensing Imagery](https://mlsanity.com/paper/2602.14201/) (2026)
- [RemoteAgent: Bridging Vague Human Intents and Earth Observation with RL-based Agentic MLLMs](https://mlsanity.com/paper/2604.07765/) (2026)
- [InstructSAM: A Training-Free Framework for Instruction-Oriented Remote Sensing Object Recognition](https://mlsanity.com/paper/2505.15818/) (2025)
- [CangLing-KnowFlow: A Unified Knowledge-and-Flow-fused Agent for Comprehensive Remote Sensing Applications](https://mlsanity.com/paper/2512.15231/) (2025)
- [SkyMoE: A Vision-Language Foundation Model for Enhancing Geospatial Interpretation with Mixture of Experts](https://mlsanity.com/paper/2512.02517/) (2025)
- [TerraScope: Pixel-Grounded Visual Reasoning for Earth Observation](https://mlsanity.com/paper/2603.19039/) (2026)
- [VLRS-Bench: A Vision-Language Reasoning Benchmark for Remote Sensing](https://mlsanity.com/paper/2602.07045/) (2026)
- [GeoMMBench and GeoMMAgent: Toward Expert-Level Multimodal Intelligence in Geoscience and Remote Sensing](https://mlsanity.com/paper/2604.08896/) (2026)
- [Landsat30-AU: A Vision-Language Dataset for Australian Landsat Imagery](https://mlsanity.com/paper/2508.03127/) (2025)
- [VICoT-Agent: A Vision-Interleaved Chain-of-Thought Framework for Interpretable Multimodal Reasoning and Scalable Remote Sensing Analysis](https://mlsanity.com/paper/2511.20085/) (2025)
- [GeoViS: Geospatially Rewarded Visual Search for Remote Sensing Visual Grounding](https://mlsanity.com/paper/2512.02715/) (2025)
- [Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Tasks](https://mlsanity.com/paper/2606.10819/) (2026)
- [FUSAR-KLIP: Towards Multimodal Foundation Models for Remote Sensing](https://mlsanity.com/paper/2509.23927/) (2025)
- [Co-Training Vision Language Models for Remote Sensing Multi-task Learning](https://mlsanity.com/paper/2511.21272/) (2025)

## Abstract (arXiv)

Automated analysis of vast Earth observation data via interactive Vision-Language Models (VLMs) can unlock new opportunities for environmental monitoring, disaster response, and {resource management}. Existing generic VLMs do not perform well on Remote Sensing data, while the recent Geo-spatial VLMs remain restricted to a fixed resolution and few sensor modalities. In this paper, we introduce EarthDial, a conversational assistant specifically designed for Earth Observation (EO) data, transforming complex, multi-sensory Earth observations into interactive, natural language dialogues. EarthDial supports multi-spectral, multi-temporal, and multi-resolution imagery, enabling a wide range of remote sensing tasks, including classification, detection, captioning, question answering, visual reasoning, and visual grounding. To achieve this, we introduce an extensive instruction tuning dataset comprising over 11.11M instruction pairs covering RGB, Synthetic Aperture Radar (SAR), and multispectral modalities such as Near-Infrared (NIR) and infrared. Furthermore, EarthDial handles bi-temporal and multi-temporal sequence analysis for applications like change detection. Our extensive experimental results on 44 downstream datasets demonstrate that EarthDial outperforms existing generic and domain-specific models, achieving better generalization across various EO tasks. Our source codes and pre-trained models are at https://github.com/hiyamdebary/EarthDial.