mlsanityData to 14 Oct 2026

EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialogues

arXiv 2412.15190 ↗First posted 19 Dec 2024cs.CVcited by 85

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Automated analysis of vast Earth observation data via interactive Vision-Language Models (VLMs) can unlock new opportunities for environmental monitoring, disaster response, and {resource management}. Existing generic VLMs do not perform well on Remote Sensing data, while the recent Geo-spatial VLMs remain restricted to a fixed resolution and few sensor modalities. In this paper, we introduce EarthDial, a conversational assistant specifically designed for Earth Observation (EO) data, transforming complex, multi-sensory Earth observations into interactive, natural language dialogues. EarthDial supports multi-spectral, multi-temporal, and multi-resolution imagery, enabling a wide range of remote sensing tasks, including classification, detection, captioning, question answering, visual reasoning, and visual grounding. To achieve this, we introduce an extensive instruction tuning dataset comprising over 11.11M instruction pairs covering RGB, Synthetic Aperture Radar (SAR), and multispectral modalities such as Near-Infrared (NIR) and infrared. Furthermore, EarthDial handles bi-temporal and multi-temporal sequence analysis for applications like change detection. Our extensive experimental results on 44 downstream datasets demonstrate that EarthDial outperforms existing generic and domain-specific models, achieving better generalization across various EO tasks. Our source codes and pre-trained models are at https://github.com/hiyamdebary/EarthDial.

Key facts. Cited by 85 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 73 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

53 of 73 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024CDChat: A Large Multimodal Model for Remote Sensing Change Description24 Sep
  2. 2024VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding18 Jun
  3. 2024SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understanding14 Jun
  4. 2024VG4D: Vision-Language Model Goes 4D Video Recognition17 Apr
  5. 2024VHM: Versatile and Honest Vision Language Model for Remote Sensing Image Analysis29 Mar
  6. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model4 Feb
  7. 2024Good at captioning, bad at counting: Benchmarking GPT-4V on Earth observation data31 Jan
  8. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain30 Jan
  9. 2024SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Model18 Jan
  10. 2023SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing20 Dec
  11. 2023GeoChat: Grounded Large Vision-Language Model for Remote Sensing24 Nov
  12. 2023Physically Grounded Vision-Language Models for Robotic Manipulation5 Sep
3 more
  1. 2023RSGPT: A Remote Sensing Vision Language Model and Benchmark28 Jul
  2. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing20 Jun
  3. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensing19 Jun

Other references · 38

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  8. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  11. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  12. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
26 more
  1. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  2. 2016AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classificationcited by 347
  3. 2019Object Detection in Optical Remote Sensing Images: A Survey and A New Benchmarkcited by 340
  4. 2017Functional Map of the Worldcited by 338
  5. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  6. 2019BigEarthNet: A Large-Scale Benchmark Archive For Remote Sensing Image Understandingcited by 234
  7. 2021Object Detection in Aerial Images: A Large-Scale Benchmark and Challengescited by 204
  8. 2020RSVQA: Visual Question Answering for Remote Sensing Datacited by 203
  9. 2017Exploring Models and Data for Remote Sensing Image Caption Generationcited by 197
  10. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 191
  11. 2020FloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understandingcited by 181
  12. 2022SatlasPretrain: A Large-Scale Dataset for Remote Sensing Image Understandingcited by 167
  13. 2021FAIR1M: A Benchmark Dataset for Fine-grained Object Recognition in High-Resolution Remote Sensing Imagerycited by 143
  14. 2022RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Datacited by 123
  15. 2017PatternNet: A Benchmark Dataset for Performance Evaluation of Remote Sensing Image Retrievalcited by 97
  16. 2022Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrievalcited by 77
  17. 2024Change-Agent: Towards Interactive Comprehensive Remote Sensing Change Interpretation and Analysiscited by 63
  18. 2024OmniSat: Self-Supervised Modality Fusion for Earth Observationcited by 63
  19. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  20. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  21. 2022HIT-UAV: A high-altitude infrared thermal dataset for Unmanned Aerial Vehicle-based object detectioncited by 34
  22. 2024Meerkat: Audio-Visual Large Language Model for Grounding in Space and Timecited by 26
  23. 2019So2Sat LCZ42: A Benchmark Dataset for Global Local Climate Zones Classificationcited by 24
  24. 2023OpenAnnotate3D: Open-Vocabulary Auto-Labeling System for Multi-modal 3D Datacited by 14
  25. 2024QuakeSet: A Dataset and Low-Resource Models to Monitor Earthquakes through Sentinel-1cited by 8
  26. 2024Made to Order: Discovering monotonic temporal changes via self-supervised video orderingcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 31; 2026: 54.

85 papers.

Sort:

By area:

  1. 2025GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensingcited by 57
  2. 2025DisasterM3: A Remote Sensing Vision-Language Dataset for Disaster Damage Assessment and Responsecited by 36
  3. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  4. 2025Geo-R1: Improving Few-Shot Geospatial Referring Expression Understanding with Reinforcement Fine-Tuningcited by 20
  5. 2025DynamicVL: Benchmarking Multimodal Large Language Models for Dynamic City Understandingcited by 18
  6. 2025EarthMind: Leveraging Cross-Sensor Data for Advanced Earth Observation Interpretation with a Unified Multimodal LLMcited by 18
  7. 2025REOBench: Benchmarking Robustness of Earth Observation Foundation Modelscited by 15
  8. 2026OpenEarthAgent: A Unified Framework for Tool-Augmented Geospatial Agentscited by 15
  9. 2025GeoZero: Incentivizing Reasoning from Scratch on Geospatial Scenescited by 14
  10. 2025When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understandingcited by 12
  11. 2026Foundation Models in Remote Sensing: Evolving from Unimodality to Multimodalitycited by 12
  12. 2026No One Knows the State of the Art in Geospatial Foundation Modelscited by 12
38 more
  1. 2025EarthGPT-X: A Spatial MLLM for Multi-level Multi-Source Remote Sensing Imagery Understanding with Visual Promptingcited by 11
  2. 2025A Benchmark for Ultra-High-Resolution Remote Sensing MLLMscited by 11
  3. 2025UniGeoSeg: Towards Unified Open-World Segmentation for Geospatial Scenescited by 10
  4. 2025SegEarth-R2: Towards Comprehensive Language-guided Segmentation for Remote Sensing Imagescited by 10
  5. 2026GeoEyes: On-Demand Visual Focusing for Evidence-Grounded Understanding of Ultra-High-Resolution Remote Sensing Imagerycited by 9
  6. 2025InstructSAM: A Training-Free Framework for Instruction-Oriented Remote Sensing Object Recognitioncited by 8
  7. 2025CangLing-KnowFlow: A Unified Knowledge-and-Flow-fused Agent for Comprehensive Remote Sensing Applicationscited by 8
  8. 2026RemoteAgent: Bridging Vague Human Intents and Earth Observation with RL-based Agentic MLLMscited by 8
  9. 2025SkyMoE: A Vision-Language Foundation Model for Enhancing Geospatial Interpretation with Mixture of Expertscited by 7
  10. 2026TerraScope: Pixel-Grounded Visual Reasoning for Earth Observationcited by 7
  11. 2026VLRS-Bench: A Vision-Language Reasoning Benchmark for Remote Sensingcited by 6
  12. 2026GeoMMBench and GeoMMAgent: Toward Expert-Level Multimodal Intelligence in Geoscience and Remote Sensingcited by 6
  13. 2025Landsat30-AU: A Vision-Language Dataset for Australian Landsat Imagerycited by 4
  14. 2025VICoT-Agent: A Vision-Interleaved Chain-of-Thought Framework for Interpretable Multimodal Reasoning and Scalable Remote Sensing Analysiscited by 4
  15. 2025GeoViS: Geospatially Rewarded Visual Search for Remote Sensing Visual Groundingcited by 4
  16. 2026Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Taskscited by 4
  17. 2025FUSAR-KLIP: Towards Multimodal Foundation Models for Remote Sensingcited by 3
  18. 2025Co-Training Vision Language Models for Remote Sensing Multi-task Learningcited by 3
  19. 2025FUSE-RSVLM: Feature Fusion Vision-Language Model for Remote Sensingcited by 3
  20. 2026OSMDA: OpenStreetMap-based Domain Adaptation for Remote Sensing VLMscited by 3
  21. 2026RS-Claw: Progressive Active Tool Exploration via Hierarchical Skill Trees for Remote Sensing Agentscited by 3
  22. 2026DisasterInsight: A Building-Centric Benchmark for Evaluating Vision--Language Models in Disaster Responsecited by 2
  23. 2026RS-WorldModel: a Unified Model for Remote Sensing Understanding and Future Sense Forecastingcited by 2
  24. 2026BigEarthNet.txt: A Large-Scale Multi-Sensor Image-Text Dataset and Benchmark for Earth Observationcited by 2
  25. 2026GeoMeld: Toward Semantically Grounded Foundation Models for Remote Sensingcited by 2
  26. 2026RemoteShield: Enable Robust Multimodal Large Language Models for Earth Observationcited by 2
  27. 2026Agentic AI for Remote Sensing: Technical Challenges and Research Directionscited by 2
  28. 2025Remote Sensing Image Intelligent Interpretation with the Language-Centered Perspective: Principles, Methods and Challengescited by 1
  29. 2025A Multimodal Conversational Assistant for the Characterization of Agricultural Plots from Geospatial Open Datacited by 1
  30. 2025REMSA: Foundation Model Selection for Remote Sensing via a Constraint-Aware Agentcited by 1
  31. 2025GeoDiT: A Diffusion-based Vision-Language Model for Geospatial Understandingcited by 1
  32. 2026DisasterVQA: A Visual Question Answering Benchmark Dataset for Disaster Scenescited by 1
  33. 2026FUSAR-GPT : A Spatiotemporal Feature-Embedded and Two-Stage Decoupled Visual Language Model for SAR Imagerycited by 1
  34. 2026UAV traffic scene understanding: A regulation embedded multi-modal network and a unified benchmarkcited by 1
  35. 2026Semantic-Geometric Dual Compression: Training-Free Visual Token Reduction for Ultra-High-Resolution Remote Sensing Understandingcited by 1
  36. 2026UHR-BAT: Budget-Aware Token Compression Vision-Language model for Ultra-High-Resolution Remote Sensingcited by 1
  37. 2026EO-Gym: A Multimodal, Interactive Environment for Earth Observation Agentscited by 1
  38. 2026Geospatial-Temporal Sensemaking of Remote Sensing Activity Detections with Multimodal Large Language Modelcited by 1

50 most cited of 85 All 85 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Remote Sensing SpatioTemporal Vision-Language Models: A Comprehensive Survey
  2. 2025ChatENV: An Interactive Vision-Language Model for Sensor-Guided Environmental Monitoring and Scenario Simulation
  3. 2024TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Data
  4. 2024From Pixels to Prose: Advancing Multi-Modal Language Models for Remote Sensing
  5. 2024ChangeChat: An Interactive Model for Remote Sensing Change Analysis via Multimodal Instruction Tuning
  6. 2026OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Tasks
  7. 2026GeoNLI - A Natural Language Interpreter for Satellite Imagery
  8. 2025GeoRSMLLM: A Multimodal Large Language Model for Vision-Language Tasks in Geoscience and Remote Sensing