mlsanityData to 7 Oct 2026

EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain

arXiv 2401.16822 ↗First posted 30 Jan 2024cs.CVcited by 181

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Multi-modal large language models (MLLMs) have demonstrated remarkable success in vision and visual-language tasks within the natural image domain. Owing to the significant diversities between the natural and remote sensing (RS) images, the development of MLLMs in the RS domain is still in the infant stage. To fill the gap, a pioneer MLLM named EarthGPT integrating various multi-sensor RS interpretation tasks uniformly is proposed in this paper for universal RS image comprehension. In EarthGPT, three key techniques are developed including a visual-enhanced perception mechanism, a cross-modal mutual comprehension approach, and a unified instruction tuning method for multi-sensor multi-task in the RS domain. More importantly, a dataset named MMRS-1M featuring large-scale multi-sensor multi-modal RS instruction-following is constructed, comprising over 1M image-text pairs based on 34 existing diverse RS datasets and including multi-sensor images such as optical, synthetic aperture radar (SAR), and infrared. The MMRS-1M dataset addresses the drawback of MLLMs on RS expert knowledge and stimulates the development of MLLMs in the RS domain. Extensive experiments are conducted, demonstrating the EarthGPT's superior performance in various RS visual interpretation tasks compared with the other specialist models and MLLMs, proving the effectiveness of the proposed EarthGPT and offering a versatile paradigm for open-set reasoning tasks.

Key facts. Cited by 181 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 75 of its 122 references that are in the map. First posted on arXiv 30 Jan 2024.

Builds on

75 of 122 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding21 Dec
  2. 2023GeoChat: Grounded Large Vision-Language Model for Remote Sensing24 Nov
  3. 2023RSGPT: A Remote Sensing Vision Language Model and Benchmark28 Jul
  4. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensing19 Jun

Other references · 71

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
38 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  8. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  10. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  11. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  12. 2019A Comprehensive Survey on Transfer Learningcited by 1,154
  13. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  14. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  15. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  16. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  17. 2023Instruction Tuning with GPT-4cited by 676
  18. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  19. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  20. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  21. 2015An Introduction to Convolutional Neural Networkscited by 510
  22. 2022SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagerycited by 402
  23. 2016AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classificationcited by 343
  24. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  25. 2022Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learningcited by 302
  26. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292
  27. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  28. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  29. 2020Detection and Tracking Meet Drones Challengecited by 261
  30. 2021LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentationcited by 259
  31. 2021Oriented R-CNN for Object Detectioncited by 230
  32. 2021Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Datacited by 214
  33. 2021Object Detection in Aerial Images: A Large-Scale Benchmark and Challengescited by 202
  34. 2020RSVQA: Visual Question Answering for Remote Sensing Datacited by 197
  35. 2017Exploring Models and Data for Remote Sensing Image Caption Generationcited by 195
  36. 2019iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Imagescited by 195
  37. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 194
  38. 2019R3Det: Refined Single-Stage Detector with Feature Refinement for Rotating Objectcited by 193

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 31; 2025: 75; 2026: 75.

181 papers.

Sort:

By area:

  1. 2024VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understandingcited by 116
  2. 2024SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understandingcited by 101
  3. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  4. 2024EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialoguescited by 81
  5. 2024TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Datacited by 69
  6. 2024GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Taskscited by 50
  7. 2025XLRS-Bench: Could Your Multimodal LLMs Understand Extremely Large Ultra-High-Resolution Remote Sensing Imagery?cited by 50
  8. 2024MetaEarth: A Generative Foundation Model for Global-Scale Remote Sensing Image Generationcited by 41
  9. 2025GeoLLaVA-8K: Scaling Remote-Sensing Multimodal Large Language Models to 8K Resolutioncited by 41
  10. 2024Good at captioning, bad at counting: Benchmarking GPT-4V on Earth observation datacited by 39
  11. 2025RemoteSAM: Towards Segment Anything for Earth Observationcited by 38
  12. 2024Towards Vision-Language Geo-Foundation Model: A Surveycited by 36
38 more
  1. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  2. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  3. 2025TESSERA: Temporal Embeddings of Surface Spectra for Earth Representation and Analysiscited by 35
  4. 2024GeoSynth: Contextually-Aware High-Resolution Satellite Image Synthesiscited by 32
  5. 2024On the Foundations of Earth and Climate Foundation Modelscited by 31
  6. 2024EarthMarker: A Visual Prompting Multi-modal Large Language Model for Remote Sensingcited by 29
  7. 2024Diffusion Models Meet Remote Sensing: Principles, Methods, and Perspectivescited by 27
  8. 2024GeoLLM-Engine: A Realistic Environment for Building Geospatial Copilotscited by 27
  9. 2024SARATR-X: Toward Building A Foundation Model for SAR Target Recognitioncited by 25
  10. 2024An LLM Agent for Automatic Geospatial Data Analysiscited by 21
  11. 2025Multi-Agent Geospatial Copilots for Remote Sensing Workflowscited by 20
  12. 2025RingMo-Agent: A Unified Remote Sensing Foundation Model for Multi-Platform and Multi-Modal Reasoningcited by 19
  13. 2025Geo-R1: Improving Few-Shot Geospatial Referring Expression Understanding with Reinforcement Fine-Tuningcited by 19
  14. 2025A Survey on Remote Sensing Foundation Models: From Vision to Multimodalitycited by 18
  15. 2025Vision-Language Modeling Meets Remote Sensing: Models, Datasets and Perspectivescited by 18
  16. 2025EarthMind: Leveraging Cross-Sensor Data for Advanced Earth Observation Interpretation with a Unified Multimodal LLMcited by 18
  17. 2025RefDrone: A Challenging Benchmark for Referring Expression Comprehension in Drone Scenescited by 17
  18. 2024TAR3D: Creating High-Quality 3D Assets via Next-Part Predictioncited by 16
  19. 2025RingMoE: Mixture-of-Modality-Experts Multi-Modal Foundation Models for Universal Remote Sensing Image Interpretationcited by 16
  20. 2024RS-MoE: A Vision-Language Model with Mixture of Experts for Remote Sensing Image Captioning and Visual Question Answeringcited by 15
  21. 2024Popeye: A Unified Visual-Language Model for Multi-Source Ship Detection from Remote Sensing Imagerycited by 14
  22. 2025CVBench: Benchmarking Cross-Video Synergies for Complex Multimodal Reasoningcited by 14
  23. 2024On Domain-Adaptive Post-Training for Multimodal Large Language Modelscited by 13
  24. 2025LISAT: Language-Instructed Segmentation Assistant for Satellite Imagerycited by 13
  25. 2025REOBench: Benchmarking Robustness of Earth Observation Foundation Modelscited by 13
  26. 2025GeoZero: Incentivizing Reasoning from Scratch on Geospatial Scenescited by 13
  27. 2025ATRNet-STAR: A Large Dataset and Benchmark Towards Remote Sensing Object Recognition in the Wildcited by 12
  28. 2026Foundation Models in Remote Sensing: Evolving from Unimodality to Multimodalitycited by 12
  29. 2025EarthGPT-X: A Spatial MLLM for Multi-level Multi-Source Remote Sensing Imagery Understanding with Visual Promptingcited by 11
  30. 2025Look Where It Matters: Training-Free Ultra-HR Remote Sensing VQA via Adaptive Zoom Searchcited by 11
  31. 2025A Benchmark for Ultra-High-Resolution Remote Sensing MLLMscited by 11
  32. 2025GeoPix: Multi-Modal Large Language Model for Pixel-level Image Understanding in Remote Sensingcited by 10
  33. 2025Quality-Driven Curation of Remote Sensing Vision-Language Data via Learned Scoring Modelscited by 10
  34. 2025Geo-OLM: Enabling Sustainable Earth Observation Studies with Cost-Efficient Open Language Models & State-Driven Workflowscited by 10
  35. 2025UniGeoSeg: Towards Unified Open-World Segmentation for Geospatial Scenescited by 10
  36. 2025SegEarth-R2: Towards Comprehensive Language-guided Segmentation for Remote Sensing Imagescited by 10
  37. 2024RS-GPT4V: A Unified Multimodal Instruction-Following Dataset for Remote Sensing Image Understandingcited by 9
  38. 2024REO-VLM: Transforming VLM to Meet Regression Challenges in Earth Observationcited by 9

50 most cited of 181 All 181 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Model
  2. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model
  3. 2024LHRS-Bot-Nova: Improved Multimodal Large Language Model for Remote Sensing Vision-Language Interpretation
  4. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trends
  5. 2024Multilingual Vision-Language Pre-training for the Remote Sensing Domain
  6. 2024RSUniVLM: A Unified Vision Language Model for Remote Sensing via Granularity-oriented Mixture of Experts
  7. 2026MMLGNet: Cross-Modal Alignment of Remote Sensing Data using CLIP
  8. 2026Exploring the Potential of Contrastive Language-Image Pre-training for Multi-Source Remote Sensing Data