mlsanityData to 7 Oct 2026

EarthMarker: A Visual Prompting Multi-modal Large Language Model for Remote Sensing

arXiv 2407.13596 ↗First posted 18 Jul 2024cs.CVcited by 29

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Recent advances in prompt learning have allowed users to interact with artificial intelligence (AI) tools in multi-turn dialogue, enabling an interactive understanding of images. However, it is difficult and inefficient to deliver information in complicated remote sensing (RS) scenarios using plain language instructions alone, which would severely hinder deep comprehension of the latent content in imagery. Besides, existing prompting strategies in natural scenes are hard to apply to interpret the RS data due to significant domain differences. To address these challenges, the first visual prompting-based multi-modal large language model (MLLM) named EarthMarker is proposed in the RS domain. EarthMarker is capable of interpreting RS imagery at the image, region, and point levels by levering visual prompts (i.e., boxes and points). Specifically, a shared visual encoding method is developed to establish the spatial pattern interpretation relationships between the multi-scale representations of input images and various visual prompts. Subsequently, the mixed visual-spatial representations are associated with language instructions to construct joint prompts, enabling the interpretation of intricate content of RS imagery. Furthermore, to bridge the domain gap between natural and RS data, and effectively transfer domain-level knowledge from natural scenes to the RS domain, a cross-domain learning strategy is developed to facilitate the RS imagery understanding. In addition, to tackle the lack of RS visual prompting data, a dataset named RSVP featuring multi-modal multi-granularity visual prompts instruction-following is constructed. Our code and dataset are available at https://github.com/wivizhang/EarthMarker.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 68 of its 105 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

68 of 105 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model4 Feb
  2. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain30 Jan
  3. 2024SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Model18 Jan
  4. 2023GeoChat: Grounded Large Vision-Language Model for Remote Sensing24 Nov
  5. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing20 Jun

Other references · 63

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2023Segment Anythingcited by 8,158
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
38 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  2. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  7. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  8. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  9. 2016Modeling Context in Referring Expressionscited by 1,054
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  11. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  12. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  13. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  14. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  15. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  16. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  17. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  18. 2016AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classificationcited by 343
  19. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  20. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  21. 2015What's the Point: Semantic Segmentation with Point Supervisioncited by 297
  22. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  23. 2020Detection and Tracking Meet Drones Challengecited by 261
  24. 2022Visual Prompting via Image Inpaintingcited by 248
  25. 2021Object Detection in Aerial Images: A Large-Scale Benchmark and Challengescited by 202
  26. 2017Exploring Models and Data for Remote Sensing Image Caption Generationcited by 195
  27. 2019iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Imagescited by 195
  28. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  29. 2023Semantic-SAM: Segment and Recognize Anything at Any Granularitycited by 185
  30. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 182
  31. 2023RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Modelcited by 180
  32. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 160
  33. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 158
  34. 2021FAIR1M: A Benchmark Dataset for Fine-grained Object Recognition in High-Resolution Remote Sensing Imagerycited by 143
  35. 2022PromptDet: Towards Open-vocabulary Detection using Uncurated Imagescited by 133
  36. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  37. 2022RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Datacited by 121
  38. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 98

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 13; 2026: 14.

29 papers.

Sort:

By area:

  1. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  2. 2025Falcon: A Remote Sensing Vision-Language Foundation Model (Technical Report)cited by 28
  3. 2024Remote Sensing SpatioTemporal Vision-Language Models: A Comprehensive Surveycited by 22
  4. 2025PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularitycited by 21
  5. 2025Vision-Language Modeling Meets Remote Sensing: Models, Datasets and Perspectivescited by 18
  6. 2025EarthMind: Leveraging Cross-Sensor Data for Advanced Earth Observation Interpretation with a Unified Multimodal LLMcited by 18
  7. 2024TAR3D: Creating High-Quality 3D Assets via Next-Part Predictioncited by 16
  8. 2025REOBench: Benchmarking Robustness of Earth Observation Foundation Modelscited by 13
  9. 2025EarthGPT-X: A Spatial MLLM for Multi-level Multi-Source Remote Sensing Imagery Understanding with Visual Promptingcited by 11
  10. 2025GeoPix: Multi-Modal Large Language Model for Pixel-level Image Understanding in Remote Sensingcited by 10
  11. 2025UniGeoSeg: Towards Unified Open-World Segmentation for Geospatial Scenescited by 10
  12. 2026GeoEyes: On-Demand Visual Focusing for Evidence-Grounded Understanding of Ultra-High-Resolution Remote Sensing Imagerycited by 9
17 more
  1. 2026RemoteAgent: Bridging Vague Human Intents and Earth Observation with RL-based Agentic MLLMscited by 7
  2. 2025DescribeEarth: Describe Anything for Remote Sensing Imagescited by 5
  3. 2026GRASP: Guided Region-Aware Sparse Prompting for Adapting MLLMs to Remote Sensingcited by 5
  4. 2026Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Taskscited by 3
  5. 2025GeoProg3D: Compositional Visual Reasoning for City-Scale 3D Language Fieldscited by 2
  6. 2026GeoMeld: Toward Semantically Grounded Foundation Models for Remote Sensingcited by 2
  7. 2026RemoteShield: Enable Robust Multimodal Large Language Models for Earth Observationcited by 2
  8. 2026Semantic-Geometric Dual Compression: Training-Free Visual Token Reduction for Ultra-High-Resolution Remote Sensing Understandingcited by 1
  9. 2026Agentic AI for Remote Sensing: Technical Challenges and Research Directionscited by 1
  10. 2025A Vision-Language Framework for Multispectral Scene Representation Using Language-Grounded Featurescited by 0
  11. 2025UniDet-D: A Unified Dynamic Spectral Attention Model for Object Detection under Adverse Weatherscited by 0
  12. 2026Continual Vision-Language Learning for Remote Sensing: Benchmarking and Analysiscited by 0
  13. 2026Evaluating Remote Sensing Image Captions Beyond Metric Biasescited by 0
  14. 2026Referring Multiple Regions with Large Multimodal Models via Contextual Latent Steeringcited by 0
  15. 2026More with Less: a Large Scale Remote Sensing VLM with a Simple Recipecited by 0
  16. 2026UniEvo-RS: Omni-Prompt Unified Remote Sensing Segmentation with Representative Exemplar-Driven Prototype Evolutioncited by 0
  17. 2026Earth-Agent-Pro: Towards Real-World Full-Chain Earth Observation with Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024From Pixels to Prose: Advancing Multi-Modal Language Models for Remote Sensing
  2. 2026Multimodal Large Language Models for Remote Sensing Image Understanding: Domain-Specific or General-Purpose?
  3. 2024LHRS-Bot-Nova: Improved Multimodal Large Language Model for Remote Sensing Vision-Language Interpretation
  4. 2025GeoRSMLLM: A Multimodal Large Language Model for Vision-Language Tasks in Geoscience and Remote Sensing
  5. 2025A Vision Centric Remote Sensing Benchmark
  6. 2026bi-modal textual prompt learning for vision-language models in remote sensing
  7. 2024Advancements in Visual Language Models for Remote Sensing: Datasets, Capabilities, and Enhancement Techniques
  8. 2023RSGPT: A Remote Sensing Vision Language Model and Benchmark