# RemoteSAM: Towards Segment Anything for Earth Observation

arXiv: https://arxiv.org/abs/2505.18022 · HTML: https://mlsanity.com/paper/2505.18022/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 89 references that are in the map.
- First posted on arXiv 23 May 2025.

## Builds on (most cited first, 30 of 63)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [DOTA: A Large-scale Dataset for Object Detection in Aerial Images](https://mlsanity.com/paper/1711.10398/) (2017)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery](https://mlsanity.com/paper/2207.08051/) (2022)
- [RemoteCLIP: A Vision Language Foundation Model for Remote Sensing](https://mlsanity.com/paper/2306.11029/) (2023)
- [Pix2seq: A Language Modeling Framework for Object Detection](https://mlsanity.com/paper/2109.10852/) (2021)
- [LAVT: Language-Aware Vision Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2112.02244/) (2021)
- [Object Detection in Optical Remote Sensing Images: A Survey and A New Benchmark](https://mlsanity.com/paper/1909.00133/) (2019)
- [Functional Map of the World](https://mlsanity.com/paper/1711.07846/) (2017)
- [GeoChat: Grounded Large Vision-Language Model for Remote Sensing](https://mlsanity.com/paper/2311.15826/) (2023)
- [Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning](https://mlsanity.com/paper/2212.14532/) (2022)
- [SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models](https://mlsanity.com/paper/2311.07575/) (2023)
- [LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2110.08733/) (2021)
- [GRES: Generalized Referring Expression Segmentation](https://mlsanity.com/paper/2306.00968/) (2023)
- [Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data](https://mlsanity.com/paper/2103.16607/) (2021)
- [iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Images](https://mlsanity.com/paper/1905.12886/) (2019)
- [EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain](https://mlsanity.com/paper/2401.16822/) (2024)
- [SCLIP: Rethinking Self-Attention for Dense Vision-Language Inference](https://mlsanity.com/paper/2312.01597/) (2023)
- [A Unified Sequence Interface for Vision Tasks](https://mlsanity.com/paper/2206.07669/) (2022)
- [RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing](https://mlsanity.com/paper/2306.11300/) (2023)
- [Advancing Plain Vision Transformer Towards Remote Sensing Foundation Model](https://mlsanity.com/paper/2208.03987/) (2022)
- [Referring Transformer: A One-step Approach to Multi-task Visual Grounding](https://mlsanity.com/paper/2106.03089/) (2021)
- [LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model](https://mlsanity.com/paper/2402.02544/) (2024)

## Built on by (most cited first, 30 of 39)

- [RemoteReasoner: Towards Unifying Geospatial Reasoning Workflow](https://mlsanity.com/paper/2507.19280/) (2025)
- [Geo-R1: Improving Few-Shot Geospatial Referring Expression Understanding with Reinforcement Fine-Tuning](https://mlsanity.com/paper/2509.21976/) (2025)
- [SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Images](https://mlsanity.com/paper/2512.08730/) (2025)
- [UniGeoSeg: Towards Unified Open-World Segmentation for Geospatial Scenes](https://mlsanity.com/paper/2511.23332/) (2025)
- [RemoteAgent: Bridging Vague Human Intents and Earth Observation with RL-based Agentic MLLMs](https://mlsanity.com/paper/2604.07765/) (2026)
- [RSVG-ZeroOV: Exploring a Training-Free Framework for Zero-Shot Open-Vocabulary Visual Grounding in Remote Sensing Images](https://mlsanity.com/paper/2509.18711/) (2025)
- [ProVG: Progressive Visual Grounding via Language Decoupling for Remote Sensing Imagery](https://mlsanity.com/paper/2604.01893/) (2026)
- [Bridging Semantics and Geometry: A Decoupled LVLM-SAM Framework for Reasoning Segmentation in Optical Remote Sensing](https://mlsanity.com/paper/2512.19302/) (2025)
- [Object Fidelity Diffusion for Remote Sensing Image Generation](https://mlsanity.com/paper/2508.10801/) (2025)
- [VectorSynth: Fine-Grained Satellite Image Synthesis with Structured Semantics](https://mlsanity.com/paper/2511.07744/) (2025)
- [DVGBench: Implicit-to-Explicit Visual Grounding Benchmark in UAV Imagery with Large Vision-Language Models](https://mlsanity.com/paper/2601.00998/) (2026)
- [Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Tasks](https://mlsanity.com/paper/2606.10819/) (2026)
- [OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Tasks](https://mlsanity.com/paper/2603.09471/) (2026)
- [Towards Self-cognitive Exploration: Metacognitive Knowledge Graph Retrieval Augmented Generation](https://mlsanity.com/paper/2508.09460/) (2025)
- [RemoteShield: Enable Robust Multimodal Large Language Models for Earth Observation](https://mlsanity.com/paper/2604.17243/) (2026)
- [Agentic AI for Remote Sensing: Technical Challenges and Research Directions](https://mlsanity.com/paper/2604.24919/) (2026)
- [RemoteZero: Geospatial Reasoning with Zero Labels](https://mlsanity.com/paper/2605.04451/) (2026)
- [Chain-of-Talkers (CoTalk): Fast Human Annotation of Dense Image Captions](https://mlsanity.com/paper/2505.22627/) (2025)
- [Remote Sensing Image Intelligent Interpretation with the Language-Centered Perspective: Principles, Methods and Challenges](https://mlsanity.com/paper/2508.06832/) (2025)
- [An Open-Source Benchmark and Baseline for Multi-temporal Referring Segmentation](https://mlsanity.com/paper/2606.00987/) (2026)
- [Parse, Search, and Confirmation: Training-Free Aerial Vision-and-Dialog Navigation with Chain-of-Thought Reasoning and Structured Spatial Memory](https://mlsanity.com/paper/2607.11529/) (2026)
- [Disentangle Object and Non-object Infrared Features via Language Guidance](https://mlsanity.com/paper/2601.09228/) (2026)
- [FarmMind: Reasoning-Query-Driven Dynamic Segmentation for Farmland Remote Sensing Images](https://mlsanity.com/paper/2601.22809/) (2026)
- [VisionNVS: Self-Supervised Inpainting for Novel View Synthesis under the Virtual-Shift Paradigm](https://mlsanity.com/paper/2603.17382/) (2026)
- [PC-SAM: Patch-Constrained Fine-Grained Interactive Road Segmentation in High-Resolution Remote Sensing Images](https://mlsanity.com/paper/2604.00495/) (2026)
- [DiffuSAM: Diffusion Guided Zero-Shot Object Grounding for Remote Sensing Imagery](https://mlsanity.com/paper/2604.18201/) (2026)
- [Evaluating Remote Sensing Image Captions Beyond Metric Biases](https://mlsanity.com/paper/2604.22855/) (2026)
- [Image-Conditioned Instance Prompt Network for Referring Remote Sensing Image Segmentation](https://mlsanity.com/paper/2605.24532/) (2026)
- [Improving Visual Grounding in Remote Sensing via Cluster-Guided Refinement and Model Ensemble Voting](https://mlsanity.com/paper/2606.00556/) (2026)
- [RSGPNet: Geometric Prompting for Remote Sensing Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2606.28410/) (2026)

## Abstract (arXiv)

We aim to develop a robust yet flexible visual foundation model for Earth observation. It should possess strong capabilities in recognizing and localizing diverse visual targets while providing compatibility with various input-output interfaces required across different task scenarios. Current systems cannot meet these requirements, as they typically utilize task-specific architecture trained on narrow data domains with limited semantic coverage. Our study addresses these limitations from two aspects: data and modeling. We first introduce an automatic data engine that enjoys significantly better scalability compared to previous human annotation or rule-based approaches. It has enabled us to create the largest dataset of its kind to date, comprising 270K image-text-mask triplets covering an unprecedented range of diverse semantic categories and attribute specifications. Based on this data foundation, we further propose a task unification paradigm that centers around referring expression segmentation. It effectively handles a wide range of vision-centric perception tasks, including classification, detection, segmentation, grounding, etc, using a single model without any task-specific heads. Combining these innovations on data and modeling, we present RemoteSAM, a foundation model that establishes new SoTA on several earth observation perception benchmarks, outperforming other foundation models such as Falcon, GeoChat, and LHRS-Bot with significantly higher efficiency. Models and data are publicly available at https://github.com/1e12Leon/RemoteSAM.