# RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Model

arXiv: https://arxiv.org/abs/2306.16269 · HTML: https://mlsanity.com/paper/2306.16269/

## Key facts

- Cited by 181 papers in the map (data through 14 Oct 2026).
- Builds on 80 of its 112 references that are in the map.
- First posted on arXiv 28 Jun 2023.

## Builds on (most cited first, 30 of 80)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)

## Built on by (most cited first, 30 of 181)

- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Foundational Models Defining a New Era in Vision: A Survey and Outlook](https://mlsanity.com/paper/2307.13721/) (2023)
- [Change-Agent: Towards Interactive Comprehensive Remote Sensing Change Interpretation and Analysis](https://mlsanity.com/paper/2403.19646/) (2024)
- [SAM2-Adapter: Evaluating & Adapting Segment Anything 2 in Downstream Tasks: Camouflage, Shadow, Medical Image Segmentation, and More](https://mlsanity.com/paper/2408.04579/) (2024)
- [CDMamba: Incorporating Local Clues into Mamba for Remote Sensing Image Binary Change Detection](https://mlsanity.com/paper/2406.04207/) (2024)
- [Text2Earth: Unlocking Text-driven Remote Sensing Image Generation with a Global-Scale Dataset and a Foundation Model](https://mlsanity.com/paper/2501.00895/) (2025)
- [LSKNet: A Foundation Lightweight Backbone for Remote Sensing](https://mlsanity.com/paper/2403.11735/) (2024)
- [TESSERA: Temporal Embeddings of Surface Spectra for Earth Representation and Analysis](https://mlsanity.com/paper/2506.20380/) (2025)
- [SAM-Assisted Remote Sensing Imagery Semantic Segmentation with Object and Boundary Constraints](https://mlsanity.com/paper/2312.02464/) (2023)
- [RSCaMa: Remote Sensing Image Change Captioning with State Space Model](https://mlsanity.com/paper/2404.18895/) (2024)
- [Diving into Underwater: Segment Anything Model Guided Underwater Salient Instance Segmentation and A Large-scale Dataset](https://mlsanity.com/paper/2406.06039/) (2024)
- [SAGD: Boundary-Enhanced Segment Anything in 3D Gaussian via Gaussian Decomposition](https://mlsanity.com/paper/2401.17857/) (2024)
- [UV-SAM: Adapting Segment Anything Model for Urban Village Identification](https://mlsanity.com/paper/2401.08083/) (2024)
- [EarthMarker: A Visual Prompting Multi-modal Large Language Model for Remote Sensing](https://mlsanity.com/paper/2407.13596/) (2024)
- [Segment Anything Model for Road Network Graph Extraction](https://mlsanity.com/paper/2403.16051/) (2024)
- [Time Travelling Pixels: Bitemporal Features Integration with Foundation Model for Remote Sensing Image Change Detection](https://mlsanity.com/paper/2312.16202/) (2023)
- [CrossEarth: Geospatial Vision Foundation Model for Domain Generalizable Remote Sensing Semantic Segmentation](https://mlsanity.com/paper/2410.22629/) (2024)
- [APSeg: Auto-Prompt Network for Cross-Domain Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2406.08372/) (2024)
- [PointSAM: Pointly-Supervised Segment Anything Model for Remote Sensing Images](https://mlsanity.com/paper/2409.13401/) (2024)
- [Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Models](https://mlsanity.com/paper/2402.01749/) (2024)
- [Fantastic Animals and Where to Find Them: Segment Any Marine Animal with Dual SAM](https://mlsanity.com/paper/2404.04996/) (2024)
- [EdgeTAM: On-Device Track Anything Model](https://mlsanity.com/paper/2501.07256/) (2025)
- [DynamicEarth: How Far are We from Open-Vocabulary Change Detection?](https://mlsanity.com/paper/2501.12931/) (2025)
- [Prompt-based Adaptation in Large-scale Vision Models: A Survey](https://mlsanity.com/paper/2510.13219/) (2025)
- [SlimSAM: 0.1% Data Makes Segment Anything Slim](https://mlsanity.com/paper/2312.05284/) (2023)
- [A Survey on Remote Sensing Foundation Models: From Vision to Multimodality](https://mlsanity.com/paper/2503.22081/) (2025)
- [SAM-DA: UAV Tracks Anything at Night with SAM-Powered Domain Adaptation](https://mlsanity.com/paper/2307.01024/) (2023)
- [PA-SAM: Prompt Adapter SAM for High-Quality Image Segmentation](https://mlsanity.com/paper/2401.13051/) (2024)
- [Semantic-CC: Boosting Remote Sensing Image Change Captioning via Foundational Knowledge and Semantic Guidance](https://mlsanity.com/paper/2407.14032/) (2024)
- [HyperFree: A Channel-adaptive and Tuning-free Foundation Model for Hyperspectral Remote Sensing Imagery](https://mlsanity.com/paper/2503.21841/) (2025)

## Abstract (arXiv)

Leveraging the extensive training data from SA-1B, the Segment Anything Model (SAM) demonstrates remarkable generalization and zero-shot capabilities. However, as a category-agnostic instance segmentation method, SAM heavily relies on prior manual guidance, including points, boxes, and coarse-grained masks. Furthermore, its performance in remote sensing image segmentation tasks remains largely unexplored and unproven. In this paper, we aim to develop an automated instance segmentation approach for remote sensing images, based on the foundational SAM model and incorporating semantic category information. Drawing inspiration from prompt learning, we propose a method to learn the generation of appropriate prompts for SAM. This enables SAM to produce semantically discernible segmentation results for remote sensing images, a concept we have termed RSPrompter. We also propose several ongoing derivatives for instance segmentation tasks, drawing on recent advancements within the SAM community, and compare their performance with RSPrompter. Extensive experimental results, derived from the WHU building, NWPU VHR-10, and SSDD datasets, validate the effectiveness of our proposed method. The code for our method is publicly available at kychen.me/RSPrompter.