# Promptable Concept Segmentation from Above: Evaluating SAM 3's Zero-Shot and One-Shot Capabilities in Remote Sensing

arXiv: https://arxiv.org/abs/2607.09583 · HTML: https://mlsanity.com/paper/2607.09583/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 28 references that are in the map.
- First posted on arXiv 10 Jul 2026.

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [YOLO-World: Real-Time Open-Vocabulary Object Detection](https://mlsanity.com/paper/2401.17270/) (2024)
- [RemoteCLIP: A Vision Language Foundation Model for Remote Sensing](https://mlsanity.com/paper/2306.11029/) (2023)
- [AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classification](https://mlsanity.com/paper/1608.05167/) (2016)
- [Object Detection in Optical Remote Sensing Images: A Survey and A New Benchmark](https://mlsanity.com/paper/1909.00133/) (2019)
- [Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP](https://mlsanity.com/paper/2308.02487/) (2023)
- [iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Images](https://mlsanity.com/paper/1905.12886/) (2019)
- [RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing](https://mlsanity.com/paper/2306.11300/) (2023)
- [SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing](https://mlsanity.com/paper/2312.12856/) (2023)
- [Zero-Shot Instance Segmentation](https://mlsanity.com/paper/2104.06601/) (2021)
- [Semantics-Guided Contrastive Network for Zero-Shot Object detection](https://mlsanity.com/paper/2109.06062/) (2021)
- [Toward Open Vocabulary Aerial Object Detection with CLIP-Activated Student-Teacher Learning](https://mlsanity.com/paper/2311.11646/) (2023)
- [Robust Region Feature Synthesizer for Zero-Shot Object Detection](https://mlsanity.com/paper/2201.00103/) (2022)
- [Semantic-Promoted Debiasing and Background Disambiguation for Zero-Shot Instance Segmentation](https://mlsanity.com/paper/2305.13173/) (2023)
- [Enhancing Remote Sensing Vision-Language Models Through MLLM and LLM-Based High-Quality Image-Text Dataset Generation](https://mlsanity.com/paper/2507.16716/) (2025)
- [ZoRI: Towards Discriminative Zero-Shot Remote Sensing Instance Segmentation](https://mlsanity.com/paper/2412.12798/) (2024)
- [Frustratingly Simple but Effective Zero-shot Detection and Segmentation: Analysis and a Strong Baseline](https://mlsanity.com/paper/2302.07319/) (2023)
- [VK-Det: Visual Knowledge Guided Prototype Learning for Open-Vocabulary Aerial Object Detection](https://mlsanity.com/paper/2511.18075/) (2025)

## Abstract (arXiv)

The deployment of large-scale foundation models, such as the Segment Anything Model 3 (SAM 3), promises a transition toward open-vocabulary, training-free computer vision. However, their capacity to generalize out-of-distribution to the complex, top-down geometric structures of Earth Observation imagery remains largely unquantified. Driven by SAM 3's performance disparities in highly specialized domains, we present a comprehensive, multi-task empirical evaluation across remote sensing scene classification, object detection, and instance segmentation under strict zero-shot and one-shot constraints. To achieve this, we introduce a structural adaptation of SAM 3 by repurposing its decoupled binary presence head into a standalone zero-shot classifier. Furthermore, by systematically isolating textual and visual prompt modalities across five configurations, we explicitly diagnose the alignment mechanics within the model's multimodal decoder. Our findings reveal severe cross-modal interference: while visual prompts successfully align the decoder to complex remote sensing geometry, textual prompts inject misaligned, ground-level semantic bias, actively degrading coordinate regression. To benchmark these capabilities without resource-intensive training, we formulate a novel training-free proxy evaluation protocol for Generalized Zero-Shot tasks (scene classification and instance segmentation). Ultimately, our results demonstrate that SAM 3 avoids the overfitting commonly seen in legacy domain-adapted models, achieving high Harmonic Mean scores in segmentation tasks. However, it remains fundamentally constrained by sub-pixel resolution limits and overhead semantic blind spots, charting a definitive mandate for parameter-efficient geospatial fine-tuning of its multimodal decoder.