# AerOSeg: Harnessing SAM for Open-Vocabulary Segmentation in Remote Sensing Images arXiv: https://arxiv.org/abs/2504.09203 ยท HTML: https://mlsanity.com/paper/2504.09203/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 32 of its 48 references that are in the map. - First posted on arXiv 12 Apr 2025. ## Builds on (most cited first, 30 of 32) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021) - [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024) - [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021) - [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020) - [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021) - [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022) - [DOTA: A Large-scale Dataset for Object Detection in Aerial Images](https://mlsanity.com/paper/1711.10398/) (2017) - [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023) - [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023) - [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](https://mlsanity.com/paper/2112.12143/) (2021) - [MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers](https://mlsanity.com/paper/2012.00759/) (2020) - [Side Adapter Network for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2302.12242/) (2023) - [Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/1906.00817/) (2019) - [CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2303.11797/) (2023) - [Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP](https://mlsanity.com/paper/2308.02487/) (2023) - [ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2212.03588/) (2022) - [UNetFormer: A UNet-like Transformer for Efficient Semantic Segmentation of Remote Sensing Urban Scene Imagery](https://mlsanity.com/paper/2109.08937/) (2021) - [RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing](https://mlsanity.com/paper/2306.11300/) (2023) - [Convolutional neural network architecture for geometric matching](https://mlsanity.com/paper/1703.05593/) (2017) - [SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing](https://mlsanity.com/paper/2312.12856/) (2023) - [SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2311.15537/) (2023) - [OpenEarthMap: A Benchmark Dataset for Global High-Resolution Land Cover Mapping](https://mlsanity.com/paper/2210.10732/) (2022) - [Open-Vocabulary Segmentation with Semantic-Assisted Calibration](https://mlsanity.com/paper/2312.04089/) (2023) - [Open-Vocabulary Semantic Segmentation with Image Embedding Balancing](https://mlsanity.com/paper/2406.09829/) (2024) - [SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Images](https://mlsanity.com/paper/2410.01768/) (2024) - [USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation](https://mlsanity.com/paper/2406.05271/) (2024) - [Semantic Segmentation with Labeling Uncertainty and Class Imbalance](https://mlsanity.com/paper/2102.04566/) (2021) ## Built on by (most cited first, 9 of 9) - [SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Images](https://mlsanity.com/paper/2512.08730/) (2025) - [UniVCD: A New Method for Unsupervised Change Detection in the Open-Vocabulary Era](https://mlsanity.com/paper/2512.13089/) (2025) - [SSS: Semi-Supervised SAM-2 with Efficient Prompting for Medical Imaging Segmentation](https://mlsanity.com/paper/2506.08949/) (2025) - [ViRefSAM: Visual Reference-Guided Segment Anything Model for Remote Sensing Segmentation](https://mlsanity.com/paper/2507.02294/) (2025) - [DINO Soars: DINOv3 for Open-Vocabulary Semantic Segmentation of Remote Sensing Imagery](https://mlsanity.com/paper/2605.03175/) (2026) - [Open-Vocabulary Semantic Segmentation in Remote Sensing via Hierarchical Attention Masking and Model Composition](https://mlsanity.com/paper/2602.23869/) (2026) - [dinov3.seg: Open-Vocabulary Semantic Segmentation with DINOv3](https://mlsanity.com/paper/2603.19531/) (2026) - [Seeking Consensus: Geometric-Semantic On-the-Fly Recalibration for Open-Vocabulary Remote Sensing Semantic Segmentation](https://mlsanity.com/paper/2604.26221/) (2026) - [A Geometric-Transformation Feature-Adaptive Manifold Restoration Method for Open-Vocabulary Semantic Segmentation of Remote Sensing Images](https://mlsanity.com/paper/2610.04300/) (2026) ## Abstract (arXiv) Image segmentation beyond predefined categories is a key challenge in remote sensing, where novel and unseen classes often emerge during inference. Open-vocabulary image Segmentation addresses these generalization issues in traditional supervised segmentation models while reducing reliance on extensive per-pixel annotations, which are both expensive and labor-intensive to obtain. Most Open-Vocabulary Segmentation (OVS) methods are designed for natural images but struggle with remote sensing data due to scale variations, orientation changes, and complex scene compositions. This necessitates the development of OVS approaches specifically tailored for remote sensing. In this context, we propose AerOSeg, a novel OVS approach for remote sensing data. First, we compute robust image-text correlation features using multiple rotated versions of the input image and domain-specific prompts. These features are then refined through spatial and class refinement blocks. Inspired by the success of the Segment Anything Model (SAM) in diverse domains, we leverage SAM features to guide the spatial refinement of correlation features. Additionally, we introduce a semantic back-projection module and loss to ensure the seamless propagation of SAM's semantic information throughout the segmentation pipeline. Finally, we enhance the refined correlation features using a multi-scale attention-aware decoder to produce the final segmentation map. We validate our SAM-guided Open-Vocabulary Remote Sensing Segmentation model on three benchmark remote sensing datasets: iSAID, DLRSD, and OpenEarthMap. Our model outperforms state-of-the-art open-vocabulary segmentation methods, achieving an average improvement of 2.54 h-mIoU.