# EOVSAM: Efficient Open-Vocabulary Segmentation with SAM 3 in One Pass

arXiv: https://arxiv.org/abs/2608.02284 · HTML: https://mlsanity.com/paper/2608.02284/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 3 Aug 2026.

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2303.11797/) (2023)
- [AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One](https://mlsanity.com/paper/2312.06709/) (2023)
- [SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2311.15537/) (2023)
- [RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Models](https://mlsanity.com/paper/2412.07679/) (2024)
- [Collaborative Vision-Text Representation Optimizing for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2408.00744/) (2024)
- [Open-Vocabulary Semantic Segmentation with Decoupled One-Pass Network](https://mlsanity.com/paper/2304.01198/) (2023)
- [Open-Vocabulary Semantic Segmentation with Image Embedding Balancing](https://mlsanity.com/paper/2406.09829/) (2024)
- [Effective SAM Combination for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2411.14723/) (2024)
- [OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts](https://mlsanity.com/paper/2507.05427/) (2025)
- [C-RADIOv4 (Tech Report)](https://mlsanity.com/paper/2601.17237/) (2026)
- [Generalization Boosted Adapter for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2409.08468/) (2024)
- [FrozenSeg: Harmonizing Frozen Foundation Models for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2409.03525/) (2024)
- [FGAseg: Fine-Grained Pixel-Text Alignment for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2501.00877/) (2025)
- [SAM-MI: A Mask-Injected Framework for Enhancing Open-Vocabulary Semantic Segmentation with SAM](https://mlsanity.com/paper/2511.20027/) (2025)
- [Open-Vocabulary Panoptic Segmentation Using BERT Pre-Training of Vision-Language Multiway Transformer Model](https://mlsanity.com/paper/2412.18917/) (2024)
- [Novel Category Discovery with X-Agent Attention for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2509.01275/) (2025)
- [Mitigating Objectness Bias and Region-to-Text Misalignment for Open-Vocabulary Panoptic Segmentation](https://mlsanity.com/paper/2603.21386/) (2026)

## Abstract (arXiv)

Open-vocabulary segmentation identifies and segments objects from arbitrary textual descriptions. SAM 3 supports noun-phrase-guided segmentation and achieves competitive open-vocabulary performance through exhaustive vocabulary traversal, yet suffers from prohibitive computational overhead as target categories scale. In this paper, we propose an Efficient Open-Vocabulary segmentation framework with SAM 3 (EOVSAM), which adapts SAM 3 for single-pass prediction. EOVSAM removes prompt conditioning to turn SAM 3 into an efficient mask generator and introduces a new Attentional Aggregation strategy to optimize open-vocabulary classification end-to-end. This formulation avoids the multi-stage pipelines and post-processing heuristics commonly used by existing methods, while mitigating the closed-set collapse that can arise when classification is optimized directly. EOVSAM consistently improves segmentation accuracy over vanilla SAM 3 on all evaluated datasets and accelerates inference by up to 338×. Furthermore, EOVSAM maintains high accuracy at lower resolutions while achieving even more remarkable inference speeds. Experiments on standard semantic and panoptic segmentation benchmarks show that EOVSAM combines competitive or state-of-the-art accuracy with a substantial speed advantage over existing open-vocabulary segmentation models. Code and models are available at https://github.com/hustvl/EOVSAM.