# Open-Vocabulary Semantic Segmentation with Decoupled One-Pass Network

arXiv: https://arxiv.org/abs/2304.01198 · HTML: https://mlsanity.com/paper/2304.01198/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 45 references that are in the map.
- First posted on arXiv 3 Apr 2023.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [Vision Transformer Adapter for Dense Predictions](https://mlsanity.com/paper/2205.08534/) (2022)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection](https://mlsanity.com/paper/1912.02424/) (2019)
- [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](https://mlsanity.com/paper/2112.12143/) (2021)
- [Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.04803/) (2023)
- [Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation](https://mlsanity.com/paper/2206.02777/) (2022)
- [Prompting Visual-Language Models for Efficient Video Understanding](https://mlsanity.com/paper/2112.04478/) (2021)
- [Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/1906.00817/) (2019)
- [Connecting Vision and Language with Localized Narratives](https://mlsanity.com/paper/1912.03098/) (2019)
- [A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model](https://mlsanity.com/paper/2112.14757/) (2021)
- [TOOD: Task-aligned One-stage Object Detection](https://mlsanity.com/paper/2108.07755/) (2021)

## Built on by (most cited first, 30 of 40)

- [ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference](https://mlsanity.com/paper/2407.12442/) (2024)
- [Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2407.08268/) (2024)
- [Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2411.19331/) (2024)
- [Image Segmentation in Foundation Model Era: A Survey](https://mlsanity.com/paper/2408.12957/) (2024)
- [HyperSeg: Towards Universal Visual Segmentation with Large Language Model](https://mlsanity.com/paper/2411.17606/) (2024)
- [ResCLIP: Residual Attention for Training-free Dense Vision-language Inference](https://mlsanity.com/paper/2411.15851/) (2024)
- [OpenVIS: Open-vocabulary Video Instance Segmentation](https://mlsanity.com/paper/2305.16835/) (2023)
- [Image-Text Co-Decomposition for Text-Supervised Semantic Segmentation](https://mlsanity.com/paper/2404.04231/) (2024)
- [Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2412.04533/) (2024)
- [Cascade-CLIP: Cascaded Vision-Language Embeddings Alignment for Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2406.00670/) (2024)
- [Effective SAM Combination for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2411.14723/) (2024)
- [Image-to-Image Matching via Foundation Models: A New Perspective for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2404.00262/) (2024)
- [Vision-Language Model for Object Detection and Segmentation: A Review and Evaluation](https://mlsanity.com/paper/2504.09480/) (2025)
- [ITACLIP: Boosting Training-Free Semantic Segmentation with Image, Text, and Architectural Enhancements](https://mlsanity.com/paper/2411.12044/) (2024)
- [Exploring Simple Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2401.12217/) (2024)
- [Language-Driven Visual Consensus for Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2403.08426/) (2024)
- [Tuning-free Universally-Supervised Semantic Segmentation](https://mlsanity.com/paper/2405.14294/) (2024)
- [AlignZeg: Mitigating Objective Misalignment for Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2404.05667/) (2024)
- [High-Quality Mask Tuning Matters for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2412.11464/) (2024)
- [Refer to Any Segmentation Mask Group With Vision-Language Prompts](https://mlsanity.com/paper/2506.05342/) (2025)
- [PCA-Seg: Revisiting Cost Aggregation for Open-Vocabulary Semantic and Part Segmentation](https://mlsanity.com/paper/2603.17520/) (2026)
- [Towards Open-Vocabulary Video Semantic Segmentation](https://mlsanity.com/paper/2412.09329/) (2024)
- [DCSEG: Decoupled 3D Open-Set Segmentation using Gaussian Splatting](https://mlsanity.com/paper/2412.10972/) (2024)
- [FLOSS: Free Lunch in Open-vocabulary Semantic Segmentation](https://mlsanity.com/paper/2504.10487/) (2025)
- [What Holds Back Open-Vocabulary Segmentation?](https://mlsanity.com/paper/2508.04211/) (2025)
- [SAM-MI: A Mask-Injected Framework for Enhancing Open-Vocabulary Semantic Segmentation with SAM](https://mlsanity.com/paper/2511.20027/) (2025)
- [Instance Brownian Bridge as Texts for Open-vocabulary Video Instance Segmentation](https://mlsanity.com/paper/2401.09732/) (2024)
- [Advancing Visual Large Language Model for Multi-granular Versatile Perception](https://mlsanity.com/paper/2507.16213/) (2025)
- [Annotation Free Semantic Segmentation with Vision Foundation Models](https://mlsanity.com/paper/2403.09307/) (2024)
- [DreamMask: Boosting Open-vocabulary Panoptic Segmentation with Synthetic Data](https://mlsanity.com/paper/2501.02048/) (2025)

## Abstract (arXiv)

Recently, the open-vocabulary semantic segmentation problem has attracted increasing attention and the best performing methods are based on two-stream networks: one stream for proposal mask generation and the other for segment classification using a pretrained visual-language model. However, existing two-stream methods require passing a great number of (up to a hundred) image crops into the visual-language model, which is highly inefficient. To address the problem, we propose a network that only needs a single pass through the visual-language model for each input image. Specifically, we first propose a novel network adaptation approach, termed patch severance, to restrict the harmful interference between the patch embeddings in the pre-trained visual encoder. We then propose classification anchor learning to encourage the network to spatially focus on more discriminative features for classification. Extensive experiments demonstrate that the proposed method achieves outstanding performance, surpassing state-of-the-art methods while being 4 to 7 times faster at inference. Code: https://github.com/CongHan0808/DeOP.git