# Semantic-Promoted Debiasing and Background Disambiguation for Zero-Shot Instance Segmentation

arXiv: https://arxiv.org/abs/2305.13173 · HTML: https://mlsanity.com/paper/2305.13173/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 73 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](https://mlsanity.com/paper/2112.12143/) (2021)
- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)
- [Open-Vocabulary Object Detection Using Captions](https://mlsanity.com/paper/2011.10678/) (2020)
- [CRIS: CLIP-Driven Referring Image Segmentation](https://mlsanity.com/paper/2111.15174/) (2021)
- [LAVT: Language-Aware Vision Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2112.02244/) (2021)
- [SOLO: Segmenting Objects by Locations](https://mlsanity.com/paper/1912.04488/) (2019)
- [Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/1906.00817/) (2019)
- [Vision-Language Transformer and Query Generation for Referring Segmentation](https://mlsanity.com/paper/2108.05565/) (2021)
- [Zero-Shot Learning Through Cross-Modal Transfer](https://mlsanity.com/paper/1301.3666/) (2013)
- [Open-Vocabulary DETR with Conditional Matching](https://mlsanity.com/paper/2203.11876/) (2022)
- [Decoupling Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2112.07910/) (2021)
- [Zero-Shot Object Detection](https://mlsanity.com/paper/1804.04340/) (2018)
- [Evaluation of Output Embeddings for Fine-Grained Image Classification](https://mlsanity.com/paper/1409.8403/) (2014)
- [An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wild](https://mlsanity.com/paper/1605.04253/) (2016)
- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [Simpler is Better: Few-shot Semantic Segmentation with Classifier Weight Transformer](https://mlsanity.com/paper/2108.03032/) (2021)
- [VLT: Vision-Language Transformer and Query Generation for Referring Segmentation](https://mlsanity.com/paper/2210.15871/) (2022)
- [Learning Open-World Object Proposals without Learning to Classify](https://mlsanity.com/paper/2108.06753/) (2021)

## Built on by (most cited first, 16 of 16)

- [GRES: Generalized Referring Expression Segmentation](https://mlsanity.com/paper/2306.00968/) (2023)
- [Towards Open Vocabulary Learning: A Survey](https://mlsanity.com/paper/2306.15880/) (2023)
- [Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance](https://mlsanity.com/paper/2312.10671/) (2023)
- [A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Future](https://mlsanity.com/paper/2307.09220/) (2023)
- [MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2401.07745/) (2024)
- [GREC: Generalized Referring Expression Comprehension](https://mlsanity.com/paper/2308.16182/) (2023)
- [Multi-Modal Mutual Attention and Iterative Interaction for Referring Image Segmentation](https://mlsanity.com/paper/2305.15302/) (2023)
- [Primitive Generation and Semantic-related Alignment for Universal Zero-Shot Segmentation](https://mlsanity.com/paper/2306.11087/) (2023)
- [Prototype Adaption and Projection for Few- and Zero-shot 3D Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2305.14335/) (2023)
- [RefMask3D: Language-Guided Transformer for 3D Referring Segmentation](https://mlsanity.com/paper/2407.18244/) (2024)
- [Region Generation and Assessment Network for Occluded Person Re-Identification](https://mlsanity.com/paper/2309.03558/) (2023)
- [VGSG: Vision-Guided Semantic-Group Network for Text-based Person Search](https://mlsanity.com/paper/2311.07514/) (2023)
- [AgMTR: Agent Mining Transformer for Few-shot Segmentation in Remote Sensing](https://mlsanity.com/paper/2409.17453/) (2024)
- [AlignZeg: Mitigating Objective Misalignment for Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2404.05667/) (2024)
- [Transferable and Principled Efficiency for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2404.07448/) (2024)
- [Promptable Concept Segmentation from Above: Evaluating SAM 3's Zero-Shot and One-Shot Capabilities in Remote Sensing](https://mlsanity.com/paper/2607.09583/) (2026)

## Abstract (arXiv)

Zero-shot instance segmentation aims to detect and precisely segment objects of unseen categories without any training samples. Since the model is trained on seen categories, there is a strong bias that the model tends to classify all the objects into seen categories. Besides, there is a natural confusion between background and novel objects that have never shown up in training. These two challenges make novel objects hard to be raised in the final instance segmentation results. It is desired to rescue novel objects from background and dominated seen categories. To this end, we propose D2Zero with Semantic-Promoted Debiasing and Background Disambiguation to enhance the performance of Zero-shot instance segmentation. Semantic-promoted debiasing utilizes inter-class semantic relationships to involve unseen categories in visual feature training and learns an input-conditional classifier to conduct dynamical classification based on the input image. Background disambiguation produces image-adaptive background representation to avoid mistaking novel objects for background. Extensive experiments show that we significantly outperform previous state-of-the-art methods by a large margin, e.g., 16.86% improvement on COCO. Project page: https://henghuiding.github.io/D2Zero/