# Single-Stage Open-world Instance Segmentation with Cross-task Consistency Regularization

arXiv: https://arxiv.org/abs/2208.09023 · HTML: https://mlsanity.com/paper/2208.09023/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 35 references that are in the map.
- First posted on arXiv 18 Aug 2022.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)
- [Towards Open World Object Detection](https://mlsanity.com/paper/2103.02603/) (2021)
- [SOLO: Segmenting Objects by Locations](https://mlsanity.com/paper/1912.04488/) (2019)
- [Instances as Queries](https://mlsanity.com/paper/2105.01928/) (2021)
- [BlendMask: Top-Down Meets Bottom-Up for Instance Segmentation](https://mlsanity.com/paper/2001.00309/) (2020)
- [Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation](https://mlsanity.com/paper/2104.04691/) (2021)
- [OW-DETR: Open-world Detection Transformer](https://mlsanity.com/paper/2112.01513/) (2021)
- [Instance-aware Semantic Segmentation via Multi-task Network Cascades](https://mlsanity.com/paper/1512.04412/) (2015)
- [Learning Open-World Object Proposals without Learning to Classify](https://mlsanity.com/paper/2108.06753/) (2021)
- [Highly Efficient Salient Object Detection with 100K Parameters](https://mlsanity.com/paper/2003.05643/) (2020)
- [Open-World Entity Segmentation](https://mlsanity.com/paper/2107.14228/) (2021)
- [SOLQ: Segmenting Objects by Learning Queries](https://mlsanity.com/paper/2106.02351/) (2021)
- [Cascade RPN: Delving into High-Quality Region Proposal Network with Adaptive Convolution](https://mlsanity.com/paper/1909.06720/) (2019)
- [Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity](https://mlsanity.com/paper/2204.06107/) (2022)
- [Extending One-Stage Detection with Open-World Proposals](https://mlsanity.com/paper/2201.02302/) (2022)
- [1st Place Solution for the UVO Challenge on Image-based Open-World Segmentation 2021](https://mlsanity.com/paper/2110.10239/) (2021)

## Built on by (most cited first, 4 of 4)

- [SegPrompt: Boosting Open-world Segmentation via Category-level Prompt Learning](https://mlsanity.com/paper/2308.06531/) (2023)
- [Exploring Transformers for Open-world Instance Segmentation](https://mlsanity.com/paper/2308.04206/) (2023)
- [OpenInst: A Simple Query-Based Method for Open-World Instance Segmentation](https://mlsanity.com/paper/2303.15859/) (2023)
- [SOS: Segment Object System for Open-World Instance Segmentation With Object Priors](https://mlsanity.com/paper/2409.14627/) (2024)

## Abstract (arXiv)

Open-World Instance Segmentation (OWIS) is an emerging research topic that aims to segment class-agnostic object instances from images. The mainstream approaches use a two-stage segmentation framework, which first locates the candidate object bounding boxes and then performs instance segmentation. In this work, we instead promote a single-stage framework for OWIS. We argue that the end-to-end training process in the single-stage framework can be more convenient for directly regularizing the localization of class-agnostic object pixels. Based on the single-stage instance segmentation framework, we propose a regularization model to predict foreground pixels and use its relation to instance segmentation to construct a cross-task consistency loss. We show that such a consistency loss could alleviate the problem of incomplete instance annotation -- a common problem in the existing OWIS datasets. We also show that the proposed loss lends itself to an effective solution to semi-supervised OWIS that could be considered an extreme case that all object annotations are absent for some images. Our extensive experiments demonstrate that the proposed method achieves impressive results in both fully-supervised and semi-supervised settings. Compared to SOTA methods, the proposed method significantly improves the AP100 score by 4.75% in UVOrightarrowUVO setting and 4.05% in COCOrightarrowUVO setting. In the case of semi-supervised learning, our model learned with only 30% labeled data, even outperforms its fully-supervised counterpart with 50% labeled data. The code will be released soon.