# Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity

arXiv: https://arxiv.org/abs/2204.06107 · HTML: https://mlsanity.com/paper/2204.06107/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 66 references that are in the map.
- First posted on arXiv 12 Apr 2022.

## Builds on (most cited first, 30 of 40)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [Rethinking ImageNet Pre-training](https://mlsanity.com/paper/1811.08883/) (2018)
- [MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers](https://mlsanity.com/paper/2012.00759/) (2020)
- [Rethinking Pre-training and Self-training](https://mlsanity.com/paper/2006.06882/) (2020)
- [Towards Open World Object Detection](https://mlsanity.com/paper/2103.02603/) (2021)
- [A Simple Semi-Supervised Learning Framework for Object Detection](https://mlsanity.com/paper/2005.04757/) (2020)
- [Towards Open World Recognition](https://mlsanity.com/paper/1412.5687/) (2014)
- [Learning Data Augmentation Strategies for Object Detection](https://mlsanity.com/paper/1906.11172/) (2019)
- [Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation](https://mlsanity.com/paper/2104.04691/) (2021)
- [Learning the Depths of Moving People by Watching Frozen People](https://mlsanity.com/paper/1904.11111/) (2019)
- [Learning Open-World Object Proposals without Learning to Classify](https://mlsanity.com/paper/2108.06753/) (2021)
- [Dense Extreme Inception Network: Towards a Robust CNN Model for Edge Detection](https://mlsanity.com/paper/1909.01955/) (2019)
- [SSAP: Single-Shot Instance Segmentation With Affinity Pyramid](https://mlsanity.com/paper/1909.01616/) (2019)

## Built on by (most cited first, 30 of 41)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.04803/) (2023)
- [Towards Open Vocabulary Learning: A Survey](https://mlsanity.com/paper/2306.15880/) (2023)
- [PROB: Probabilistic Objectness for Open World Object Detection](https://mlsanity.com/paper/2212.01424/) (2022)
- [MobileSAMv2: Faster Segment Anything to Everything](https://mlsanity.com/paper/2312.09579/) (2023)
- [EgoTracks: A Long-term Egocentric Visual Object Tracking Dataset](https://mlsanity.com/paper/2301.03213/) (2023)
- [A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application](https://mlsanity.com/paper/2310.14277/) (2023)
- [Spatially Covariant Image Registration with Text Prompts](https://mlsanity.com/paper/2311.15607/) (2023)
- [SegPrompt: Boosting Open-world Segmentation via Category-level Prompt Learning](https://mlsanity.com/paper/2308.06531/) (2023)
- [Hyp-OW: Exploiting Hierarchical Structure Learning with Hyperbolic Distance Enhances Open World Object Detection](https://mlsanity.com/paper/2306.14291/) (2023)
- [GOOD: Exploring Geometric Cues for Detecting Objects in an Open World](https://mlsanity.com/paper/2212.11720/) (2022)
- [HASSOD: Hierarchical Adaptive Self-Supervised Object Detection](https://mlsanity.com/paper/2402.03311/) (2024)
- [Generative Active Learning for Long-tailed Instance Segmentation](https://mlsanity.com/paper/2406.02435/) (2024)
- [Open-world Instance Segmentation: Top-down Learning with Bottom-up Supervision](https://mlsanity.com/paper/2303.05503/) (2023)
- [FAST-Splat: Fast, Ambiguity-Free Semantics Transfer in Gaussian Splatting](https://mlsanity.com/paper/2411.13753/) (2024)
- [3D Indoor Instance Segmentation in an Open-World](https://mlsanity.com/paper/2309.14338/) (2023)
- [Beyond mAP: Towards better evaluation of instance segmentation](https://mlsanity.com/paper/2207.01614/) (2022)
- [Unsupervised Recognition of Unknown Objects for Open-World Object Detection](https://mlsanity.com/paper/2308.16527/) (2023)
- [SOHES: Self-supervised Open-world Hierarchical Entity Segmentation](https://mlsanity.com/paper/2404.12386/) (2024)
- [Video Instance Segmentation in an Open-World](https://mlsanity.com/paper/2304.01200/) (2023)
- [Exploring Transformers for Open-world Instance Segmentation](https://mlsanity.com/paper/2308.04206/) (2023)
- [Single-Stage Open-world Instance Segmentation with Cross-task Consistency Regularization](https://mlsanity.com/paper/2208.09023/) (2022)
- [SupeRGB-D: Zero-shot Instance Segmentation in Cluttered Indoor Environments](https://mlsanity.com/paper/2212.11922/) (2022)
- [OpenInst: A Simple Query-Based Method for Open-World Instance Segmentation](https://mlsanity.com/paper/2303.15859/) (2023)
- [ActionSwitch: Class-agnostic Detection of Simultaneous Actions in Streaming Videos](https://mlsanity.com/paper/2407.12987/) (2024)
- [ElC-OIS: Ellipsoidal Clustering for Open-World Instance Segmentation on LiDAR Data](https://mlsanity.com/paper/2303.04351/) (2023)
- [SOS: Segment Object System for Open-World Instance Segmentation With Object Priors](https://mlsanity.com/paper/2409.14627/) (2024)
- [Addressing the Challenges of Open-World Object Detection](https://mlsanity.com/paper/2303.14930/) (2023)
- [Low-shot Object Learning with Mutual Exclusivity Bias](https://mlsanity.com/paper/2312.03533/) (2023)
- [UADet: A Remarkably Simple Yet Effective Uncertainty-Aware Open-Set Object Detection Framework](https://mlsanity.com/paper/2412.09229/) (2024)

## Abstract (arXiv)

Open-world instance segmentation is the task of grouping pixels into object instances without any pre-determined taxonomy. This is challenging, as state-of-the-art methods rely on explicit class semantics obtained from large labeled datasets, and out-of-domain evaluation performance drops significantly. Here we propose a novel approach for mask proposals, Generic Grouping Networks (GGNs), constructed without semantic supervision. Our approach combines a local measure of pixel affinity with instance-level mask supervision, producing a training regimen designed to make the model as generic as the data diversity allows. We introduce a method for predicting Pairwise Affinities (PA), a learned local relationship between pairs of pixels. PA generalizes very well to unseen categories. From PA we construct a large set of pseudo-ground-truth instance masks; combined with human-annotated instance masks we train GGNs and significantly outperform the SOTA on open-world instance segmentation on various benchmarks including COCO, LVIS, ADE20K, and UVO. Code is available on project website: https://sites.google.com/view/generic-grouping/.