# Learning to Segment Every Thing

arXiv: https://arxiv.org/abs/1711.10370 · HTML: https://mlsanity.com/paper/1711.10370/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Nov 2017.

## Built on by (most cited first, 30 of 65)

- [Decoupling Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2112.07910/) (2021)
- [Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection](https://mlsanity.com/paper/2207.03482/) (2022)
- [Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs](https://mlsanity.com/paper/2101.06894/) (2021)
- [Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation](https://mlsanity.com/paper/2104.04691/) (2021)
- [ReCo: Retrieve and Co-segment for Zero-shot Transfer](https://mlsanity.com/paper/2206.07045/) (2022)
- [Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling](https://mlsanity.com/paper/2111.12698/) (2021)
- [Leveraging Auxiliary Tasks with Affinity Learning for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2107.11787/) (2021)
- [Open-World Entity Segmentation](https://mlsanity.com/paper/2107.14228/) (2021)
- [Evaluating Large-Vocabulary Object Detectors: The Devil is in the Details](https://mlsanity.com/paper/2102.01066/) (2021)
- [Opening up Open-World Tracking](https://mlsanity.com/paper/2104.11221/) (2021)
- [A Simple Long-Tailed Recognition Baseline via Vision-Language Model](https://mlsanity.com/paper/2111.14745/) (2021)
- [A Survey on Label-efficient Deep Image Segmentation: Bridging the Gap between Weak Supervision and Dense Prediction](https://mlsanity.com/paper/2207.01223/) (2022)
- [Learning Generative Models of Textured 3D Meshes from Real-World Images](https://mlsanity.com/paper/2103.15627/) (2021)
- [Unsupervised Discovery of the Long-Tail in Instance Segmentation Using Hierarchical Self-Supervision](https://mlsanity.com/paper/2104.01257/) (2021)
- [Learning to Detect Every Thing in an Open World](https://mlsanity.com/paper/2112.01698/) (2021)
- [Class-Balanced Distillation for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2104.05279/) (2021)
- [Segmentation-grounded Scene Graph Generation](https://mlsanity.com/paper/2104.14207/) (2021)
- [State-of-the-Art in Human Scanpath Prediction](https://mlsanity.com/paper/2102.12239/) (2021)
- [Learning to Detect and Segment for Open Vocabulary Object Detection](https://mlsanity.com/paper/2212.12130/) (2022)
- [Noisy Boundaries: Lemon or Lemonade for Semi-supervised Instance Segmentation?](https://mlsanity.com/paper/2203.13427/) (2022)
- [The Devil is in the Points: Weakly Semi-Supervised Instance Segmentation via Point-Guided Mask Representation](https://mlsanity.com/paper/2303.15062/) (2023)
- [Weakly-supervised Instance Segmentation via Class-agnostic Learning with Salient Images](https://mlsanity.com/paper/2104.01526/) (2021)
- [ContrastMask: Contrastive Learning to Segment Every Thing](https://mlsanity.com/paper/2203.09775/) (2022)
- [MOVE: Unsupervised Movable Object Segmentation and Detection](https://mlsanity.com/paper/2210.07920/) (2022)
- [Segment Anything in Non-Euclidean Domains: Challenges and Opportunities](https://mlsanity.com/paper/2304.11595/) (2023)
- [The surprising impact of mask-head architecture on novel class segmentation](https://mlsanity.com/paper/2104.00613/) (2021)
- [Mixed Supervised Object Detection by Transferring Mask Prior and Semantic Similarity](https://mlsanity.com/paper/2110.14191/) (2021)
- [Learning to Discover and Detect Objects](https://mlsanity.com/paper/2210.10774/) (2022)
- [Weak-shot Semantic Segmentation via Dual Similarity Transfer](https://mlsanity.com/paper/2210.02270/) (2022)
- [Trapped in texture bias? A large scale comparison of deep instance segmentation](https://mlsanity.com/paper/2401.09109/) (2024)

## Abstract (arXiv)

Most methods for object instance segmentation require all training examples to be labeled with segmentation masks. This requirement makes it expensive to annotate new categories and has restricted instance segmentation models to ~100 well-annotated classes. The goal of this paper is to propose a new partially supervised training paradigm, together with a novel weight transfer function, that enables training instance segmentation models on a large set of categories all of which have box annotations, but only a small fraction of which have mask annotations. These contributions allow us to train Mask R-CNN to detect and segment 3000 visual concepts using box annotations from the Visual Genome dataset and mask annotations from the 80 classes in the COCO dataset. We evaluate our approach in a controlled study on the COCO dataset. This work is a first step towards instance segmentation models that have broad comprehension of the visual world.