# ShapeMask: Learning to Segment Novel Objects by Refining Shape Priors

arXiv: https://arxiv.org/abs/1904.03239 · HTML: https://mlsanity.com/paper/1904.03239/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Apr 2019.

## Built on by (most cited first, 30 of 43)

- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)
- [Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers](https://mlsanity.com/paper/2103.12340/) (2021)
- [Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling](https://mlsanity.com/paper/2111.12698/) (2021)
- [ROCA: Robust CAD Model Retrieval and Alignment from a Single Image](https://mlsanity.com/paper/2112.01988/) (2021)
- [Mask Transfiner for High-Quality Instance Segmentation](https://mlsanity.com/paper/2111.13673/) (2021)
- [Incremental Few-Shot Instance Segmentation](https://mlsanity.com/paper/2105.05312/) (2021)
- [A Survey on Label-efficient Deep Image Segmentation: Bridging the Gap between Weak Supervision and Dense Prediction](https://mlsanity.com/paper/2207.01223/) (2022)
- [Unity Perception: Generate Synthetic Data for Computer Vision](https://mlsanity.com/paper/2107.04259/) (2021)
- [SOTR: Segmenting Objects with Transformers](https://mlsanity.com/paper/2108.06747/) (2021)
- [Instance Neural Radiance Field](https://mlsanity.com/paper/2304.04395/) (2023)
- [Unsupervised Discovery of the Long-Tail in Instance Segmentation Using Hierarchical Self-Supervision](https://mlsanity.com/paper/2104.01257/) (2021)
- [Patch2CAD: Patchwise Embedding Learning for In-the-Wild Shape Retrieval from a Single Image](https://mlsanity.com/paper/2108.09368/) (2021)
- [Learning to Detect Every Thing in an Open World](https://mlsanity.com/paper/2112.01698/) (2021)
- [Human De-occlusion: Invisible Perception and Recovery for Humans](https://mlsanity.com/paper/2103.11597/) (2021)
- [Learning to Detect and Segment for Open Vocabulary Object Detection](https://mlsanity.com/paper/2212.12130/) (2022)
- [Noisy Boundaries: Lemon or Lemonade for Semi-supervised Instance Segmentation?](https://mlsanity.com/paper/2203.13427/) (2022)
- [ContrastMask: Contrastive Learning to Segment Every Thing](https://mlsanity.com/paper/2203.09775/) (2022)
- [Delving into Shape-aware Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2304.08491/) (2023)
- [RICE: Refining Instance Masks in Cluttered Environments with Graph Neural Networks](https://mlsanity.com/paper/2106.15711/) (2021)
- [A Unified Efficient Pyramid Transformer for Semantic Segmentation](https://mlsanity.com/paper/2107.14209/) (2021)
- [The surprising impact of mask-head architecture on novel class segmentation](https://mlsanity.com/paper/2104.00613/) (2021)
- [Adversarial Shape Learning for Building Extraction in VHR Remote Sensing Images](https://mlsanity.com/paper/2102.11262/) (2021)
- [BoxSnake: Polygonal Instance Segmentation with Box Supervision](https://mlsanity.com/paper/2303.11630/) (2023)
- [Mixed Supervised Object Detection by Transferring Mask Prior and Semantic Similarity](https://mlsanity.com/paper/2110.14191/) (2021)
- [Weak-shot Semantic Segmentation via Dual Similarity Transfer](https://mlsanity.com/paper/2210.02270/) (2022)
- [Learning to Better Segment Objects from Unseen Classes with Unlabeled Videos](https://mlsanity.com/paper/2104.12276/) (2021)
- [Generating Masks from Boxes by Mining Spatio-Temporal Consistencies in Videos](https://mlsanity.com/paper/2101.02196/) (2021)
- [Open-world Instance Segmentation: Top-down Learning with Bottom-up Supervision](https://mlsanity.com/paper/2303.05503/) (2023)
- [Weak-shot Semantic Segmentation by Transferring Semantic Affinity and Boundary](https://mlsanity.com/paper/2110.01519/) (2021)

## Abstract (arXiv)

Instance segmentation aims to detect and segment individual objects in a scene. Most existing methods rely on precise mask annotations of every category. However, it is difficult and costly to segment objects in novel categories because a large number of mask annotations is required. We introduce ShapeMask, which learns the intermediate concept of object shape to address the problem of generalization in instance segmentation to novel categories. ShapeMask starts with a bounding box detection and gradually refines it by first estimating the shape of the detected object through a collection of shape priors. Next, ShapeMask refines the coarse shape into an instance level mask by learning instance embeddings. The shape priors provide a strong cue for object-like prediction, and the instance embeddings model the instance specific appearance information. ShapeMask significantly outperforms the state-of-the-art by 6.4 and 3.8 AP when learning across categories, and obtains competitive performance in the fully supervised setting. It is also robust to inaccurate detections, decreased model capacity, and small training data. Moreover, it runs efficiently with 150ms inference time and trains within 11 hours on TPUs. With a larger backbone model, ShapeMask increases the gap with state-of-the-art to 9.4 and 6.2 AP across categories. Code will be released.