# MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features

arXiv: https://arxiv.org/abs/1712.04837 · HTML: https://mlsanity.com/paper/1712.04837/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Dec 2017.

## Built on by (most cited first, 30 of 54)

- [Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP](https://mlsanity.com/paper/2308.02487/) (2023)
- [Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers](https://mlsanity.com/paper/2103.12340/) (2021)
- [Crossover Learning for Fast Online Video Instance Segmentation](https://mlsanity.com/paper/2104.05970/) (2021)
- [Mask Transfiner for High-Quality Instance Segmentation](https://mlsanity.com/paper/2111.13673/) (2021)
- [ISTR: End-to-End Instance Segmentation with Transformers](https://mlsanity.com/paper/2105.00637/) (2021)
- [Incremental Few-Shot Instance Segmentation](https://mlsanity.com/paper/2105.05312/) (2021)
- [RefineMask: Towards High-Quality Instance Segmentation with Fine-Grained Features](https://mlsanity.com/paper/2104.08569/) (2021)
- [Spatial Feature Calibration and Temporal Fusion for Effective One-stage Video Instance Segmentation](https://mlsanity.com/paper/2104.05606/) (2021)
- [SOTR: Segmenting Objects with Transformers](https://mlsanity.com/paper/2108.06747/) (2021)
- [Unsupervised Discovery of the Long-Tail in Instance Segmentation Using Hierarchical Self-Supervision](https://mlsanity.com/paper/2104.01257/) (2021)
- [On Model Calibration for Long-Tailed Object Detection and Instance Segmentation](https://mlsanity.com/paper/2107.02170/) (2021)
- [SOLO: A Simple Framework for Instance Segmentation](https://mlsanity.com/paper/2106.15947/) (2021)
- [CLUSTSEG: Clustering for Universal Segmentation](https://mlsanity.com/paper/2305.02187/) (2023)
- [Semantic Image Segmentation: Two Decades of Research](https://mlsanity.com/paper/2302.06378/) (2023)
- [Instance Shadow Detection with A Single-Stage Detector](https://mlsanity.com/paper/2207.04614/) (2022)
- [A Weakly Supervised Amodal Segmenter with Boundary Uncertainty Estimation](https://mlsanity.com/paper/2108.09897/) (2021)
- [Learning Equivariant Segmentation with Instance-Unique Querying](https://mlsanity.com/paper/2210.00911/) (2022)
- [Mask-free OVIS: Open-Vocabulary Instance Segmentation without Manual Mask Annotations](https://mlsanity.com/paper/2303.16891/) (2023)
- [FAPIS: A Few-shot Anchor-free Part-based Instance Segmenter](https://mlsanity.com/paper/2104.00073/) (2021)
- [RICE: Refining Instance Masks in Cluttered Environments with Graph Neural Networks](https://mlsanity.com/paper/2106.15711/) (2021)
- [Computation-efficient Deep Learning for Computer Vision: A Survey](https://mlsanity.com/paper/2308.13998/) (2023)
- [VideoClick: Video Object Segmentation with a Single Click](https://mlsanity.com/paper/2101.06545/) (2021)
- [Training Generative Adversarial Networks in One Stage](https://mlsanity.com/paper/2103.00430/) (2021)
- [Towards Robust Part-aware Instance Segmentation for Industrial Bin Picking](https://mlsanity.com/paper/2203.02767/) (2022)
- [KTN: Knowledge Transfer Network for Learning Multi-person 2D-3D Correspondences](https://mlsanity.com/paper/2206.10090/) (2022)
- [Occlusion-Aware Instance Segmentation via BiLayer Network Architectures](https://mlsanity.com/paper/2208.04438/) (2022)
- [SIAN: Style-Guided Instance-Adaptive Normalization for Multi-Organ Histopathology Image Synthesis](https://mlsanity.com/paper/2209.02412/) (2022)
- [Frustratingly Simple but Effective Zero-shot Detection and Segmentation: Analysis and a Strong Baseline](https://mlsanity.com/paper/2302.07319/) (2023)
- [DeepDamageNet: A two-step deep-learning model for multi-disaster building damage segmentation and classification using satellite imagery](https://mlsanity.com/paper/2405.04800/) (2024)
- [Superpixel-based Refinement for Object Proposal Generation](https://mlsanity.com/paper/2101.04574/) (2021)

## Abstract (arXiv)

In this work, we tackle the problem of instance segmentation, the task of simultaneously solving object detection and semantic segmentation. Towards this goal, we present a model, called MaskLab, which produces three outputs: box detection, semantic segmentation, and direction prediction. Building on top of the Faster-RCNN object detector, the predicted boxes provide accurate localization of object instances. Within each region of interest, MaskLab performs foreground/background segmentation by combining semantic and direction prediction. Semantic segmentation assists the model in distinguishing between objects of different semantic classes including background, while the direction prediction, estimating each pixel's direction towards its corresponding center, allows separating instances of the same semantic class. Moreover, we explore the effect of incorporating recent successful methods from both segmentation and detection (i.e. atrous convolution and hypercolumn). Our proposed model is evaluated on the COCO instance segmentation benchmark and shows comparable performance with other state-of-art models.