# Pointly-Supervised Instance Segmentation

arXiv: https://arxiv.org/abs/2104.06404 · HTML: https://mlsanity.com/paper/2104.06404/

## Key facts

- Cited by 83 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 63 references that are in the map.
- First posted on arXiv 13 Apr 2021.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [Local Implicit Grid Representations for 3D Scenes](https://mlsanity.com/paper/2003.08981/) (2020)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [Rethinking Pre-training and Self-training](https://mlsanity.com/paper/2006.06882/) (2020)
- [ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1604.05144/) (2016)
- [Conditional Convolutions for Instance Segmentation](https://mlsanity.com/paper/2003.05664/) (2020)
- [What's the Point: Semantic Segmentation with Point Supervision](https://mlsanity.com/paper/1506.02106/) (2015)
- [Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relations](https://mlsanity.com/paper/1904.05044/) (2019)
- [Simple Does It: Weakly Supervised Instance and Semantic Segmentation](https://mlsanity.com/paper/1603.07485/) (2016)
- [Deep Interactive Object Selection](https://mlsanity.com/paper/1603.04042/) (2016)
- [BoxInst: High-Performance Instance Segmentation with Box Annotations](https://mlsanity.com/paper/2012.02310/) (2020)
- [Deep Extreme Cut: From Extreme Points to Object Segmentation](https://mlsanity.com/paper/1711.09081/) (2017)
- [Large-scale interactive object segmentation with human annotators](https://mlsanity.com/paper/1903.10830/) (2019)
- [Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation](https://mlsanity.com/paper/1503.00848/) (2015)
- [YOLACT++: Better Real-time Instance Segmentation](https://mlsanity.com/paper/1912.06218/) (2019)
- [Extreme clicking for efficient object annotation](https://mlsanity.com/paper/1708.02750/) (2017)
- [Deep Watershed Transform for Instance Segmentation](https://mlsanity.com/paper/1611.08303/) (2016)
- [Weakly Supervised Instance Segmentation using Class Peak Response](https://mlsanity.com/paper/1804.00880/) (2018)

## Built on by (most cited first, 30 of 83)

- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation](https://mlsanity.com/paper/2206.02777/) (2022)
- [ReCo: Retrieve and Co-segment for Zero-shot Transfer](https://mlsanity.com/paper/2206.07045/) (2022)
- [FreeSOLO: Learning to Segment Objects without Annotations](https://mlsanity.com/paper/2202.12181/) (2022)
- [Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling](https://mlsanity.com/paper/2111.12698/) (2021)
- [Segment Anything Meets Point Tracking](https://mlsanity.com/paper/2307.01197/) (2023)
- [Mapping Degeneration Meets Label Evolution: Learning Infrared Small Target Detection with Single Point Supervision](https://mlsanity.com/paper/2304.01484/) (2023)
- [Point-Level Region Contrast for Object Detection Pre-Training](https://mlsanity.com/paper/2202.04639/) (2022)
- [Incremental Learning in Semantic Segmentation from Image Labels](https://mlsanity.com/paper/2112.01882/) (2021)
- [Fully Convolutional Networks for Panoptic Segmentation with Point-based Supervision](https://mlsanity.com/paper/2108.07682/) (2021)
- [Learning Implicit Feature Alignment Function for Semantic Segmentation](https://mlsanity.com/paper/2206.08655/) (2022)
- [Point-to-Box Network for Accurate Object Detection via Single Point Supervision](https://mlsanity.com/paper/2207.06827/) (2022)
- [BoxTeacher: Exploring High-Quality Pseudo Labels for Weakly Supervised Instance Segmentation](https://mlsanity.com/paper/2210.05174/) (2022)
- [DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Model](https://mlsanity.com/paper/2306.01736/) (2023)
- [PointOBB: Learning Oriented Object Detection via Single Point Supervision](https://mlsanity.com/paper/2311.14757/) (2023)
- [Point2Mask: Point-supervised Panoptic Segmentation via Optimal Transport](https://mlsanity.com/paper/2308.01779/) (2023)
- [PointSAM: Pointly-Supervised Segment Anything Model for Remote Sensing Images](https://mlsanity.com/paper/2409.13401/) (2024)
- [From colouring-in to pointillism: revisiting semantic segmentation supervision](https://mlsanity.com/paper/2210.14142/) (2022)
- [Box2Mask: Box-supervised Instance Segmentation via Level-set Evolution](https://mlsanity.com/paper/2212.01579/) (2022)
- [ROADWork: A Dataset and Benchmark for Learning to Recognize, Observe, Analyze and Drive Through Work Zones](https://mlsanity.com/paper/2406.07661/) (2024)
- [Pointly-Supervised Panoptic Segmentation](https://mlsanity.com/paper/2210.13950/) (2022)
- [The Devil is in the Points: Weakly Semi-Supervised Instance Segmentation via Point-Guided Mask Representation](https://mlsanity.com/paper/2303.15062/) (2023)
- [Mask-Free Video Instance Segmentation](https://mlsanity.com/paper/2303.15904/) (2023)
- [Semantic-aware SAM for Point-Prompted Instance Segmentation](https://mlsanity.com/paper/2312.15895/) (2023)
- [Active Pointly-Supervised Instance Segmentation](https://mlsanity.com/paper/2207.11493/) (2022)
- [UVOSAM: A Mask-free Paradigm for Unsupervised Video Object Segmentation via Segment Anything Model](https://mlsanity.com/paper/2305.12659/) (2023)
- [Tree Detection and Diameter Estimation Based on Deep Learning](https://mlsanity.com/paper/2210.17424/) (2022)
- [Reference Twice: A Simple and Unified Baseline for Few-Shot Instance Segmentation](https://mlsanity.com/paper/2301.01156/) (2023)
- [Implicit Anatomical Rendering for Medical Image Segmentation with Stochastic Experts](https://mlsanity.com/paper/2304.03209/) (2023)
- [From Easy to Hard: Progressive Active Learning Framework for Infrared Small Target Detection with Single Point Supervision](https://mlsanity.com/paper/2412.11154/) (2024)

## Abstract (arXiv)

We propose an embarrassingly simple point annotation scheme to collect weak supervision for instance segmentation. In addition to bounding boxes, we collect binary labels for a set of points uniformly sampled inside each bounding box. We show that the existing instance segmentation models developed for full mask supervision can be seamlessly trained with point-based supervision collected via our scheme. Remarkably, Mask R-CNN trained on COCO, PASCAL VOC, Cityscapes, and LVIS with only 10 annotated random points per object achieves 94%--98% of its fully-supervised performance, setting a strong baseline for weakly-supervised instance segmentation. The new point annotation scheme is approximately 5 times faster than annotating full object masks, making high-quality instance segmentation more accessible in practice. Inspired by the point-based annotation form, we propose a modification to PointRend instance segmentation module. For each object, the new architecture, called Implicit PointRend, generates parameters for a function that makes the final point-level mask prediction. Implicit PointRend is more straightforward and uses a single point-level mask loss. Our experiments show that the new module is more suitable for the point-based supervision.