# The Devil is in the Points: Weakly Semi-Supervised Instance Segmentation via Point-Guided Mask Representation

arXiv: https://arxiv.org/abs/2303.15062 · HTML: https://mlsanity.com/paper/2303.15062/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 56 references that are in the map.
- First posted on arXiv 27 Mar 2023.

## Builds on (most cited first, 30 of 45)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Hybrid Task Cascade for Instance Segmentation](https://mlsanity.com/paper/1901.07518/) (2019)
- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)
- [Semi-Supervised Semantic Segmentation with Cross-Consistency Training](https://mlsanity.com/paper/2003.09005/) (2020)
- [Conditional Convolutions for Instance Segmentation](https://mlsanity.com/paper/2003.05664/) (2020)
- [What's the Point: Semantic Segmentation with Point Supervision](https://mlsanity.com/paper/1506.02106/) (2015)
- [SOLO: Segmenting Objects by Locations](https://mlsanity.com/paper/1912.04488/) (2019)
- [Mask Scoring R-CNN](https://mlsanity.com/paper/1903.00241/) (2019)
- [Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relations](https://mlsanity.com/paper/1904.05044/) (2019)
- [Simple Does It: Weakly Supervised Instance and Semantic Segmentation](https://mlsanity.com/paper/1603.07485/) (2016)
- [DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution](https://mlsanity.com/paper/2006.02334/) (2020)
- [CenterMask : Real-Time Anchor-Free Instance Segmentation](https://mlsanity.com/paper/1911.06667/) (2019)
- [FickleNet: Weakly and Semi-supervised Semantic Image Segmentation using Stochastic Inference](https://mlsanity.com/paper/1902.10421/) (2019)
- [Instances as Queries](https://mlsanity.com/paper/2105.01928/) (2021)
- [BoxInst: High-Performance Instance Segmentation with Box Annotations](https://mlsanity.com/paper/2012.02310/) (2020)
- [BlendMask: Top-Down Meets Bottom-Up for Instance Segmentation](https://mlsanity.com/paper/2001.00309/) (2020)
- [Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2103.08896/) (2021)
- [BBAM: Bounding Box Attribution Map for Weakly Supervised Semantic and Instance Segmentation](https://mlsanity.com/paper/2103.08907/) (2021)
- [Pointly-Supervised Instance Segmentation](https://mlsanity.com/paper/2104.06404/) (2021)
- [Weakly-Supervised Semantic Segmentation by Iteratively Mining Common Object Features](https://mlsanity.com/paper/1806.04659/) (2018)
- [Re-distributing Biased Pseudo Labels for Semi-supervised Semantic Segmentation: A Baseline Investigation](https://mlsanity.com/paper/2107.11279/) (2021)
- [Weakly Supervised Instance Segmentation using Class Peak Response](https://mlsanity.com/paper/1804.00880/) (2018)

## Built on by (most cited first, 21 of 21)

- [ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning](https://mlsanity.com/paper/2403.20126/) (2024)
- [SAMRefiner: Taming Segment Anything Model for Universal Mask Refinement](https://mlsanity.com/paper/2502.06756/) (2025)
- [SAM as the Guide: Mastering Pseudo-Label Refinement in Semi-Supervised Referring Expression Segmentation](https://mlsanity.com/paper/2406.01451/) (2024)
- [ZIM: Zero-Shot Image Matting for Anything](https://mlsanity.com/paper/2411.00626/) (2024)
- [Learning Camouflaged Object Detection from Noisy Pseudo Label](https://mlsanity.com/paper/2407.13157/) (2024)
- [Boundary-aware Contrastive Learning for Semi-supervised Nuclei Instance Segmentation](https://mlsanity.com/paper/2402.04756/) (2024)
- [Semi-Supervised Panoptic Narrative Grounding](https://mlsanity.com/paper/2310.18142/) (2023)
- [ProCNS: Progressive Prototype Calibration and Noise Suppression for Weakly-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2401.14074/) (2024)
- [Point-DETR3D: Leveraging Imagery Data with Spatial Point Prior for Weakly Semi-supervised 3D Object Detection](https://mlsanity.com/paper/2403.15317/) (2024)
- [Towards Label-Efficient Human Matting: A Simple Baseline for Weakly Semi-Supervised Trimap-Free Human Matting](https://mlsanity.com/paper/2404.00921/) (2024)
- [Extreme Point Supervised Instance Segmentation](https://mlsanity.com/paper/2405.20729/) (2024)
- [Boosting Medical Image-based Cancer Detection via Text-guided Supervision from Reports](https://mlsanity.com/paper/2405.14230/) (2024)
- [A Point-Neighborhood Learning Framework for Nasal Endoscope Image Segmentation](https://mlsanity.com/paper/2405.20044/) (2024)
- [Predicting Perceived Gloss: Do Weak Labels Suffice?](https://mlsanity.com/paper/2403.17672/) (2024)
- [Hierarchical Salient Patch Identification for Interpretable Fundus Disease Localization](https://mlsanity.com/paper/2405.14334/) (2024)
- [UAVDB: Point-Guided Masks for UAV Detection and Segmentation](https://mlsanity.com/paper/2409.06490/) (2024)
- [ReSAM: Refine, Requery, and Reinforce: Self-Prompting Point-Supervised Segmentation for Remote Sensing Images](https://mlsanity.com/paper/2511.21606/) (2025)
- [Learning from Adversity: Semantic-Aware Mask Refinement through Adversarial Perturbation](https://mlsanity.com/paper/2607.29059/) (2026)
- [Predicting urban tree cover from incomplete point labels and limited background information](https://mlsanity.com/paper/2311.11592/) (2023)
- [Rethinking Saliency-Guided Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2404.00918/) (2024)
- [RefineSeg: Dual Coarse-to-Fine Learning for Medical Image Segmentation](https://mlsanity.com/paper/2508.02844/) (2025)

## Abstract (arXiv)

In this paper, we introduce a novel learning scheme named weakly semi-supervised instance segmentation (WSSIS) with point labels for budget-efficient and high-performance instance segmentation. Namely, we consider a dataset setting consisting of a few fully-labeled images and a lot of point-labeled images. Motivated by the main challenge of semi-supervised approaches mainly derives from the trade-off between false-negative and false-positive instance proposals, we propose a method for WSSIS that can effectively leverage the budget-friendly point labels as a powerful weak supervision source to resolve the challenge. Furthermore, to deal with the hard case where the amount of fully-labeled data is extremely limited, we propose a MaskRefineNet that refines noise in rough masks. We conduct extensive experiments on COCO and BDD100K datasets, and the proposed method achieves promising results comparable to those of the fully-supervised model, even with 50% of the fully labeled COCO data (38.8% vs. 39.7%). Moreover, when using as little as 5% of fully labeled COCO data, our method shows significantly superior performance over the state-of-the-art semi-supervised learning method (33.7% vs. 24.9%). The code is available at https://github.com/clovaai/PointWSSIS.