# All you need are a few pixels: semantic segmentation with PixelPick

arXiv: https://arxiv.org/abs/2104.06394 · HTML: https://mlsanity.com/paper/2104.06394/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 85 references that are in the map.
- First posted on arXiv 13 Apr 2021.

## Builds on (most cited first, 30 of 49)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [A Survey of Deep Active Learning](https://mlsanity.com/paper/2009.00236/) (2020)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [Semi-Supervised Semantic Segmentation with Cross-Consistency Training](https://mlsanity.com/paper/2003.09005/) (2020)
- [Learning Loss for Active Learning](https://mlsanity.com/paper/1905.03677/) (2019)
- [ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1604.05144/) (2016)
- [Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/1803.10464/) (2018)
- [What's the Point: Semantic Segmentation with Point Supervision](https://mlsanity.com/paper/1506.02106/) (2015)
- [Variational Adversarial Active Learning](https://mlsanity.com/paper/1904.00370/) (2019)
- [COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Images](https://mlsanity.com/paper/2003.09871/) (2020)
- [BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1503.01640/) (2015)
- [Simple Does It: Weakly Supervised Instance and Semantic Segmentation](https://mlsanity.com/paper/1603.07485/) (2016)
- [Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentation](https://mlsanity.com/paper/1603.06098/) (2016)
- [Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentation](https://mlsanity.com/paper/1502.02734/) (2015)
- [ParseNet: Looking Wider to See Better](https://mlsanity.com/paper/1506.04579/) (2015)

## Built on by (most cited first, 30 of 31)

- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [ReCo: Retrieve and Co-segment for Zero-shot Transfer](https://mlsanity.com/paper/2206.07045/) (2022)
- [Semi-Supervised Semantic Segmentation via Adaptive Equalization Learning](https://mlsanity.com/paper/2110.05474/) (2021)
- [Towards Fewer Annotations: Active Learning via Region Impurity and Prediction Uncertainty for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2111.12940/) (2021)
- [From colouring-in to pointillism: revisiting semantic segmentation supervision](https://mlsanity.com/paper/2210.14142/) (2022)
- [Hyperbolic Active Learning for Semantic Segmentation under Domain Shift](https://mlsanity.com/paper/2306.11180/) (2023)
- [Semantic Segmentation with Active Semi-Supervised Learning](https://mlsanity.com/paper/2203.10730/) (2022)
- [Generalized Coverage for More Robust Low-Budget Active Learning](https://mlsanity.com/paper/2407.12212/) (2024)
- [Active Pointly-Supervised Instance Segmentation](https://mlsanity.com/paper/2207.11493/) (2022)
- [Best Practices in Active Learning for Semantic Segmentation](https://mlsanity.com/paper/2302.04075/) (2023)
- [Rapid Network Adaptation: Learning to Adapt Neural Networks Using Test-Time Feedback](https://mlsanity.com/paper/2309.15762/) (2023)
- [Mutual Information Regularization for Weakly-supervised RGB-D Salient Object Detection](https://mlsanity.com/paper/2306.03630/) (2023)
- [Active Learning for Semantic Segmentation with Multi-class Label Query](https://mlsanity.com/paper/2309.09319/) (2023)
- [Semi-Supervised Semantic Segmentation via Marginal Contextual Information](https://mlsanity.com/paper/2308.13900/) (2023)
- [Deep Active Learning for Computer Vision: Past and Future](https://mlsanity.com/paper/2211.14819/) (2022)
- [Active Label Correction for Semantic Segmentation with Foundation Models](https://mlsanity.com/paper/2403.10820/) (2024)
- [Semi-Supervised Active Learning for Semantic Segmentation in Unknown Environments Using Informative Path Planning](https://mlsanity.com/paper/2312.04402/) (2023)
- [When Confidence Fails: Revisiting Pseudo-Label Selection in Semi-supervised Semantic Segmentation](https://mlsanity.com/paper/2509.16704/) (2025)
- [Active Coarse-to-Fine Segmentation of Moveable Parts from Real Images](https://mlsanity.com/paper/2303.11530/) (2023)
- [Bayesian Active Learning for Semantic Segmentation](https://mlsanity.com/paper/2408.01694/) (2024)
- [Semantic Segmentation with Active Semi-Supervised Representation Learning](https://mlsanity.com/paper/2210.08403/) (2022)
- [Integrating Semi-Supervised and Active Learning for Semantic Segmentation](https://mlsanity.com/paper/2501.19227/) (2025)
- [Leveraging image captions for selective whole slide image annotation](https://mlsanity.com/paper/2407.06363/) (2024)
- [Fully Convolutional Networks for Dense Water Flow Intensity Prediction in Swedish Catchment Areas](https://mlsanity.com/paper/2304.01658/) (2023)
- [Mining of Single-Class by Active Learning for Semantic Segmentation](https://mlsanity.com/paper/2307.09109/) (2023)
- [Maturity-Aware Active Learning for Semantic Segmentation with Hierarchically-Adaptive Sample Assessment](https://mlsanity.com/paper/2308.14904/) (2023)
- [Key Patch Proposer: Key Patches Contain Rich Information](https://mlsanity.com/paper/2402.11458/) (2024)
- [Spannotation: Enhancing Semantic Segmentation for Autonomous Navigation with Efficient Image Annotation](https://mlsanity.com/paper/2402.18084/) (2024)
- [HiLLIE: Human-in-the-Loop Training for Low-Light Image Enhancement](https://mlsanity.com/paper/2505.02134/) (2025)
- [Diffusion-Driven Two-Stage Active Learning for Low-Budget Semantic Segmentation](https://mlsanity.com/paper/2510.22229/) (2025)

## Abstract (arXiv)

A central challenge for the task of semantic segmentation is the prohibitive cost of obtaining dense pixel-level annotations to supervise model training. In this work, we show that in order to achieve a good level of segmentation performance, all you need are a few well-chosen pixel labels. We make the following contributions: (i) We investigate the novel semantic segmentation setting in which labels are supplied only at sparse pixel locations, and show that deep neural networks can use a handful of such labels to good effect; (ii) We demonstrate how to exploit this phenomena within an active learning framework, termed PixelPick, to radically reduce labelling cost, and propose an efficient "mouse-free" annotation strategy to implement our approach; (iii) We conduct extensive experiments to study the influence of annotation diversity under a fixed budget, model pretraining, model capacity and the sampling mechanism for picking pixels in this low annotation regime; (iv) We provide comparisons to the existing state of the art in semantic segmentation with active learning, and demonstrate comparable performance with up to two orders of magnitude fewer pixel annotations on the CamVid, Cityscapes and PASCAL VOC 2012 benchmarks; (v) Finally, we evaluate the efficiency of our annotation pipeline and its sensitivity to annotator error to demonstrate its practicality.