mlsanityData to 7 Oct 2026

From colouring-in to pointillism: revisiting semantic segmentation supervision

arXiv 2210.14142 ↗First posted 25 Oct 2022cs.CVcited by 22

Not on the map. Why?

Abstract (arXiv)

The prevailing paradigm for producing semantic segmentation training data relies on densely labelling each pixel of each image in the training set, akin to colouring-in books. This approach becomes a bottleneck when scaling up in the number of images, classes, and annotators. Here we propose instead a pointillist approach for semantic segmentation annotation, where only point-wise yes/no questions are answered. We explore design alternatives for such an active learning approach, measure the speed and consistency of human annotators on this task, show that this strategy enables training good segmentation models, and that it is suitable for evaluating models at test time. As concrete proof of the scalability of our method, we collected and released 22.6M point labels over 4,171 classes on the Open Images dataset. Our results enable to rethink the semantic segmentation pipeline of annotation, training, and evaluation from a pointillism point of view.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 56 of its 95 references that are in the map. First posted on arXiv 25 Oct 2022.

Builds on

56 of 95 references.

References · 56

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  11. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  12. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
38 more
  1. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  2. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  3. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  4. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  5. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  6. 2018Panoptic Segmentationcited by 796
  7. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  8. 2021Image Segmentation Using Text and Image Promptscited by 509
  9. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  10. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  11. 2016ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentationcited by 330
  12. 2020Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentationcited by 299
  13. 2015What's the Point: Semantic Segmentation with Point Supervisioncited by 297
  14. 2019Variational Adversarial Active Learningcited by 284
  15. 2019Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relationscited by 245
  16. 2015BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentationcited by 238
  17. 2016Simple Does It: Weakly Supervised Instance and Semantic Segmentationcited by 226
  18. 2019Connecting Vision and Language with Localized Narrativescited by 225
  19. 2016Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentationcited by 198
  20. 2021Decoupling Zero-Shot Semantic Segmentationcited by 198
  21. 2015Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentationcited by 189
  22. 2016Single-Image Depth Perception in the Wildcited by 179
  23. 2016Deep Interactive Object Selectioncited by 160
  24. 2019FSS-1000: A 1000-Class Dataset for Few-Shot Segmentationcited by 160
  25. 2017Suggestive Annotation: A Deep Active Learning Framework for Biomedical Image Segmentationcited by 145
  26. 2017Deep Extreme Cut: From Extreme Points to Object Segmentationcited by 127
  27. 2018Efficient Interactive Annotation of Segmentation Datasets with Polygon-RNN++cited by 119
  28. 2019Large-scale interactive object segmentation with human annotatorscited by 109
  29. 2015Constrained Convolutional Neural Networks for Weakly Supervised Segmentationcited by 100
  30. 2021Pointly-Supervised Instance Segmentationcited by 83
  31. 2017Extreme clicking for efficient object annotationcited by 80
  32. 2016Full-Resolution Residual Networks for Semantic Segmentation in Street Scenescited by 71
  33. 2018CEREALS - Cost-Effective REgion-based Active Learning for Semantic Segmentationcited by 47
  34. 2017Exploiting saliency for object segmentation from image level labelscited by 33
  35. 2016Crowdsourcing in Computer Visioncited by 31
  36. 2021All you need are a few pixels: semantic segmentation with PixelPickcited by 30
  37. 2021Semantic Segmentation In-the-Wild Without Seeing Any Segmentation Examplescited by 24
  38. 2020PCAMs: Weakly Supervised Semantic Segmentation Using Point Supervisioncited by 15

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 5; 2025: 9; 2026: 3.

22 papers.

Sort:

By area:

  1. 2023Grounding Everything: Emerging Localization Properties in Vision-Language Transformerscited by 89
  2. 2024LeGrad: An Explainability Method for Vision Transformers via Feature Formation Sensitivitycited by 32
  3. 2023Object Recognition as Next Token Predictioncited by 12
  4. 2025SAGI: Semantically Aligned and Uncertainty Guided AI Image Inpaintingcited by 10
  5. 2023Semi-Supervised Active Learning for Semantic Segmentation in Unknown Environments Using Informative Path Planningcited by 6
  6. 2025Neglected Risks: The Disturbing Reality of Children's Images in Datasets and the Urgent Call for Accountabilitycited by 6
  7. 2024Point-VOS: Pointing Up Video Object Segmentationcited by 5
  8. 2025KOFFVQA: An Objectively Evaluated Free-form VQA Benchmark for Large Vision-Language Models in the Korean Languagecited by 5
  9. 2023Neglected Free Lunch -- Learning Image Classifiers Using Annotation Byproductscited by 4
  10. 2023A Survey on Image-text Multimodal Modelscited by 4
  11. 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Modelscited by 4
  12. 2024Commonly Interesting Imagescited by 3
10 more
  1. 2024MaskInversion: Localized Embeddings via Optimization of Explainability Mapscited by 2
  2. 2025ZS-VCOS: Zero-Shot Video Camouflaged Object Segmentation By Optical Flow and Open Vocabulary Object Detectioncited by 2
  3. 2024Aberration Correcting Vision Transformers for High-Fidelity Metalens Imagingcited by 1
  4. 2025Preserve Anything: Controllable Image Synthesis with Object Preservationcited by 1
  5. 2026MetaRanker: Human-in-the-loop Active Ranking for Metalens Image Qualitycited by 1
  6. 2025SNAP: A Benchmark for Testing the Effects of Capture Conditions on Fundamental Vision Taskscited by 0
  7. 2025Seeing Further on the Shoulders of Giants: Knowledge Inheritance for Vision Foundation Modelscited by 0
  8. 2025MAJORScore: A Novel Metric for Evaluating Multimodal Relevance via Joint Representationcited by 0
  9. 2026MagicSeg: Open-World Segmentation Pretraining via Counterfactural Diffusion-Based Auto-Generationcited by 0
  10. 2026Improving Text-Instance Alignment Of Foreground Conditioned Out-Painting Via Customized Concept Embeddingcited by 0