mlsanityData to 7 Oct 2026

Discovering Object Masks with Transformers for Unsupervised Semantic Segmentation

arXiv 2206.06363 ↗First posted 13 Jun 2022cs.CV cs.LGcited by 54

Not on the map. Why?

Abstract (arXiv)

The task of unsupervised semantic segmentation aims to cluster pixels into semantically meaningful groups. Specifically, pixels assigned to the same cluster should share high-level semantic properties like their object or part category. This paper presents MaskDistill: a novel framework for unsupervised semantic segmentation based on three key ideas. First, we advocate a data-driven strategy to generate object masks that serve as a pixel grouping prior for semantic segmentation. This approach omits handcrafted priors, which are often designed for specific scene compositions and limit the applicability of competing frameworks. Second, MaskDistill clusters the object masks to obtain pseudo-ground-truth for training an initial object segmentation model. Third, we leverage this model to filter out low-quality object masks. This strategy mitigates the noise in our pixel grouping prior and results in a clean collection of masks which we use to train a final segmentation model. By combining these components, we can considerably outperform previous works for unsupervised semantic segmentation on PASCAL (+11% mIoU) and COCO (+4% mask AP50). Interestingly, as opposed to existing approaches, our framework does not latch onto low-level image cues and is not limited to object-centric datasets. The code and models will be made available.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 72 of its 92 references that are in the map. First posted on arXiv 13 Jun 2022.

Builds on

72 of 92 references.

References · 72

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2017Mask R-CNNcited by 7,259
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
38 more
  1. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  2. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  3. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  4. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  5. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  6. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  8. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  9. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  10. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  11. 2019Contrastive Multiview Codingcited by 1,297
  12. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,128
  13. 2016Colorful Image Colorizationcited by 1,113
  14. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  15. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  16. 2020Image Segmentation Using Deep Learning: A Surveycited by 659
  17. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  18. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  19. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  20. 2018Invariant Information Clustering for Unsupervised Image Classification and Segmentationcited by 454
  21. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 388
  22. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 359
  23. 2020SCAN: Learning to Classify Images without Labelscited by 351
  24. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  25. 2016ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentationcited by 330
  26. 2015What's the Point: Semantic Segmentation with Point Supervisioncited by 297
  27. 2022Unsupervised Semantic Segmentation by Distilling Feature Correspondencescited by 241
  28. 2015BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentationcited by 238
  29. 2019GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representationscited by 227
  30. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 227
  31. 2016Simple Does It: Weakly Supervised Instance and Semantic Segmentationcited by 226
  32. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  33. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 197
  34. 2021Unsupervised Semantic Segmentation by Contrasting Object Mask Proposalscited by 190
  35. 2015Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentationcited by 189
  36. 2017Colorization as a Proxy Task for Visual Understandingcited by 175
  37. 2018Adversarial Learning for Semi-Supervised Semantic Segmentationcited by 173
  38. 2021PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clusteringcited by 163

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 22; 2024: 20; 2025: 6; 2026: 1.

54 papers.

Sort:

By area:

  1. 2022Bridging the Gap to Real-World Object-Centric Learningcited by 219
  2. 2023Cut and Learn for Unsupervised Object Detection and Instance Segmentationcited by 186
  3. 2022Improving Sample Quality of Diffusion Models Using Self-Attention Guidancecited by 151
  4. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  5. 2022Unsupervised Object Localization: Observing the Background to Discover Objectscited by 66
  6. 2023LOCATE: Localize and Transfer Object Parts for Weakly Supervised Affordance Groundingcited by 64
  7. 2023Hard Patches Mining for Masked Image Modelingcited by 56
  8. 2022ACSeg: Adaptive Conceptualization for Unsupervised Semantic Segmentationcited by 42
  9. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  10. 2023Unsupervised Universal Image Segmentationcited by 34
  11. 2024Segment Anything without Supervisioncited by 27
  12. 2022NamedMask: Distilling Segmenters from Complementary Foundation Modelscited by 20
38 more
  1. 2023TagCLIP: A Local-to-Global Framework to Enhance Open-Vocabulary Multi-Label Classification of CLIP Without Trainingcited by 19
  2. 2023Few-Shot Panoptic Segmentation With Foundation Modelscited by 18
  3. 2023Causal Unsupervised Semantic Segmentationcited by 18
  4. 2024A Simple Latent Diffusion Approach for Panoptic Segmentation and Mask Inpaintingcited by 18
  5. 2023Object-centric Learning with Cyclic Walks between Parts and Wholecited by 16
  6. 2023Unsupervised Object Localization in the Era of Self-Supervised ViTs: A Surveycited by 16
  7. 2024CuVLER: Enhanced Unsupervised Object Discoveries through Exhaustive Self-Supervised Transformerscited by 16
  8. 2023RSFNet: A White-Box Image Retouching Approach using Region-Specific Color Filterscited by 15
  9. 2023MARS: Model-agnostic Biased Object Removal without Additional Supervision for Weakly-Supervised Semantic Segmentationcited by 13
  10. 2024The devil is in the object boundary: towards annotation-free instance segmentation using Foundation Modelscited by 10
  11. 2024ProMerge: Prompt and Merge for Unsupervised Instance Segmentationcited by 10
  12. 2025Scene-Centric Unsupervised Panoptic Segmentationcited by 10
  13. 2024Boosting Unsupervised Semantic Segmentation with Principal Mask Proposalscited by 9
  14. 2023LOCATE: Self-supervised Object Discovery via Flow-guided Graph-cut and Bootstrapped Self-trainingcited by 8
  15. 2023Uncovering the Inner Workings of STEGO for Safe Unsupervised Semantic Segmentationcited by 7
  16. 2023Optical Flow boosts Unsupervised Localization and Segmentationcited by 7
  17. 2023Understanding Self-Supervised Features for Learning Unsupervised Instance Segmentationcited by 7
  18. 2024A Good Foundation is Worth Many Labels: Label-Efficient Panoptic Segmentationcited by 7
  19. 2024Probing the Mid-level Vision Capabilities of Self-Supervised Learningcited by 7
  20. 2024Recasting Generic Pretrained Vision Transformers As Object-Centric Scene Encoders For Manipulation Policiescited by 6
  21. 2024CutS3D: Cutting Semantics in 3D for 2D Unsupervised Instance Segmentationcited by 6
  22. 2023Zero-shot Unsupervised Transfer Instance Segmentationcited by 4
  23. 2023Aerial Image Object Detection With Vision Transformer Detector (ViTDet)cited by 3
  24. 2023Associating Spatially-Consistent Grouping with Text-supervised Semantic Segmentationcited by 3
  25. 2024Investigating the Effectiveness of Cross-Attention to Unlock Zero-Shot Editing of Text-to-Video Diffusion Modelscited by 3
  26. 2023SimCon Loss with Multiple Views for Text Supervised Semantic Segmentationcited by 2
  27. 2024MOD-UV: Learning Mobile Object Detectors from Unlabeled Videoscited by 2
  28. 2024Motion Segmentation for Neuromorphic Aerial Surveillancecited by 2
  29. 2025Discovering and using Spelke segmentscited by 2
  30. 2023A Lightweight Clustering Framework for Unsupervised Semantic Segmentationcited by 1
  31. 2024ULTra: Unveiling Latent Token Interpretability in Transformer-Based Understanding and Segmentationcited by 1
  32. 2023Aligned Unsupervised Pretraining of Object Detectors with Self-trainingcited by 0
  33. 2024DatUS^2: Data-driven Unsupervised Semantic Segmentation with Pre-trained Self-supervised Vision Transformercited by 0
  34. 2024OMH: Structured Sparsity via Optimally Matched Hierarchy for Unsupervised Semantic Segmentationcited by 0
  35. 2024Lift, Splat, Map: Lifting Foundation Masks for Label-Free Semantic Scene Completioncited by 0
  36. 2024Joint-Embedding Predictive Architecture for Self-Supervised Learning of Mask Classification Architecturecited by 0
  37. 2024Analysis of Spatial augmentation in Self-supervised models in the purview of training and test distributionscited by 0
  38. 2025Ensemble Foreground Management for Unsupervised Object Discoverycited by 0

50 most cited of 54 All 54 (JSON)