mlsanityData to 14 Oct 2026

Unsupervised Universal Image Segmentation

arXiv 2312.17243 ↗First posted 28 Dec 2023cs.CVcited by 35

Not on the map. Why?

Abstract (arXiv)

Several unsupervised image segmentation approaches have been proposed which eliminate the need for dense manually-annotated segmentation masks; current models separately handle either semantic segmentation (e.g., STEGO) or class-agnostic instance segmentation (e.g., CutLER), but not both (i.e., panoptic segmentation). We propose an Unsupervised Universal Segmentation model (U2Seg) adept at performing various image segmentation tasks -- instance, semantic and panoptic -- using a novel unified framework. U2Seg generates pseudo semantic labels for these segmentation tasks via leveraging self-supervised models followed by clustering; each cluster represents different semantic and/or instance membership of pixels. We then self-train the model on these pseudo semantic labels, yielding substantial performance gains over specialized methods tailored to each task: a +2.6 APbox boost vs. CutLER in unsupervised instance segmentation on COCO and a +7.0 PixelAcc increase (vs. STEGO) in unsupervised semantic segmentation on COCOStuff. Moreover, our method sets up a new baseline for unsupervised panoptic segmentation, which has not been previously explored. U2Seg is also a strong pretrained model for few-shot segmentation, surpassing CutLER by +5.0 APmask when trained on a low-data regime, e.g., only 1% COCO labels. We hope our simple yet effective method can inspire more research on unsupervised universal image segmentation.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 50 of its 62 references that are in the map. First posted on arXiv 28 Dec 2023.

Builds on

50 of 62 references.

References · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2023Segment Anythingcited by 8,234
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  9. 2017Mask R-CNNcited by 7,277
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  12. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
38 more
  1. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  2. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  4. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  5. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  6. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,276
  7. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  8. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  9. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  10. 2018Panoptic Segmentationcited by 802
  11. 2015Unsupervised Deep Embedding for Clustering Analysiscited by 767
  12. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  13. 2019CenterNet: Keypoint Triplets for Object Detectioncited by 654
  14. 2019Panoptic Feature Pyramid Networkscited by 645
  15. 2019Self-labelling via simultaneous clustering and representation learningcited by 528
  16. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  17. 2018Invariant Information Clustering for Unsupervised Image Classification and Segmentationcited by 455
  18. 2019Panoptic-DeepLabcited by 363
  19. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  20. 2020SCAN: Learning to Classify Images without Labelscited by 352
  21. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 328
  22. 2022Unsupervised Semantic Segmentation by Distilling Feature Correspondencescited by 241
  23. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 230
  24. 2019Local Aggregation for Unsupervised Learning of Visual Embeddingscited by 204
  25. 2021Unsupervised Semantic Segmentation by Contrasting Object Mask Proposalscited by 190
  26. 2023Cut and Learn for Unsupervised Object Detection and Instance Segmentationcited by 187
  27. 2019UPSNet: A Unified Panoptic Segmentation Networkcited by 175
  28. 2021PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clusteringcited by 163
  29. 2021Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentationcited by 133
  30. 2022TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cutcited by 113
  31. 2022FreeSOLO: Learning to Segment Objects without Annotationscited by 102
  32. 2022Self-Supervised Learning of Object Parts for Semantic Segmentationcited by 90
  33. 2021Large-Scale Unsupervised Object Discoverycited by 54
  34. 2022Discovering Object Masks with Transformers for Unsupervised Semantic Segmentationcited by 54
  35. 2020Unsupervised Feature Learning by Cross-Level Instance-Group Discriminationcited by 51
  36. 2020Improving Unsupervised Image Clustering With Robust Learningcited by 51
  37. 2021Fully Convolutional Networks for Panoptic Segmentation with Point-based Supervisioncited by 31
  38. 2021Unsupervised Selective Labeling for More Effective Semi-Supervised Learningcited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 14; 2025: 11; 2026: 10.

35 papers.

Sort:

By area:

  1. 2024Towards Semantic Equivalence of Tokenization in Multimodal LLMcited by 45
  2. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  3. 2024Segment Anything without Supervisioncited by 28
  4. 2024DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cutcited by 26
  5. 2025Feed-Forward SceneDINO for Unsupervised Semantic Scene Completioncited by 13
  6. 2024Open-Vocabulary Segmentation with Unpaired Mask-Text Supervisioncited by 12
  7. 2025Scene-Centric Unsupervised Panoptic Segmentationcited by 10
  8. 2024Generalization Boosted Adapter for Open-Vocabulary Segmentationcited by 9
  9. 2025TRACE: Your Diffusion Model is Secretly an Instance Edge Detectorcited by 4
  10. 2025unMORE: Unsupervised Multi-Object Segmentation via Center-Boundary Reasoningcited by 4
  11. 2024UMAD: Unsupervised Mask-Level Anomaly Detection for Autonomous Drivingcited by 3
  12. 2024Unsupervised Segmentation by Diffusing, Walking and Cuttingcited by 3
23 more
  1. 2024Integrating Deep Learning in Cardiology: A Comprehensive Review of Atrial Fibrillation, Left Atrial Scar Segmentation, and the Frontiers of State-of-the-Art Techniquescited by 2
  2. 2024CS3: Cascade SAM for Sperm Segmentationcited by 1
  3. 2024Label-Free Model Failure Detection for Lidar-based Point Cloud Segmentationcited by 1
  4. 2024Exploring PCA-based feature representations of image pixels via CNN to enhance food image segmentationcited by 1
  5. 2024ULTra: Unveiling Latent Token Interpretability in Transformer-Based Understanding and Segmentationcited by 1
  6. 2026EReCu: Pseudo-label Evolution Fusion and Refinement with Multi-Cue Learning for Unsupervised Camouflage Detectioncited by 1
  7. 2026Scene-Centric Unsupervised Video Panoptic Segmentationcited by 1
  8. 2026Unlocking Diffusion Hierarchies: Adaptive Timestep Selection for Zero-Shot Segmentationcited by 1
  9. 2024Joint-Embedding Predictive Architecture for Self-Supervised Learning of Mask Classification Architecturecited by 0
  10. 2025SyncMapV2: Robust and Adaptive Unsupervised Segmentationcited by 0
  11. 2025Class Agnostic Instance-level Descriptor for Visual Instance Searchcited by 0
  12. 2025Unsupervised Instance Segmentation with Superpixelscited by 0
  13. 2025CLASP: Adaptive Spectral Clustering for Unsupervised Per-Image Segmentationcited by 0
  14. 2025A Training-Free Framework for Open-Vocabulary Image Segmentation and Recognition with EfficientNet and CLIPcited by 0
  15. 2025Unsupervised Image Classification with Adaptive Nearest Neighbor Selection and Cluster Ensemblescited by 0
  16. 2025Unsupervised Structural Scene Decomposition via Foreground-Aware Slot Attention with Pseudo-Mask Guidancecited by 0
  17. 2026Segmenting Low-Contrast XCTs of Concrete: An Unsupervised Approachcited by 0
  18. 2026High-Resolution Image Reconstruction with Unsupervised Learning and Noisy Data Applied to Ion-Beam Dynamics for Particle Acceleratorscited by 0
  19. 2026Reference-based Category Discovery: Unsupervised Object Detection with Category Awarenesscited by 0
  20. 2026Spatially Selective Self-Training for Unsupervised Building Change Detectioncited by 0
  21. 2026`Attention-Guided Cross-Temporal Clustering for Self-Supervised Video Object Segmentationcited by 0
  22. 2026ProGuT: Label-Efficient Panoptic Segmentation for Forest Scenescited by 0
  23. 2026TAG: Calibrating SAM3 for Universal Image Segmentation without Dense Supervisioncited by 0