mlsanityData to 7 Oct 2026

Large-scale Unsupervised Semantic Segmentation

arXiv 2106.03149 ↗First posted 6 Jun 2021cs.CVcited by 98

Not on the map. Why?

Abstract (arXiv)

Empowered by large datasets, e.g., ImageNet, unsupervised learning on large-scale data has enabled significant advances for classification tasks. However, whether the large-scale unsupervised semantic segmentation can be achieved remains unknown. There are two major challenges: i) we need a large-scale benchmark for assessing algorithms; ii) we need to develop methods to simultaneously learn category and shape representation in an unsupervised manner. In this work, we propose a new problem of large-scale unsupervised semantic segmentation (LUSS) with a newly created benchmark dataset to help the research progress. Building on the ImageNet dataset, we propose the ImageNet-S dataset with 1.2 million training images and 50k high-quality semantic segmentation annotations for evaluation. Our benchmark has a high data diversity and a clear task objective. We also present a simple yet effective method that works surprisingly well for LUSS. In addition, we benchmark related un/weakly/fully supervised methods accordingly, identifying the challenges and possible directions of LUSS. The benchmark and source code is publicly available at https://github.com/LUSSeg.

Key facts. Cited by 98 papers in the map (data through 7 Oct 2026). Builds on 122 of its 156 references that are in the map. First posted on arXiv 6 Jun 2021.

Builds on

122 of 156 references.

References · 122

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2017Mask R-CNNcited by 7,259
  10. 2016Densely Connected Convolutional Networkscited by 6,691
  11. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  12. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
38 more
  1. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  2. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  5. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  6. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  9. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  10. 2016Pyramid Scene Parsing Networkcited by 2,472
  11. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  13. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  14. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  15. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  16. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,483
  17. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  18. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  19. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  20. 2019Contrastive Multiview Codingcited by 1,297
  21. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  22. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  23. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,128
  24. 2016Colorful Image Colorizationcited by 1,113
  25. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  26. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  27. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  28. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  29. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  30. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  31. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  32. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  33. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  34. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  35. 2020Contrastive Learning with Hard Negative Samplescited by 619
  36. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  37. 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 573
  38. 2019Self-labelling via simultaneous clustering and representation learningcited by 525

50 most cited of 122 All 122 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 12; 2023: 21; 2024: 22; 2025: 20; 2026: 23.

98 papers.

Sort:

By area:

  1. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizercited by 260
  2. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  3. 2023A Closer Look at the Explainability of Contrastive Language-Image Pre-trainingcited by 95
  4. 2022Self-Supervised Learning of Object Parts for Semantic Segmentationcited by 90
  5. 2023CorrMatch: Label Propagation via Correlation Matching for Semi-Supervised Semantic Segmentationcited by 43
  6. 2022Optimizing Relevance Maps of Vision Transformers Improves Robustnesscited by 41
  7. 2023Frozen Transformers in Language Models Are Effective Visual Encoder Layerscited by 39
  8. 2022Self-Guided Diffusion Modelscited by 36
  9. 2024LeGrad: An Explainability Method for Vision Transformers via Feature Formation Sensitivitycited by 32
  10. 2022Sparse Visual Counterfactual Explanations in Image Spacecited by 30
  11. 2023ImageNet-E: Benchmarking Neural Network Robustness via Attribute Editingcited by 30
  12. 2025DFormerv2: Geometry Self-Attention for RGBD Semantic Segmentationcited by 30
38 more
  1. 2022Exploring Visual Interpretability for Contrastive Language-Image Pre-trainingcited by 29
  2. 2023Uncovering Prototypical Knowledge for Weakly Open-Vocabulary Semantic Segmentationcited by 27
  3. 2023Analyzing Vision Transformers for Image Classification in Class Embedding Spacecited by 26
  4. 2023Rewrite Caption Semantics: Bridging Semantic Gaps for Language-Supervised Semantic Segmentationcited by 21
  5. 2022NamedMask: Distilling Segmenters from Complementary Foundation Modelscited by 20
  6. 2023SmooSeg: Smoothness Prior for Unsupervised Semantic Segmentationcited by 18
  7. 2024Visually Consistent Hierarchical Image Classificationcited by 16
  8. 2024Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognitioncited by 16
  9. 2022Deep transfer learning for image classification: a surveycited by 13
  10. 2025Visual Representations inside the Language Modelcited by 13
  11. 2022SERE: Exploring Feature Self-relation for Self-supervised Transformercited by 12
  12. 2024Open-Vocabulary Online Semantic Mapping for SLAMcited by 11
  13. 2022RF-Next: Efficient Receptive Field Search for Convolutional Neural Networkscited by 10
  14. 2024LookHere: Vision Transformers with Directed Attention Generalize and Extrapolatecited by 9
  15. 2025Rapid Salient Object Detection with Difference Convolutional Neural Networkscited by 9
  16. 2022Towards Sustainable Self-supervised Learningcited by 8
  17. 2024Cross-Domain Graph Data Scaling: A Showcase with Diffusion Modelscited by 8
  18. 2024FALIP: Visual Prompt as Foveal Attention Boosts CLIP Zero-Shot Performancecited by 7
  19. 2024Highly Efficient and Unsupervised Framework for Moving Object Detection in Satellite Videoscited by 7
  20. 2025Grad-ECLIP: Gradient-based Visual and Textual Explanations for CLIPcited by 7
  21. 2025Decouple before Align: Visual Disentanglement Enhances Prompt Tuningcited by 7
  22. 2023Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groupscited by 6
  23. 2024Respect the model: Fine-grained and Robust Explanation with Sharing Ratio Decompositioncited by 6
  24. 2022Refine and Represent: Region-to-Object Representation Learningcited by 5
  25. 2022Extracting Semantic Knowledge from GANs with Unsupervised Learningcited by 5
  26. 2023Self-supervised Semantic Segmentation: Consistency over Transformationcited by 5
  27. 2023TagAlign: Improving Vision-Language Alignment with Multi-Tag Classificationcited by 5
  28. 2024ObjectCompose: Evaluating Resilience of Vision-Based Models on Object-to-Background Compositional Changescited by 5
  29. 2025Stepping Out of Similar Semantic Space for Open-Vocabulary Segmentationcited by 5
  30. 2026Diffusing in the Right Space: A Systematic Study of Latent Diffusabilitycited by 5
  31. 2023Zero-shot Unsupervised Transfer Instance Segmentationcited by 4
  32. 2023Exploring Limits of Diffusion-Synthetic Training with Weakly Supervised Semantic Segmentationcited by 4
  33. 2024Exposing Text-Image Inconsistency Using Diffusion Modelscited by 4
  34. 2024Progressive Proxy Anchor Propagation for Unsupervised Semantic Segmentationcited by 4
  35. 2025BatStyler: Advancing Multi-category Style Generation for Source-free Domain Generalizationcited by 4
  36. 2025PixCLIP: Achieving Fine-grained Visual Language Understanding via Any-granularity Pixel-Text Alignment Learningcited by 4
  37. 2023Task-Robust Pre-Training for Worst-Case Downstream Adaptationcited by 3
  38. 2026Investigating Single-Block Recurrence in Vision Transformers for Image Recognitioncited by 3

50 most cited of 98 All 98 (JSON)