# Leveraging Hidden Positives for Unsupervised Semantic Segmentation

arXiv: https://arxiv.org/abs/2303.15014 · HTML: https://mlsanity.com/paper/2303.15014/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 41 references that are in the map.
- First posted on arXiv 27 Mar 2023.

## Builds on (most cited first, 30 of 38)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs](https://mlsanity.com/paper/1412.7062/) (2014)
- [Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1906.12340/) (2019)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [Understanding the Behaviour of Contrastive Loss](https://mlsanity.com/paper/2012.09740/) (2020)
- [Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2004.04581/) (2020)
- [Unsupervised Semantic Segmentation by Distilling Feature Correspondences](https://mlsanity.com/paper/2203.08414/) (2022)
- [Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals](https://mlsanity.com/paper/2102.06191/) (2021)
- [PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clustering](https://mlsanity.com/paper/2103.17070/) (2021)
- [FickleNet: Weakly and Semi-supervised Semantic Image Segmentation using Stochastic Inference](https://mlsanity.com/paper/1902.10421/) (2019)
- [Multi-class Token Transformer for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2203.02891/) (2022)
- [SE-SSD: Self-Ensembling Single-Stage Object Detector From Point Cloud](https://mlsanity.com/paper/2104.09804/) (2021)

## Built on by (most cited first, 30 of 40)

- [Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2407.08268/) (2024)
- [Image Segmentation in Foundation Model Era: A Survey](https://mlsanity.com/paper/2408.12957/) (2024)
- [EAGLE: Eigen Aggregation Learning for Object-Centric Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2403.01482/) (2024)
- [Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2411.17150/) (2024)
- [Unsupervised Semantic Segmentation Through Depth-Guided Feature Correlation and Sampling](https://mlsanity.com/paper/2309.12378/) (2023)
- [Causal Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2310.07379/) (2023)
- [Feed-Forward SceneDINO for Unsupervised Semantic Scene Completion](https://mlsanity.com/paper/2507.06230/) (2025)
- [Semi-Supervised Semantic Segmentation Based on Pseudo-Labels: A Survey](https://mlsanity.com/paper/2403.01909/) (2024)
- [MaskDiffusion: Exploiting Pre-trained Diffusion Models for Semantic Segmentation](https://mlsanity.com/paper/2403.11194/) (2024)
- [Scene-Centric Unsupervised Panoptic Segmentation](https://mlsanity.com/paper/2504.01955/) (2025)
- [Deciphering 'What' and 'Where' Visual Pathways from Spectral Clustering of Layer-Distributed Neural Representations](https://mlsanity.com/paper/2312.06716/) (2023)
- [Boosting Unsupervised Semantic Segmentation with Principal Mask Proposals](https://mlsanity.com/paper/2404.16818/) (2024)
- [ProtoS-ViT: Visual foundation models for sparse self-explainable classifications](https://mlsanity.com/paper/2406.10025/) (2024)
- [Mitigating Background Shift in Class-Incremental Semantic Segmentation](https://mlsanity.com/paper/2407.11859/) (2024)
- [TAG: Guidance-free Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2403.11197/) (2024)
- [CutS3D: Cutting Semantics in 3D for 2D Unsupervised Instance Segmentation](https://mlsanity.com/paper/2411.16319/) (2024)
- [Selective Contrastive Learning for Weakly Supervised Affordance Grounding](https://mlsanity.com/paper/2508.07877/) (2025)
- [Blind Localization and Clustering of Anomalies in Textures](https://mlsanity.com/paper/2404.12246/) (2024)
- [From Vicious to Virtuous Cycles: Synergistic Representation Learning for Unsupervised Video Object-Centric Learning](https://mlsanity.com/paper/2602.03390/) (2026)
- [Task-Oriented Channel Attention for Fine-Grained Few-Shot Classification](https://mlsanity.com/paper/2308.00093/) (2023)
- [DynaSeg: A Deep Dynamic Fusion Method for Unsupervised Image Segmentation Incorporating Feature Similarity and Spatial Continuity](https://mlsanity.com/paper/2405.05477/) (2024)
- [Progressive Proxy Anchor Propagation for Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2407.12463/) (2024)
- [SPColor: Semantic Prior Guided Exemplar-based Image Colorization](https://mlsanity.com/paper/2304.06255/) (2023)
- [DHR: Dual Features-Driven Hierarchical Rebalancing in Inter- and Intra-Class Regions for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2404.00380/) (2024)
- [DIP: Unsupervised Dense In-Context Post-training of Visual Representations](https://mlsanity.com/paper/2506.18463/) (2025)
- [A Lightweight Clustering Framework for Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2311.18628/) (2023)
- [ULTra: Unveiling Latent Token Interpretability in Transformer-Based Understanding and Segmentation](https://mlsanity.com/paper/2411.12589/) (2024)
- [WakeupUrban: Unsupervised Semantic Segmentation of Mid-20th century Urban Landscapes with Satellite Imagery](https://mlsanity.com/paper/2506.09476/) (2025)
- [Deep Learning for Crack Detection: A Review of Learning Paradigms, Generalizability, and Datasets](https://mlsanity.com/paper/2508.10256/) (2025)
- [Scene-Centric Unsupervised Video Panoptic Segmentation](https://mlsanity.com/paper/2606.04925/) (2026)

## Abstract (arXiv)

Dramatic demand for manpower to label pixel-level annotations triggered the advent of unsupervised semantic segmentation. Although the recent work employing the vision transformer (ViT) backbone shows exceptional performance, there is still a lack of consideration for task-specific training guidance and local semantic consistency. To tackle these issues, we leverage contrastive learning by excavating hidden positives to learn rich semantic relationships and ensure semantic consistency in local regions. Specifically, we first discover two types of global hidden positives, task-agnostic and task-specific ones for each anchor based on the feature similarities defined by a fixed pre-trained backbone and a segmentation head-in-training, respectively. A gradual increase in the contribution of the latter induces the model to capture task-specific semantic features. In addition, we introduce a gradient propagation strategy to learn semantic consistency between adjacent patches, under the inherent premise that nearby patches are highly likely to possess the same semantics. Specifically, we add the loss propagating to local hidden positives, semantically similar nearby patches, in proportion to the predefined similarity scores. With these training schemes, our proposed method achieves new state-of-the-art (SOTA) results in COCO-stuff, Cityscapes, and Potsdam-3 datasets. Our code is available at: https://github.com/hynnsk/HP.