# Unsupervised Hierarchical Semantic Segmentation with Multiview Cosegmentation and Clustering Transformers

arXiv: https://arxiv.org/abs/2204.11432 · HTML: https://mlsanity.com/paper/2204.11432/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 69 references that are in the map.
- First posted on arXiv 25 Apr 2022.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [Dense Contrastive Learning for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2011.09157/) (2020)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentation](https://mlsanity.com/paper/1603.06098/) (2016)
- [Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals](https://mlsanity.com/paper/2102.06191/) (2021)
- [Graph Clustering with Graph Neural Networks](https://mlsanity.com/paper/2006.16904/) (2020)
- [Efficient Parameter-free Clustering Using First Neighbor Relations](https://mlsanity.com/paper/1902.11266/) (2019)
- [SegSort: Segmentation by Discriminative Sorting of Segments](https://mlsanity.com/paper/1910.06962/) (2019)
- [STEP: Segmenting and Tracking Every Pixel](https://mlsanity.com/paper/2102.11859/) (2021)
- [Fast Edge Detection Using Structured Forests](https://mlsanity.com/paper/1406.5549/) (2014)
- [Self-Supervised Visual Representation Learning from Hierarchical Grouping](https://mlsanity.com/paper/2012.03044/) (2020)
- [CASTing Your Model: Learning to Localize Improves Self-Supervised Representations](https://mlsanity.com/paper/2012.04630/) (2020)
- [Unsupervised Feature Learning by Cross-Level Instance-Group Discrimination](https://mlsanity.com/paper/2008.03813/) (2020)
- [Adaptive Affinity Fields for Semantic Segmentation](https://mlsanity.com/paper/1803.10335/) (2018)
- [Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations](https://mlsanity.com/paper/2106.05967/) (2021)
- [Universal Weakly Supervised Segmentation by Pixel-to-Segment Contrastive Learning](https://mlsanity.com/paper/2105.00957/) (2021)

## Built on by (most cited first, 30 of 39)

- [ReCo: Retrieve and Co-segment for Zero-shot Transfer](https://mlsanity.com/paper/2206.07045/) (2022)
- [GARField: Group Anything with Radiance Fields](https://mlsanity.com/paper/2401.09419/) (2024)
- [Unsupervised Semantic Segmentation with Self-supervised Object-centric Representations](https://mlsanity.com/paper/2207.05027/) (2022)
- [ACSeg: Adaptive Conceptualization for Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2210.05944/) (2022)
- [CLIP-S4: Language-Guided Self-Supervised Semantic Segmentation](https://mlsanity.com/paper/2305.01040/) (2023)
- [An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels](https://mlsanity.com/paper/2406.09415/) (2024)
- [SAMRefiner: Taming Segment Anything Model for Universal Mask Refinement](https://mlsanity.com/paper/2502.06756/) (2025)
- [Causal Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2310.07379/) (2023)
- [SmooSeg: Smoothness Prior for Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2310.17874/) (2023)
- [LOGICSEG: Parsing Visual Semantics with Neural Logic Learning and Reasoning](https://mlsanity.com/paper/2309.13556/) (2023)
- [Learning Hierarchical Image Segmentation For Recognition and By Recognition](https://mlsanity.com/paper/2210.00314/) (2022)
- [Visually Consistent Hierarchical Image Classification](https://mlsanity.com/paper/2406.11608/) (2024)
- [Adaptive Superpixel for Active Learning in Semantic Segmentation](https://mlsanity.com/paper/2303.16817/) (2023)
- [PrPSeg: Universal Proposition Learning for Panoramic Renal Pathology Segmentation](https://mlsanity.com/paper/2402.19286/) (2024)
- [Weakly-supervised Contrastive Learning for Unsupervised Object Discovery](https://mlsanity.com/paper/2307.03376/) (2023)
- [GCoNet+: A Stronger Group Collaborative Co-Salient Object Detector](https://mlsanity.com/paper/2205.15469/) (2022)
- [Boosting Unsupervised Semantic Segmentation with Principal Mask Proposals](https://mlsanity.com/paper/2404.16818/) (2024)
- [Beyond Pixels: Enhancing LIME with Hierarchical Features and Segmentation Foundation Models](https://mlsanity.com/paper/2403.07733/) (2024)
- [View-Consistent Hierarchical 3D Segmentation Using Ultrametric Feature Fields](https://mlsanity.com/paper/2405.19678/) (2024)
- [ReCLIP++: Learn to Rectify the Bias of CLIP for Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2408.06747/) (2024)
- [Optical Flow boosts Unsupervised Localization and Segmentation](https://mlsanity.com/paper/2307.13640/) (2023)
- [FreePoint: Unsupervised Point Cloud Instance Segmentation](https://mlsanity.com/paper/2305.06973/) (2023)
- [DynaSeg: A Deep Dynamic Fusion Method for Unsupervised Image Segmentation Incorporating Feature Similarity and Spatial Continuity](https://mlsanity.com/paper/2405.05477/) (2024)
- [Revisiting Surgical Instrument Segmentation Without Human Intervention: A Graph Partitioning View](https://mlsanity.com/paper/2408.14789/) (2024)
- [IRS: Incremental Relationship-guided Segmentation for Digital Pathology](https://mlsanity.com/paper/2505.22855/) (2025)
- [A Lightweight Clustering Framework for Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2311.18628/) (2023)
- [Hierarchical Insights: Exploiting Structural Similarities for Reliable 3D Semantic Segmentation](https://mlsanity.com/paper/2404.06124/) (2024)
- [ULTra: Unveiling Latent Token Interpretability in Transformer-Based Understanding and Segmentation](https://mlsanity.com/paper/2411.12589/) (2024)
- [Hierarchical Compact Clustering Attention (COCA) for Unsupervised Object-Centric Learning](https://mlsanity.com/paper/2505.02071/) (2025)
- [Unsupervised Image Semantic Segmentation through Superpixels and Graph Neural Networks](https://mlsanity.com/paper/2210.11810/) (2022)

## Abstract (arXiv)

Unsupervised semantic segmentation aims to discover groupings within and across images that capture object and view-invariance of a category without external supervision. Grouping naturally has levels of granularity, creating ambiguity in unsupervised segmentation. Existing methods avoid this ambiguity and treat it as a factor outside modeling, whereas we embrace it and desire hierarchical grouping consistency for unsupervised segmentation. We approach unsupervised segmentation as a pixel-wise feature learning problem. Our idea is that a good representation shall reveal not just a particular level of grouping, but any level of grouping in a consistent and predictable manner. We enforce spatial consistency of grouping and bootstrap feature learning with co-segmentation among multiple views of the same image, and enforce semantic consistency across the grouping hierarchy with clustering transformers between coarse- and fine-grained features. We deliver the first data-driven unsupervised hierarchical semantic segmentation method called Hierarchical Segment Grouping (HSG). Capturing visual similarity and statistical co-occurrences, HSG also outperforms existing unsupervised segmentation methods by a large margin on five major object- and scene-centric benchmarks. Our code is publicly available at https://github.com/twke18/HSG .