# Unsupervised Semantic Segmentation Through Depth-Guided Feature Correlation and Sampling

arXiv: https://arxiv.org/abs/2309.12378 · HTML: https://mlsanity.com/paper/2309.12378/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 47 references that are in the map.
- First posted on arXiv 21 Sep 2023.

## Builds on (most cited first, 30 of 37)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs](https://mlsanity.com/paper/1412.7062/) (2014)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation](https://mlsanity.com/paper/1907.10326/) (2019)
- [Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges](https://mlsanity.com/paper/1902.07830/) (2019)
- [Unsupervised Semantic Segmentation by Distilling Feature Correspondences](https://mlsanity.com/paper/2203.08414/) (2022)
- [Bridging the Gap to Real-World Object-Centric Learning](https://mlsanity.com/paper/2209.14860/) (2022)
- [Medical Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2009.13120/) (2020)
- [PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clustering](https://mlsanity.com/paper/2103.17070/) (2021)
- [Pattern-Affinitive Propagation across Depth, Surface Normal and Semantic Segmentation](https://mlsanity.com/paper/1906.03525/) (2019)
- [Multimodal Token Fusion for Vision Transformers](https://mlsanity.com/paper/2204.08721/) (2022)
- [DADA: Depth-aware Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/1904.01886/) (2019)
- [LocalBins: Improving Depth Estimation by Learning Local Distributions](https://mlsanity.com/paper/2203.15132/) (2022)
- [Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimation](https://mlsanity.com/paper/2104.13613/) (2021)
- [Three Ways to Improve Semantic Segmentation with Self-Supervised Depth Estimation](https://mlsanity.com/paper/2012.10782/) (2020)
- [Learning visual groups from co-occurrences in space and time](https://mlsanity.com/paper/1511.06811/) (2015)

## Built on by (most cited first, 19 of 19)

- [Image Segmentation in Foundation Model Era: A Survey](https://mlsanity.com/paper/2408.12957/) (2024)
- [Feed-Forward SceneDINO for Unsupervised Semantic Scene Completion](https://mlsanity.com/paper/2507.06230/) (2025)
- [Scene-Centric Unsupervised Panoptic Segmentation](https://mlsanity.com/paper/2504.01955/) (2025)
- [ReCLIP++: Learn to Rectify the Bias of CLIP for Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2408.06747/) (2024)
- [CutS3D: Cutting Semantics in 3D for 2D Unsupervised Instance Segmentation](https://mlsanity.com/paper/2411.16319/) (2024)
- [Progressive Proxy Anchor Propagation for Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2407.12463/) (2024)
- [Unsupervised Segmentation by Diffusing, Walking and Cutting](https://mlsanity.com/paper/2412.04678/) (2024)
- [ULTra: Unveiling Latent Token Interpretability in Transformer-Based Understanding and Segmentation](https://mlsanity.com/paper/2411.12589/) (2024)
- [GeomPrompt: Geometric Prompt Learning for RGB-D Semantic Segmentation Under Missing and Degraded Depth](https://mlsanity.com/paper/2604.11585/) (2026)
- [Scene-Centric Unsupervised Video Panoptic Segmentation](https://mlsanity.com/paper/2606.04925/) (2026)
- [Falcon: Fractional Alternating Cut with Overcoming Minima in Unsupervised Segmentation](https://mlsanity.com/paper/2504.05613/) (2025)
- [SyncMapV2: Robust and Adaptive Unsupervised Segmentation](https://mlsanity.com/paper/2506.16297/) (2025)
- [Depth Edge Alignment Loss: DEALing with Depth in Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2509.17702/) (2025)
- [Efficient Domain-Adaptive Multi-Task Dense Prediction with Vision Foundation Models](https://mlsanity.com/paper/2509.23626/) (2025)
- [CLASP: Adaptive Spectral Clustering for Unsupervised Per-Image Segmentation](https://mlsanity.com/paper/2509.25016/) (2025)
- [Unsupervised Monocular Road Segmentation for Autonomous Driving via Scene Geometry](https://mlsanity.com/paper/2510.16790/) (2025)
- [S2D: Sparse-To-Dense Keymask Distillation for Unsupervised Video Instance Segmentation](https://mlsanity.com/paper/2512.14440/) (2025)
- [MSSSeg: Learning Multi-Scale Structural Complexity for Self-Supervised Segmentation](https://mlsanity.com/paper/2512.23997/) (2025)
- [ProGuT: Label-Efficient Panoptic Segmentation for Forest Scenes](https://mlsanity.com/paper/2609.36891/) (2026)

## Abstract (arXiv)

Traditionally, training neural networks to perform semantic segmentation required expensive human-made annotations. But more recently, advances in the field of unsupervised learning have made significant progress on this issue and towards closing the gap to supervised algorithms. To achieve this, semantic knowledge is distilled by learning to correlate randomly sampled features from images across an entire dataset. In this work, we build upon these advances by incorporating information about the structure of the scene into the training process through the use of depth information. We achieve this by (1) learning depth-feature correlation by spatially correlate the feature maps with the depth maps to induce knowledge about the structure of the scene and (2) implementing farthest-point sampling to more effectively select relevant features by utilizing 3D sampling techniques on depth information of the scene. Finally, we demonstrate the effectiveness of our technical contributions through extensive experimentation and present significant improvements in performance across multiple benchmark datasets.