# Self-supervised Semantic Segmentation Grounded in Visual Concepts

arXiv: https://arxiv.org/abs/2203.13868 · HTML: https://mlsanity.com/paper/2203.13868/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 29 references that are in the map.
- First posted on arXiv 25 Mar 2022.

## Builds on (most cited first, 23 of 23)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [Learning Correspondence from the Cycle-Consistency of Time](https://mlsanity.com/paper/1903.07593/) (2019)
- [One-Shot Video Object Segmentation](https://mlsanity.com/paper/1611.05198/) (2016)
- [Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals](https://mlsanity.com/paper/2102.06191/) (2021)
- [Tracking Emerges by Colorizing Videos](https://mlsanity.com/paper/1806.09594/) (2018)
- [Learning Video Object Segmentation from Static Images](https://mlsanity.com/paper/1612.02646/) (2016)
- [SegSort: Segmentation by Discriminative Sorting of Segments](https://mlsanity.com/paper/1910.06962/) (2019)
- [Interpretable and Accurate Fine-grained Recognition via Region Grouping](https://mlsanity.com/paper/2005.10411/) (2020)
- [Self-Supervised Visual Representation Learning from Hierarchical Grouping](https://mlsanity.com/paper/2012.03044/) (2020)
- [Universal Weakly Supervised Segmentation by Pixel-to-Segment Contrastive Learning](https://mlsanity.com/paper/2105.00957/) (2021)
- [Multigrid Predictive Filter Flow for Unsupervised Learning on Videos](https://mlsanity.com/paper/1904.01693/) (2019)

## Built on by (most cited first, 6 of 6)

- [CLIP-S4: Language-Guided Self-Supervised Semantic Segmentation](https://mlsanity.com/paper/2305.01040/) (2023)
- [DenseDINO: Boosting Dense Self-Supervised Learning with Token-Based Point-Level Consistency](https://mlsanity.com/paper/2306.04654/) (2023)
- [UniParser: Multi-Human Parsing with Unified Correlation Representation Learning](https://mlsanity.com/paper/2310.08984/) (2023)
- [Single-stage Multi-human Parsing via Point Sets and Center-based Offsets](https://mlsanity.com/paper/2304.11356/) (2023)
- [InterVLS: Interactive Model Understanding and Improvement with Vision-Language Surrogates](https://mlsanity.com/paper/2311.03547/) (2023)
- [MSSSeg: Learning Multi-Scale Structural Complexity for Self-Supervised Segmentation](https://mlsanity.com/paper/2512.23997/) (2025)

## Abstract (arXiv)

Unsupervised semantic segmentation requires assigning a label to every pixel without any human annotations. Despite recent advances in self-supervised representation learning for individual images, unsupervised semantic segmentation with pixel-level representations is still a challenging task and remains underexplored. In this work, we propose a self-supervised pixel representation learning method for semantic segmentation by using visual concepts (i.e., groups of pixels with semantic meanings, such as parts, objects, and scenes) extracted from images. To guide self-supervised learning, we leverage three types of relationships between pixels and concepts, including the relationships between pixels and local concepts, local and global concepts, as well as the co-occurrence of concepts. We evaluate the learned pixel embeddings and visual concepts on three datasets, including PASCAL VOC 2012, COCO 2017, and DAVIS 2017. Our results show that the proposed method gains consistent and substantial improvements over recent unsupervised semantic segmentation approaches, and also demonstrate that visual concepts can reveal insights into image datasets.