# PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clustering

arXiv: https://arxiv.org/abs/2103.17070 · HTML: https://mlsanity.com/paper/2103.17070/

## Key facts

- Cited by 163 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 67 references that are in the map.
- First posted on arXiv 30 Mar 2021.

## Builds on (most cited first, 30 of 46)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [Self-labelling via simultaneous clustering and representation learning](https://mlsanity.com/paper/1911.05371/) (2019)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [SCAN: Learning to Classify Images without Labels](https://mlsanity.com/paper/2005.12320/) (2020)
- [Learning to Discover Novel Visual Categories via Deep Transfer Clustering](https://mlsanity.com/paper/1908.09884/) (2019)
- [Joint Unsupervised Learning of Deep Representations and Image Clusters](https://mlsanity.com/paper/1604.03628/) (2016)
- [Local Aggregation for Unsupervised Learning of Visual Embeddings](https://mlsanity.com/paper/1903.12355/) (2019)
- [Automatically Discovering and Learning New Visual Categories with Ranking Statistics](https://mlsanity.com/paper/2002.05714/) (2020)
- [Unsupervised Learning of Object Landmarks through Conditional Image Generation](https://mlsanity.com/paper/1806.07823/) (2018)
- [Unsupervised Learning of Object Keypoints for Perception and Control](https://mlsanity.com/paper/1906.11883/) (2019)
- [Online Deep Clustering for Unsupervised Representation Learning](https://mlsanity.com/paper/2006.10645/) (2020)
- [Unsupervised Learning of Object Structure and Dynamics from Videos](https://mlsanity.com/paper/1906.07889/) (2019)

## Built on by (most cited first, 30 of 163)

- [Unsupervised Semantic Segmentation by Distilling Feature Correspondences](https://mlsanity.com/paper/2203.08414/) (2022)
- [What the DAAM: Interpreting Stable Diffusion Using Cross Attention](https://mlsanity.com/paper/2210.04885/) (2022)
- [Bridging the Gap to Real-World Object-Centric Learning](https://mlsanity.com/paper/2209.14860/) (2022)
- [Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion](https://mlsanity.com/paper/2308.12469/) (2023)
- [ReCo: Retrieve and Co-segment for Zero-shot Transfer](https://mlsanity.com/paper/2206.07045/) (2022)
- [Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning](https://mlsanity.com/paper/2212.04994/) (2022)
- [Parametric Classification for Generalized Category Discovery: A Baseline Study](https://mlsanity.com/paper/2211.11727/) (2022)
- [Large-scale Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2106.03149/) (2021)
- [Beyond Appearance: a Semantic Controllable Self-Supervised Learning Framework for Human-Centric Visual Tasks](https://mlsanity.com/paper/2303.17602/) (2023)
- [Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrast](https://mlsanity.com/paper/2110.07110/) (2021)
- [Perceptual Grouping in Contrastive Vision-Language Models](https://mlsanity.com/paper/2210.09996/) (2022)
- [Object discovery and representation networks](https://mlsanity.com/paper/2203.08777/) (2022)
- [NeRF-SOS: Any-View Self-supervised Object Segmentation on Complex Scenes](https://mlsanity.com/paper/2209.08776/) (2022)
- [Self-Supervised Visual Representation Learning with Semantic Grouping](https://mlsanity.com/paper/2205.15288/) (2022)
- [Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2407.08268/) (2024)
- [Semi-supervised Semantic Segmentation with Error Localization Network](https://mlsanity.com/paper/2204.02078/) (2022)
- [Discovering Object Masks with Transformers for Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2206.06363/) (2022)
- [Unsupervised Semantic Segmentation with Self-supervised Object-centric Representations](https://mlsanity.com/paper/2207.05027/) (2022)
- [ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features](https://mlsanity.com/paper/2502.04320/) (2025)
- [A Survey on Label-efficient Deep Image Segmentation: Bridging the Gap between Weak Supervision and Dense Prediction](https://mlsanity.com/paper/2207.01223/) (2022)
- [CoDet: Co-Occurrence Guided Region-Word Alignment for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2310.16667/) (2023)
- [EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models](https://mlsanity.com/paper/2401.11739/) (2024)
- [ACSeg: Adaptive Conceptualization for Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2210.05944/) (2022)
- [Leveraging Hidden Positives for Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2303.15014/) (2023)
- [VDD: Varied Drone Dataset for Semantic Segmentation](https://mlsanity.com/paper/2305.13608/) (2023)
- [Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI](https://mlsanity.com/paper/2111.06061/) (2021)
- [TransFGU: A Top-down Approach to Fine-Grained Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2112.01515/) (2021)
- [Self-Guided Diffusion Models](https://mlsanity.com/paper/2210.06462/) (2022)
- [Unsupervised Universal Image Segmentation](https://mlsanity.com/paper/2312.17243/) (2023)
- [Peekaboo: Text to Image Diffusion Models are Zero-Shot Segmentors](https://mlsanity.com/paper/2211.13224/) (2022)

## Abstract (arXiv)

We present a new framework for semantic segmentation without annotations via clustering. Off-the-shelf clustering methods are limited to curated, single-label, and object-centric images yet real-world data are dominantly uncurated, multi-label, and scene-centric. We extend clustering from images to pixels and assign separate cluster membership to different instances within each image. However, solely relying on pixel-wise feature similarity fails to learn high-level semantic concepts and overfits to low-level visual cues. We propose a method to incorporate geometric consistency as an inductive bias to learn invariance and equivariance for photometric and geometric variations. With our novel learning objective, our framework can learn high-level semantic concepts. Our method, PiCIE (Pixel-level feature Clustering using Invariance and Equivariance), is the first method capable of segmenting both things and stuff categories without any hyperparameter tuning or task-specific pre-processing. Our method largely outperforms existing baselines on COCO and Cityscapes with +17.5 Acc. and +4.5 mIoU. We show that PiCIE gives a better initialization for standard supervised training. The code is available at https://github.com/janghyuncho/PiCIE.