# Finding an Unsupervised Image Segmenter in Each of Your Deep Generative Models

arXiv: https://arxiv.org/abs/2105.08127 · HTML: https://mlsanity.com/paper/2105.08127/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 60 references that are in the map.
- First posted on arXiv 17 May 2021.

## Builds on (most cited first, 30 of 39)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/1811.12833/) (2018)
- [Learning to Adapt Structured Output Space for Semantic Segmentation](https://mlsanity.com/paper/1802.10349/) (2018)
- [Conditional Image Synthesis With Auxiliary Classifier GANs](https://mlsanity.com/paper/1610.09585/) (2016)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [Closed-Form Factorization of Latent Semantics in GANs](https://mlsanity.com/paper/2007.06600/) (2020)
- [Unsupervised Discovery of Interpretable Directions in the GAN Latent Space](https://mlsanity.com/paper/2002.03754/) (2020)
- [Learning from Simulated and Unsupervised Images through Adversarial Training](https://mlsanity.com/paper/1612.07828/) (2016)
- [Large Scale Adversarial Representation Learning](https://mlsanity.com/paper/1907.02544/) (2019)
- [Geometric GAN](https://mlsanity.com/paper/1705.02894/) (2017)
- [Deeply supervised salient object detection with short connections](https://mlsanity.com/paper/1611.04849/) (2016)
- [Seeing What a GAN Cannot Generate](https://mlsanity.com/paper/1910.11626/) (2019)
- [Amulet: Aggregating Multi-level Convolutional Features for Salient Object Detection](https://mlsanity.com/paper/1708.02001/) (2017)
- [The Hessian Penalty: A Weak Prior for Unsupervised Disentanglement](https://mlsanity.com/paper/2008.10599/) (2020)
- [Unsupervised Domain Adaptation in Semantic Segmentation: a Review](https://mlsanity.com/paper/2005.10876/) (2020)
- [Disentangling factors of variation in deep representations using adversarial training](https://mlsanity.com/paper/1611.03383/) (2016)
- [Unsupervised Object Segmentation by Redrawing](https://mlsanity.com/paper/1905.13539/) (2019)
- [Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs](https://mlsanity.com/paper/2011.00844/) (2020)
- [Autoregressive Unsupervised Image Segmentation](https://mlsanity.com/paper/2007.08247/) (2020)
- [W-Net: A Deep Model for Fully Unsupervised Image Segmentation](https://mlsanity.com/paper/1711.08506/) (2017)
- [Emergence of Object Segmentation in Perturbed Generative Models](https://mlsanity.com/paper/1905.12663/) (2019)
- [Diverse Image Generation via Self-Conditioned GANs](https://mlsanity.com/paper/2006.10728/) (2020)

## Built on by (most cited first, 30 of 44)

- [Label-Efficient Semantic Segmentation with Diffusion Models](https://mlsanity.com/paper/2112.03126/) (2021)
- [Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localization](https://mlsanity.com/paper/2205.07839/) (2022)
- [Dataset Diffusion: Diffusion-based Synthetic Dataset Generation for Pixel-Level Semantic Segmentation](https://mlsanity.com/paper/2309.14303/) (2023)
- [ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation](https://mlsanity.com/paper/2111.10265/) (2021)
- [Unsupervised Object Localization: Observing the Background to Discover Objects](https://mlsanity.com/paper/2212.07834/) (2022)
- [Diffusion Models for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2306.09316/) (2023)
- [GAN-Supervised Dense Visual Alignment](https://mlsanity.com/paper/2112.05143/) (2021)
- [Improving Object-centric Learning with Query Optimization](https://mlsanity.com/paper/2210.08990/) (2022)
- [Unsupervised Part Discovery from Contrastive Reconstruction](https://mlsanity.com/paper/2111.06349/) (2021)
- [Unsupervised Salient Object Detection with Spectral Cluster Voting](https://mlsanity.com/paper/2203.12614/) (2022)
- [DiffusionSeg: Adapting Diffusion Towards Unsupervised Object Discovery](https://mlsanity.com/paper/2303.09813/) (2023)
- [EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models](https://mlsanity.com/paper/2401.11739/) (2024)
- [Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motion](https://mlsanity.com/paper/2205.07844/) (2022)
- [ACSeg: Adaptive Conceptualization for Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2210.05944/) (2022)
- [Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Models](https://mlsanity.com/paper/2401.12978/) (2024)
- [LD-ZNet: A Latent Diffusion Approach for Text-Based Image Segmentation](https://mlsanity.com/paper/2303.12343/) (2023)
- [PandA: Unsupervised Learning of Parts and Appearances in the Feature Maps of GANs](https://mlsanity.com/paper/2206.00048/) (2022)
- [MOVE: Unsupervised Movable Object Segmentation and Detection](https://mlsanity.com/paper/2210.07920/) (2022)
- [CHORUS: Learning Canonicalized 3D Human-Object Spatial Relations from Unbounded Synthesized Images](https://mlsanity.com/paper/2308.12288/) (2023)
- [Unsupervised Object Localization in the Era of Self-Supervised ViTs: A Survey](https://mlsanity.com/paper/2310.12904/) (2023)
- [Data synthesis and adversarial networks: A review and meta-analysis in cancer imaging](https://mlsanity.com/paper/2107.09543/) (2021)
- [Synthetic Data Supervised Salient Object Detection](https://mlsanity.com/paper/2210.13835/) (2022)
- [Weakly-supervised Contrastive Learning for Unsupervised Object Discovery](https://mlsanity.com/paper/2307.03376/) (2023)
- [Contrastive Fine-grained Class Clustering via Generative Adversarial Networks](https://mlsanity.com/paper/2112.14971/) (2021)
- [LOCATE: Self-supervised Object Discovery via Flow-guided Graph-cut and Bootstrapped Self-training](https://mlsanity.com/paper/2308.11239/) (2023)
- [PaintSeg: Training-free Segmentation via Painting](https://mlsanity.com/paper/2305.19406/) (2023)
- [FurryGAN: High Quality Foreground-aware Image Synthesis](https://mlsanity.com/paper/2208.10422/) (2022)
- [Refine and Represent: Region-to-Object Representation Learning](https://mlsanity.com/paper/2208.11821/) (2022)
- [Extracting Semantic Knowledge from GANs with Unsupervised Learning](https://mlsanity.com/paper/2211.16710/) (2022)
- [Foreground-Background Separation through Concept Distillation from Generative Image Foundation Models](https://mlsanity.com/paper/2212.14306/) (2022)

## Abstract (arXiv)

Recent research has shown that numerous human-interpretable directions exist in the latent space of GANs. In this paper, we develop an automatic procedure for finding directions that lead to foreground-background image separation, and we use these directions to train an image segmentation model without human supervision. Our method is generator-agnostic, producing strong segmentation results with a wide range of different GAN architectures. Furthermore, by leveraging GANs pretrained on large datasets such as ImageNet, we are able to segment images from a range of domains without further training or finetuning. Evaluating our method on image segmentation benchmarks, we compare favorably to prior work while using neither human supervision nor access to the training data. Broadly, our results demonstrate that automatically extracting foreground-background structure from pretrained deep generative models can serve as a remarkably effective substitute for human supervision.