# Emergence of Object Segmentation in Perturbed Generative Models

arXiv: https://arxiv.org/abs/1905.12663 · HTML: https://mlsanity.com/paper/1905.12663/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2019.

## Built on by (most cited first, 30 of 57)

- [PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clustering](https://mlsanity.com/paper/2103.17070/) (2021)
- [Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localization](https://mlsanity.com/paper/2205.07839/) (2022)
- [Illiterate DALL-E Learns to Compose](https://mlsanity.com/paper/2110.11405/) (2021)
- [GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement](https://mlsanity.com/paper/2104.09958/) (2021)
- [ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation](https://mlsanity.com/paper/2111.10265/) (2021)
- [GIRAFFE HD: A High-Resolution 3D-aware Generative Model](https://mlsanity.com/paper/2203.14954/) (2022)
- [CLIP2StyleGAN: Unsupervised Extraction of StyleGAN Edit Directions](https://mlsanity.com/paper/2112.05219/) (2021)
- [Mind the Gap: Domain Gap Control for Single Shot Domain Adaptation for Generative Adversarial Networks](https://mlsanity.com/paper/2110.08398/) (2021)
- [Unsupervised Learning of Efficient Geometry-Aware Neural Articulated Representations](https://mlsanity.com/paper/2204.08839/) (2022)
- [Diffusion Models for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2306.09316/) (2023)
- [Unsupervised Salient Object Detection with Spectral Cluster Voting](https://mlsanity.com/paper/2203.12614/) (2022)
- [DiffusionSeg: Adapting Diffusion Towards Unsupervised Object Discovery](https://mlsanity.com/paper/2303.09813/) (2023)
- [Labels4Free: Unsupervised Segmentation using StyleGAN](https://mlsanity.com/paper/2103.14968/) (2021)
- [Finding an Unsupervised Image Segmenter in Each of Your Deep Generative Models](https://mlsanity.com/paper/2105.08127/) (2021)
- [Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motion](https://mlsanity.com/paper/2205.07844/) (2022)
- [TransFGU: A Top-down Approach to Fine-Grained Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2112.01515/) (2021)
- [Object-aware Contrastive Learning for Debiased Scene Representation](https://mlsanity.com/paper/2108.00049/) (2021)
- [Unsupervised Multi-View Object Segmentation Using Radiance Field Propagation](https://mlsanity.com/paper/2210.00489/) (2022)
- [Decoder Denoising Pretraining for Semantic Segmentation](https://mlsanity.com/paper/2205.11423/) (2022)
- [NamedMask: Distilling Segmenters from Complementary Foundation Models](https://mlsanity.com/paper/2209.11228/) (2022)
- [MOVE: Unsupervised Movable Object Segmentation and Detection](https://mlsanity.com/paper/2210.07920/) (2022)
- [GANSeg: Learning to Segment by Unsupervised Hierarchical Image Generation](https://mlsanity.com/paper/2112.01036/) (2021)
- [Drive&Segment: Unsupervised Semantic Segmentation of Urban Scenes via Cross-modal Distillation](https://mlsanity.com/paper/2203.11160/) (2022)
- [Promising or Elusive? Unsupervised Object Segmentation from Real-world Single Images](https://mlsanity.com/paper/2210.02324/) (2022)
- [Unsupervised Object Localization in the Era of Self-Supervised ViTs: A Survey](https://mlsanity.com/paper/2310.12904/) (2023)
- [Synthetic Data Supervised Salient Object Detection](https://mlsanity.com/paper/2210.13835/) (2022)
- [DeSplat: Decomposed Gaussian Splatting for Distractor-Free Rendering](https://mlsanity.com/paper/2411.19756/) (2024)
- [Unsupervised Image Segmentation by Mutual Information Maximization and Adversarial Regularization](https://mlsanity.com/paper/2107.00691/) (2021)
- [Unsupervised Multi-object Segmentation Using Attention and Soft-argmax](https://mlsanity.com/paper/2205.13271/) (2022)
- [UCOD-DPL: Unsupervised Camouflaged Object Detection via Dynamic Pseudo-label Learning](https://mlsanity.com/paper/2506.07087/) (2025)

## Abstract (arXiv)

We introduce a novel framework to build a model that can learn how to segment objects from a collection of images without any human annotation. Our method builds on the observation that the location of object segments can be perturbed locally relative to a given background without affecting the realism of a scene. Our approach is to first train a generative model of a layered scene. The layered representation consists of a background image, a foreground image and the mask of the foreground. A composite image is then obtained by overlaying the masked foreground image onto the background. The generative model is trained in an adversarial fashion against a discriminator, which forces the generative model to produce realistic composite images. To force the generator to learn a representation where the foreground layer corresponds to an object, we perturb the output of the generative model by introducing a random shift of both the foreground image and mask relative to the background. Because the generator is unaware of the shift before computing its output, it must produce layered representations that are realistic for any such random perturbation. Finally, we learn to segment an image by defining an autoencoder consisting of an encoder, which we train, and the pre-trained generator as the decoder, which we freeze. The encoder maps an image to a feature vector, which is fed as input to the generator to give a composite image matching the original input image. Because the generator outputs an explicit layered representation of the scene, the encoder learns to detect and segment objects. We demonstrate this framework on real images of several object categories.