# Repurposing GANs for One-shot Semantic Part Segmentation

arXiv: https://arxiv.org/abs/2103.04379 · HTML: https://mlsanity.com/paper/2103.04379/

## Key facts

- Cited by 88 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 83 references that are in the map.
- First posted on arXiv 7 Mar 2021.

## Builds on (most cited first, 30 of 61)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Interpreting the Latent Space of GANs for Semantic Face Editing](https://mlsanity.com/paper/1907.10786/) (2019)
- [Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?](https://mlsanity.com/paper/1904.03189/) (2019)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment](https://mlsanity.com/paper/1908.06391/) (2019)
- [DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition](https://mlsanity.com/paper/1310.1531/) (2013)
- [One-Shot Learning for Semantic Segmentation](https://mlsanity.com/paper/1709.03410/) (2017)
- [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014)
- [Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks](https://mlsanity.com/paper/1406.6909/) (2014)

## Built on by (most cited first, 30 of 88)

- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Label-Efficient Semantic Segmentation with Diffusion Models](https://mlsanity.com/paper/2112.03126/) (2021)
- [Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.04803/) (2023)
- [Synthetic Data from Diffusion Models Improves ImageNet Classification](https://mlsanity.com/paper/2304.08466/) (2023)
- [Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold](https://mlsanity.com/paper/2305.10973/) (2023)
- [Dataset Diffusion: Diffusion-based Synthetic Dataset Generation for Pixel-Level Semantic Segmentation](https://mlsanity.com/paper/2309.14303/) (2023)
- [Generative Models as a Data Source for Multiview Representation Learning](https://mlsanity.com/paper/2106.05258/) (2021)
- [Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentation](https://mlsanity.com/paper/2305.16289/) (2023)
- [BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations](https://mlsanity.com/paper/2201.04684/) (2022)
- [Learning to See by Looking at Noise](https://mlsanity.com/paper/2106.05963/) (2021)
- [CLIP2StyleGAN: Unsupervised Extraction of StyleGAN Edit Directions](https://mlsanity.com/paper/2112.05219/) (2021)
- [Mind the Gap: Domain Gap Control for Single Shot Domain Adaptation for Generative Adversarial Networks](https://mlsanity.com/paper/2110.08398/) (2021)
- [Real-Time Radiance Fields for Single-Image Portrait View Synthesis](https://mlsanity.com/paper/2305.02310/) (2023)
- [GAN-Supervised Dense Visual Alignment](https://mlsanity.com/paper/2112.05143/) (2021)
- [Unsupervised Part Discovery from Contrastive Reconstruction](https://mlsanity.com/paper/2111.06349/) (2021)
- [Ensembling with Deep Generative Views](https://mlsanity.com/paper/2104.14551/) (2021)
- [EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models](https://mlsanity.com/paper/2401.11739/) (2024)
- [Exploiting Diffusion Prior for Generalizable Dense Prediction](https://mlsanity.com/paper/2311.18832/) (2023)
- [EditWorld: Simulating World Dynamics for Instruction-Following Image Editing](https://mlsanity.com/paper/2405.14785/) (2024)
- [State-of-the-Art in the Architecture, Methods and Applications of StyleGAN](https://mlsanity.com/paper/2202.14020/) (2022)
- [SLiMe: Segment Like Me](https://mlsanity.com/paper/2309.03179/) (2023)
- [Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Models](https://mlsanity.com/paper/2401.12978/) (2024)
- [LD-ZNet: A Latent Diffusion Approach for Text-Based Image Segmentation](https://mlsanity.com/paper/2303.12343/) (2023)
- [User-Controllable Latent Transformer for StyleGAN Image Layout Editing](https://mlsanity.com/paper/2208.12408/) (2022)
- [PanopticPartFormer++: A Unified and Decoupled View for Panoptic Part Segmentation](https://mlsanity.com/paper/2301.00954/) (2023)
- [CHORUS: Learning Canonicalized 3D Human-Object Spatial Relations from Unbounded Synthesized Images](https://mlsanity.com/paper/2308.12288/) (2023)
- [Few Shot Part Segmentation Reveals Compositional Logic for Industrial Anomaly Detection](https://mlsanity.com/paper/2312.13783/) (2023)
- [Synth2: Boosting Visual-Language Models with Synthetic Captions and Image Embeddings](https://mlsanity.com/paper/2403.07750/) (2024)
- [Synthetic Data Supervised Salient Object Detection](https://mlsanity.com/paper/2210.13835/) (2022)
- [Towards Open-World Segmentation of Parts](https://mlsanity.com/paper/2305.16804/) (2023)

## Abstract (arXiv)

While GANs have shown success in realistic image generation, the idea of using GANs for other tasks unrelated to synthesis is underexplored. Do GANs learn meaningful structural parts of objects during their attempt to reproduce those objects? In this work, we test this hypothesis and propose a simple and effective approach based on GANs for semantic part segmentation that requires as few as one label example along with an unlabeled dataset. Our key idea is to leverage a trained GAN to extract pixel-wise representation from the input image and use it as feature vectors for a segmentation network. Our experiments demonstrate that GANs representation is "readily discriminative" and produces surprisingly good results that are comparable to those from supervised baselines trained with significantly more labels. We believe this novel repurposing of GANs underlies a new class of unsupervised representation learning that is applicable to many other tasks. More results are available at https://repurposegans.github.io/.