# DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort

arXiv: https://arxiv.org/abs/2104.06490 · HTML: https://mlsanity.com/paper/2104.06490/

## Key facts

- Cited by 264 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 59 references that are in the map.
- First posted on arXiv 13 Apr 2021.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/1811.12833/) (2018)
- [Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results](https://mlsanity.com/paper/1703.01780/) (2017)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer](https://mlsanity.com/paper/1908.01210/) (2019)
- [Adversarial Learning for Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/1802.07934/) (2018)
- [Image to Image Translation for Domain Adaptation](https://mlsanity.com/paper/1712.00479/) (2017)
- [Semi-Supervised Semantic Segmentation with High- and Low-level Consistency](https://mlsanity.com/paper/1908.05724/) (2019)
- [Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization](https://mlsanity.com/paper/2104.05833/) (2021)
- [Meta-Sim: Learning to Generate Synthetic Datasets](https://mlsanity.com/paper/1904.11621/) (2019)
- [Self-Ensembling with GAN-based Data Augmentation for Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/1909.00589/) (2019)

## Built on by (most cited first, 30 of 264)

- [Label-Efficient Semantic Segmentation with Diffusion Models](https://mlsanity.com/paper/2112.03126/) (2021)
- [GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images](https://mlsanity.com/paper/2209.11163/) (2022)
- [Emergent Correspondence from Image Diffusion](https://mlsanity.com/paper/2306.03881/) (2023)
- [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022)
- [Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.04803/) (2023)
- [MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations](https://mlsanity.com/paper/2310.17596/) (2023)
- [Is synthetic data from generative models ready for image recognition?](https://mlsanity.com/paper/2210.07574/) (2022)
- [Your Diffusion Model is Secretly a Zero-Shot Classifier](https://mlsanity.com/paper/2303.16203/) (2023)
- [Effective Data Augmentation With Diffusion Models](https://mlsanity.com/paper/2302.07944/) (2023)
- [Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers](https://mlsanity.com/paper/2111.13587/) (2021)
- [Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold](https://mlsanity.com/paper/2305.10973/) (2023)
- [DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models](https://mlsanity.com/paper/2303.11681/) (2023)
- [EditGAN: High-Precision Semantic Image Editing](https://mlsanity.com/paper/2111.03186/) (2021)
- [Localizing Object-level Shape Variations with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.11306/) (2023)
- [DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models](https://mlsanity.com/paper/2308.06160/) (2023)
- [SemMAE: Semantic-Guided Masking for Learning Masked Autoencoders](https://mlsanity.com/paper/2206.10207/) (2022)
- [Dataset Diffusion: Diffusion-based Synthetic Dataset Generation for Pixel-Level Semantic Segmentation](https://mlsanity.com/paper/2309.14303/) (2023)
- [Generative Models as a Data Source for Multiview Representation Learning](https://mlsanity.com/paper/2106.05258/) (2021)
- [Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentation](https://mlsanity.com/paper/2305.16289/) (2023)
- [Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations](https://mlsanity.com/paper/2310.07849/) (2023)
- [BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations](https://mlsanity.com/paper/2201.04684/) (2022)
- [Diffusion Model as Representation Learner](https://mlsanity.com/paper/2308.10916/) (2023)
- [Learning to See by Looking at Noise](https://mlsanity.com/paper/2106.05963/) (2021)
- [CLIP2StyleGAN: Unsupervised Extraction of StyleGAN Edit Directions](https://mlsanity.com/paper/2112.05219/) (2021)
- [Synthesizing Informative Training Samples with GAN](https://mlsanity.com/paper/2204.07513/) (2022)
- [Adaptive Shells for Efficient Neural Radiance Field Rendering](https://mlsanity.com/paper/2311.10091/) (2023)
- [Mind the Gap: Domain Gap Control for Single Shot Domain Adaptation for Generative Adversarial Networks](https://mlsanity.com/paper/2110.08398/) (2021)
- [Real-Time Radiance Fields for Single-Image Portrait View Synthesis](https://mlsanity.com/paper/2305.02310/) (2023)
- [GAN-Supervised Dense Visual Alignment](https://mlsanity.com/paper/2112.05143/) (2021)
- [Expanding Small-Scale Datasets with Guided Imagination](https://mlsanity.com/paper/2211.13976/) (2022)

## Abstract (arXiv)

We introduce DatasetGAN: an automatic procedure to generate massive datasets of high-quality semantically segmented images requiring minimal human effort. Current deep networks are extremely data-hungry, benefiting from training on large-scale datasets, which are time consuming to annotate. Our method relies on the power of recent GANs to generate realistic images. We show how the GAN latent code can be decoded to produce a semantic segmentation of the image. Training the decoder only needs a few labeled examples to generalize to the rest of the latent space, resulting in an infinite annotated dataset generator! These generated datasets can then be used for training any computer vision architecture just as real datasets are. As only a few images need to be manually segmented, it becomes possible to annotate images in extreme detail and generate datasets with rich object and part segmentations. To showcase the power of our approach, we generated datasets for 7 image segmentation tasks which include pixel-level labels for 34 human face parts, and 32 car parts. Our approach outperforms all semi-supervised baselines significantly and is on par with fully supervised methods, which in some cases require as much as 100x more annotated data as our method.