# Generative Models as a Data Source for Multiview Representation Learning

arXiv: https://arxiv.org/abs/2106.05258 · HTML: https://mlsanity.com/paper/2106.05258/

## Key facts

- Cited by 104 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 90 references that are in the map.
- First posted on arXiv 9 Jun 2021.

## Builds on (most cited first, 30 of 62)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Interpreting the Latent Space of GANs for Semantic Face Editing](https://mlsanity.com/paper/1907.10786/) (2019)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Contrastive Multi-View Representation Learning on Graphs](https://mlsanity.com/paper/2006.05582/) (2020)
- [GANSpace: Discovering Interpretable GAN Controls](https://mlsanity.com/paper/2004.02546/) (2020)
- [Do Better ImageNet Models Transfer Better?](https://mlsanity.com/paper/1805.08974/) (2018)

## Built on by (most cited first, 30 of 104)

- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [Is synthetic data from generative models ready for image recognition?](https://mlsanity.com/paper/2210.07574/) (2022)
- [Effective Data Augmentation With Diffusion Models](https://mlsanity.com/paper/2302.07944/) (2023)
- [StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners](https://mlsanity.com/paper/2306.00984/) (2023)
- [Fake it till you make it: Learning transferable representations from synthetic ImageNet clones](https://mlsanity.com/paper/2212.08420/) (2022)
- [URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Images](https://mlsanity.com/paper/2405.11656/) (2024)
- [Scaling Laws of Synthetic Images for Model Training ... for Now](https://mlsanity.com/paper/2312.04567/) (2023)
- [Learning to See by Looking at Noise](https://mlsanity.com/paper/2106.05963/) (2021)
- [Leaving Reality to Imagination: Robust Classification via Generated Datasets](https://mlsanity.com/paper/2302.02503/) (2023)
- [Vision-Language Models Do Not Understand Negation](https://mlsanity.com/paper/2501.09425/) (2025)
- [Learning Vision from Models Rivals Learning Vision from Data](https://mlsanity.com/paper/2312.17742/) (2023)
- [SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?](https://mlsanity.com/paper/2402.01832/) (2024)
- [GAN-Supervised Dense Visual Alignment](https://mlsanity.com/paper/2112.05143/) (2021)
- [Expanding Small-Scale Datasets with Guided Imagination](https://mlsanity.com/paper/2211.13976/) (2022)
- [FreeMask: Synthetic Images with Dense Annotations Make Stronger Segmentation Models](https://mlsanity.com/paper/2310.15160/) (2023)
- [Generative Models: What Do They Know? Do They Know Things? Let's Find Out!](https://mlsanity.com/paper/2311.17137/) (2023)
- [StyleGAN knows Normal, Depth, Albedo, and More](https://mlsanity.com/paper/2306.00987/) (2023)
- [Diffusion Models and Semi-Supervised Learners Benefit Mutually with Few Labels](https://mlsanity.com/paper/2302.10586/) (2023)
- [Self-Correcting Self-Consuming Loops for Generative Model Training](https://mlsanity.com/paper/2402.07087/) (2024)
- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [Fill-Up: Balancing Long-Tailed Data with Generative Models](https://mlsanity.com/paper/2306.07200/) (2023)
- [Feedback-guided Data Synthesis for Imbalanced Classification](https://mlsanity.com/paper/2310.00158/) (2023)
- [Do text-free diffusion models learn discriminative visual representations?](https://mlsanity.com/paper/2311.17921/) (2023)
- [A General Protocol to Probe Large Vision Models for 3D Physical Understanding](https://mlsanity.com/paper/2310.06836/) (2023)
- [Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Models](https://mlsanity.com/paper/2401.12978/) (2024)
- [Class-Incremental Learning using Diffusion Model for Distillation and Replay](https://mlsanity.com/paper/2306.17560/) (2023)
- [PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learning](https://mlsanity.com/paper/2308.03977/) (2023)
- [textitRevelio: Interpreting and leveraging semantic information in diffusion models](https://mlsanity.com/paper/2411.16725/) (2024)
- [Procedural Image Programs for Representation Learning](https://mlsanity.com/paper/2211.16412/) (2022)
- [Covariance Matrix Adaptation MAP-Annealing](https://mlsanity.com/paper/2205.10752/) (2022)

## Abstract (arXiv)

Generative models are now capable of producing highly realistic images that look nearly indistinguishable from the data on which they are trained. This raises the question: if we have good enough generative models, do we still need datasets? We investigate this question in the setting of learning general-purpose visual representations from a black-box generative model rather than directly from data. Given an off-the-shelf image generator without any access to its training data, we train representations from the samples output by this generator. We compare several representation learning methods that can be applied to this setting, using the latent space of the generator to generate multiple "views" of the same semantic content. We show that for contrastive methods, this multiview data can naturally be used to identify positive pairs (nearby in latent space) and negative pairs (far apart in latent space). We find that the resulting representations rival or even outperform those learned directly from real data, but that good performance requires care in the sampling strategy applied and the training method. Generative models can be viewed as a compressed and organized copy of a dataset, and we envision a future where more and more "model zoos" proliferate while datasets become increasingly unwieldy, missing, or private. This paper suggests several techniques for dealing with visual representation learning in such a future. Code is available on our project page https://ali-design.github.io/GenRep/.