# Learning to See by Looking at Noise

arXiv: https://arxiv.org/abs/2106.05963 · HTML: https://mlsanity.com/paper/2106.05963/

## Key facts

- Cited by 92 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 68 references that are in the map.
- First posted on arXiv 10 Jun 2021.

## Builds on (most cited first, 30 of 33)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark](https://mlsanity.com/paper/1910.04867/) (2019)
- [SinGAN: Learning a Generative Model from a Single Natural Image](https://mlsanity.com/paper/1905.01164/) (2019)
- [DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort](https://mlsanity.com/paper/2104.06490/) (2021)
- [Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction](https://mlsanity.com/paper/1611.09842/) (2016)
- [MineRL: A Large-Scale Dataset of Minecraft Demonstrations](https://mlsanity.com/paper/1907.13440/) (2019)
- [Self-supervised Pretraining of Visual Features in the Wild](https://mlsanity.com/paper/2103.01988/) (2021)
- [Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization](https://mlsanity.com/paper/2104.05833/) (2021)
- [Classification Accuracy Score for Conditional Generative Models](https://mlsanity.com/paper/1905.10887/) (2019)
- [Generative Models as a Data Source for Multiview Representation Learning](https://mlsanity.com/paper/2106.05258/) (2021)
- [Pre-training without Natural Images](https://mlsanity.com/paper/2101.08515/) (2021)
- [Repurposing GANs for One-shot Semantic Part Segmentation](https://mlsanity.com/paper/2103.04379/) (2021)
- [A critical analysis of self-supervision, or what we can learn from a single image](https://mlsanity.com/paper/1904.13132/) (2019)

## Built on by (most cited first, 30 of 92)

- [The Platonic Representation Hypothesis](https://mlsanity.com/paper/2405.07987/) (2024)
- [StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners](https://mlsanity.com/paper/2306.00984/) (2023)
- [PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures](https://mlsanity.com/paper/2112.05135/) (2021)
- [Generalizing Dataset Distillation via Deep Generative Prior](https://mlsanity.com/paper/2305.01649/) (2023)
- [Scaling Laws of Synthetic Images for Model Training ... for Now](https://mlsanity.com/paper/2312.04567/) (2023)
- [On the Importance and Applicability of Pre-Training for Federated Learning](https://mlsanity.com/paper/2206.11488/) (2022)
- [GAN-Supervised Dense Visual Alignment](https://mlsanity.com/paper/2112.05143/) (2021)
- [FreeMask: Synthetic Images with Dense Annotations Make Stronger Segmentation Models](https://mlsanity.com/paper/2310.15160/) (2023)
- [Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2206.02465/) (2022)
- [A Vision Check-up for Language Models](https://mlsanity.com/paper/2401.01862/) (2024)
- [ModelDiff: A Framework for Comparing Learning Algorithms](https://mlsanity.com/paper/2211.12491/) (2022)
- [Learning General-Purpose Biomedical Volume Representations using Randomized Synthesis](https://mlsanity.com/paper/2411.02372/) (2024)
- [Procedural Image Programs for Representation Learning](https://mlsanity.com/paper/2211.16412/) (2022)
- [Task2Sim : Towards Effective Pre-training and Transfer from Synthetic Data](https://mlsanity.com/paper/2112.00054/) (2021)
- [Do Generated Data Always Help Contrastive Learning?](https://mlsanity.com/paper/2403.12448/) (2024)
- [Pre-training Vision Transformers with Formula-driven Supervised Learning](https://mlsanity.com/paper/2206.09132/) (2022)
- [Differentially Private Image Classification by Learning Priors from Random Processes](https://mlsanity.com/paper/2306.06076/) (2023)
- [The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better](https://mlsanity.com/paper/2406.05184/) (2024)
- [Deep Linear Probe Generators for Weight Space Learning](https://mlsanity.com/paper/2410.10811/) (2024)
- [FreeTumor: Advance Tumor Segmentation via Large-Scale Tumor Synthesis](https://mlsanity.com/paper/2406.01264/) (2024)
- [Training Language Models via Neural Cellular Automata](https://mlsanity.com/paper/2603.10055/) (2026)
- [IPMix: Label-Preserving Data Augmentation Method for Training Robust Classifiers](https://mlsanity.com/paper/2310.04780/) (2023)
- [Scaling Backwards: Minimal Synthetic Pre-training?](https://mlsanity.com/paper/2408.00677/) (2024)
- [General Intelligence Requires Reward-based Pretraining](https://mlsanity.com/paper/2502.19402/) (2025)
- [Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data](https://mlsanity.com/paper/2503.01167/) (2025)
- [DMLR: Data-centric Machine Learning Research -- Past, Present and Future](https://mlsanity.com/paper/2311.13028/) (2023)
- [Procedural Pretraining: Warming Up Language Models with Abstract Data](https://mlsanity.com/paper/2601.21725/) (2026)
- [What Makes a Good Dataset for Knowledge Distillation?](https://mlsanity.com/paper/2411.12817/) (2024)
- [Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning](https://mlsanity.com/paper/2505.22308/) (2025)
- [Visual Pre-training for Navigation: What Can We Learn from Noise?](https://mlsanity.com/paper/2207.00052/) (2022)

## Abstract (arXiv)

Current vision systems are trained on huge datasets, and these datasets come with costs: curation is expensive, they inherit human biases, and there are concerns over privacy and usage rights. To counter these costs, interest has surged in learning from cheaper data sources, such as unlabeled images. In this paper we go a step further and ask if we can do away with real image datasets entirely, instead learning from noise processes. We investigate a suite of image generation models that produce images from simple random processes. These are then used as training data for a visual representation learner with a contrastive loss. We study two types of noise processes, statistical image models and deep generative models under different random initializations. Our findings show that it is important for the noise to capture certain structural properties of real data but that good performance can be achieved even with processes that are far from realistic. We also find that diversity is a key property to learn good representations. Datasets, models, and code are available at https://mbaradad.github.io/learning_with_noise.