# Can Vision Transformers Learn without Natural Images?

arXiv: https://arxiv.org/abs/2103.13023 · HTML: https://mlsanity.com/paper/2103.13023/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 42 references that are in the map.
- First posted on arXiv 24 Mar 2021.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Rethinking ImageNet Pre-training](https://mlsanity.com/paper/1811.08883/) (2018)
- [Meta Pseudo Labels](https://mlsanity.com/paper/2003.10580/) (2020)
- [Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchy](https://mlsanity.com/paper/1912.07726/) (2019)

## Built on by (most cited first, 30 of 32)

- [PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures](https://mlsanity.com/paper/2112.05135/) (2021)
- [Procedural Image Programs for Representation Learning](https://mlsanity.com/paper/2211.16412/) (2022)
- [EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithm](https://mlsanity.com/paper/2206.09325/) (2022)
- [Post-training deep neural network pruning via layer-wise calibration](https://mlsanity.com/paper/2104.15023/) (2021)
- [Pre-training Vision Transformers with Formula-driven Supervised Learning](https://mlsanity.com/paper/2206.09132/) (2022)
- [Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model](https://mlsanity.com/paper/2105.15089/) (2021)
- [Visual Atoms: Pre-training Vision Transformers with Sinusoidal Waves](https://mlsanity.com/paper/2303.01112/) (2023)
- [ExAug: Robot-Conditioned Navigation Policies via Geometric Experience Augmentation](https://mlsanity.com/paper/2210.07450/) (2022)
- [Pre-training Vision Transformers with Very Limited Synthesized Images](https://mlsanity.com/paper/2307.14710/) (2023)
- [Visual Pre-training for Navigation: What Can We Learn from Noise?](https://mlsanity.com/paper/2207.00052/) (2022)
- [SegRCDB: Semantic Segmentation via Formula-Driven Supervised Learning](https://mlsanity.com/paper/2309.17083/) (2023)
- [OOD Augmentation May Be at Odds with Open-Set Recognition](https://mlsanity.com/paper/2206.04242/) (2022)
- [Primitive Geometry Segment Pre-training for 3D Medical Image Segmentation](https://mlsanity.com/paper/2401.03665/) (2024)
- [Efficient Diffusion-Driven Corruption Editor for Test-Time Adaptation](https://mlsanity.com/paper/2403.10911/) (2024)
- [Learning Fractals by Gradient Descent](https://mlsanity.com/paper/2303.12722/) (2023)
- [Data Collection-free Masked Video Modeling](https://mlsanity.com/paper/2409.06665/) (2024)
- [FractalAD: A simple industrial anomaly detection method using fractal anomaly generation and backbone knowledge distillation](https://mlsanity.com/paper/2301.12739/) (2023)
- [Generalized Face Forgery Detection via Adaptive Learning for Pre-trained Vision Transformer](https://mlsanity.com/paper/2309.11092/) (2023)
- [Complex Style Image Transformations for Domain Generalization in Medical Images](https://mlsanity.com/paper/2406.00298/) (2024)
- [Predicting Winning Captions for Weekly New Yorker Comics](https://mlsanity.com/paper/2407.18949/) (2024)
- [Pre-training for Action Recognition with Automatically Generated Fractal Datasets](https://mlsanity.com/paper/2411.17584/) (2024)
- [VibrantLeaves: A principled parametric image generator for training deep restoration models](https://mlsanity.com/paper/2504.10201/) (2025)
- [Stylized Structural Patterns for Improved Neural Network Pre-training](https://mlsanity.com/paper/2506.19465/) (2025)
- [Self-supervised Synthetic Pretraining for Inference of Stellar Mass Embedded in Dense Gas](https://mlsanity.com/paper/2510.24159/) (2025)
- [Detect Fake with Fake: Leveraging Synthetic Data-driven Representation for Synthetic Image Detection](https://mlsanity.com/paper/2409.08884/) (2024)
- [LayerMix: Enhanced Data Augmentation through Fractal Integration for Robust Deep Learning](https://mlsanity.com/paper/2501.04861/) (2025)
- [Formula-Supervised Sound Event Detection: Pre-Training Without Real Data](https://mlsanity.com/paper/2504.04428/) (2025)
- [Industrial Synthetic Segment Pre-training](https://mlsanity.com/paper/2505.13099/) (2025)
- [Separating Knowledge and Perception with Procedural Data](https://mlsanity.com/paper/2508.11697/) (2025)
- [Symmetria: A Synthetic Dataset for Learning in Point Clouds](https://mlsanity.com/paper/2510.23414/) (2025)

## Abstract (arXiv)

Can we complete pre-training of Vision Transformers (ViT) without natural images and human-annotated labels? Although a pre-trained ViT seems to heavily rely on a large-scale dataset and human-annotated labels, recent large-scale datasets contain several problems in terms of privacy violations, inadequate fairness protection, and labor-intensive annotation. In the present paper, we pre-train ViT without any image collections and annotation labor. We experimentally verify that our proposed framework partially outperforms sophisticated Self-Supervised Learning (SSL) methods like SimCLRv2 and MoCov2 without using any natural images in the pre-training phase. Moreover, although the ViT pre-trained without natural images produces some different visualizations from ImageNet pre-trained ViT, it can interpret natural image datasets to a large extent. For example, the performance rates on the CIFAR-10 dataset are as follows: our proposal 97.6 vs. SimCLRv2 97.4 vs. ImageNet 98.0.