mlsanityData to 7 Oct 2026

Can Vision Transformers Learn without Natural Images?

arXiv 2103.13023 ↗First posted 24 Mar 2021cs.CVcited by 32

Not on the map. Why?

Abstract (arXiv)

Can we complete pre-training of Vision Transformers (ViT) without natural images and human-annotated labels? Although a pre-trained ViT seems to heavily rely on a large-scale dataset and human-annotated labels, recent large-scale datasets contain several problems in terms of privacy violations, inadequate fairness protection, and labor-intensive annotation. In the present paper, we pre-train ViT without any image collections and annotation labor. We experimentally verify that our proposed framework partially outperforms sophisticated Self-Supervised Learning (SSL) methods like SimCLRv2 and MoCov2 without using any natural images in the pre-training phase. Moreover, although the ViT pre-trained without natural images produces some different visualizations from ImageNet pre-trained ViT, it can interpret natural image datasets to a large extent. For example, the performance rates on the CIFAR-10 dataset are as follows: our proposal 97.6 vs. SimCLRv2 97.4 vs. ImageNet 98.0.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 33 of its 42 references that are in the map. First posted on arXiv 24 Mar 2021.

Builds on

33 of 42 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2016Densely Connected Convolutional Networkscited by 6,691
  11. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  12. 2020Scaling Laws for Neural Language Modelscited by 6,411
21 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  2. 2014Going Deeper with Convolutionscited by 4,231
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  4. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  5. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  6. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  7. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  8. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  9. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  10. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  11. 2016Colorful Image Colorizationcited by 1,113
  12. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  13. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  14. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  15. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  16. 2018Rethinking ImageNet Pre-trainingcited by 376
  17. 2020Meta Pseudo Labelscited by 372
  18. 2019Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchycited by 156
  19. 2017Representation Learning by Learning to Countcited by 120
  20. 2021Pre-training without Natural Imagescited by 100
  21. 2018Boosting Self-Supervised Learning via Knowledge Transfercited by 92

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 6; 2023: 6; 2024: 7; 2025: 9; 2026: 1.

32 papers.

Sort:

By area:

  1. 2021PixMix: Dreamlike Pictures Comprehensively Improve Safety Measurescited by 131
  2. 2022Procedural Image Programs for Representation Learningcited by 28
  3. 2022EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithmcited by 27
  4. 2021Post-training deep neural network pruning via layer-wise calibrationcited by 23
  5. 2022Pre-training Vision Transformers with Formula-driven Supervised Learningcited by 21
  6. 2021Analogous to Evolutionary Algorithm: Designing a Unified Sequence Modelcited by 20
  7. 2023Visual Atoms: Pre-training Vision Transformers with Sinusoidal Wavescited by 20
  8. 2022ExAug: Robot-Conditioned Navigation Policies via Geometric Experience Augmentationcited by 17
  9. 2023Pre-training Vision Transformers with Very Limited Synthesized Imagescited by 12
  10. 2022Visual Pre-training for Navigation: What Can We Learn from Noise?cited by 8
  11. 2023SegRCDB: Semantic Segmentation via Formula-Driven Supervised Learningcited by 8
  12. 2022OOD Augmentation May Be at Odds with Open-Set Recognitioncited by 6
20 more
  1. 2024Primitive Geometry Segment Pre-training for 3D Medical Image Segmentationcited by 6
  2. 2024Efficient Diffusion-Driven Corruption Editor for Test-Time Adaptationcited by 4
  3. 2023Learning Fractals by Gradient Descentcited by 2
  4. 2023FractalAD: A simple industrial anomaly detection method using fractal anomaly generation and backbone knowledge distillationcited by 1
  5. 2023Generalized Face Forgery Detection via Adaptive Learning for Pre-trained Vision Transformercited by 1
  6. 2024Complex Style Image Transformations for Domain Generalization in Medical Imagescited by 1
  7. 2024Predicting Winning Captions for Weekly New Yorker Comicscited by 1
  8. 2024Data Collection-free Masked Video Modelingcited by 1
  9. 2024Pre-training for Action Recognition with Automatically Generated Fractal Datasetscited by 1
  10. 2025VibrantLeaves: A principled parametric image generator for training deep restoration modelscited by 1
  11. 2025Stylized Structural Patterns for Improved Neural Network Pre-trainingcited by 1
  12. 2025Self-supervised Synthetic Pretraining for Inference of Stellar Mass Embedded in Dense Gascited by 1
  13. 2024Detect Fake with Fake: Leveraging Synthetic Data-driven Representation for Synthetic Image Detectioncited by 0
  14. 2025LayerMix: Enhanced Data Augmentation through Fractal Integration for Robust Deep Learningcited by 0
  15. 2025Formula-Supervised Sound Event Detection: Pre-Training Without Real Datacited by 0
  16. 2025Industrial Synthetic Segment Pre-trainingcited by 0
  17. 2025Separating Knowledge and Perception with Procedural Datacited by 0
  18. 2025Symmetria: A Synthetic Dataset for Learning in Point Cloudscited by 0
  19. 2025Towards Provably Unlearnable Examples via Bayes Error Optimisationcited by 0
  20. 2026IDeaL: Data-Free Multi-Teacher Distillation via Improved Dead Leavescited by 0