mlsanityData to 7 Oct 2026

Pre-training without Natural Images

arXiv 2101.08515 ↗First posted 21 Jan 2021cs.CV cs.LGcited by 100

Not on the map. Why?

Abstract (arXiv)

Is it possible to use convolutional neural networks pre-trained without any natural images to assist natural image understanding? The paper proposes a novel concept, Formula-driven Supervised Learning. We automatically generate image patterns and their category labels by assigning fractals, which are based on a natural law existing in the background knowledge of the real world. Theoretically, the use of automatically generated images instead of natural images in the pre-training phase allows us to generate an infinite scale dataset of labeled images. Although the models pre-trained with the proposed Fractal DataBase (FractalDB), a database without natural images, does not necessarily outperform models pre-trained with human annotated datasets at all settings, we are able to partially surpass the accuracy of ImageNet/Places pre-trained models. The image representation with the proposed FractalDB captures a unique feature in the visualization of convolutional layers and attentions.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). Builds on 41 of its 68 references that are in the map. First posted on arXiv 21 Jan 2021.

Builds on

41 of 68 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2016Densely Connected Convolutional Networkscited by 6,691
  8. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  9. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  10. 2017Squeeze-and-Excitation Networkscited by 4,763
  11. 2014Going Deeper with Convolutionscited by 4,231
  12. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
29 more
  1. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  2. 2019Searching for MobileNetV3cited by 1,984
  3. 2017The Kinetics Human Action Video Datasetcited by 1,894
  4. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  5. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  6. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  7. 2016Colorful Image Colorizationcited by 1,113
  8. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  9. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  10. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  11. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  12. 2018Do Better ImageNet Models Transfer Better?cited by 560
  13. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  14. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 422
  15. 2017Learning from Synthetic Humanscited by 358
  16. 2016Learning from Simulated and Unsupervised Images through Adversarial Trainingcited by 287
  17. 2019Large Scale Adversarial Representation Learningcited by 238
  18. 2020Large image datasets: A pyrrhic win for computer vision?cited by 237
  19. 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
  20. 2018Moments in Time Dataset: one million videos for event understandingcited by 215
  21. 2017Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detectioncited by 209
  22. 2016What makes ImageNet good for transfer learning?cited by 192
  23. 2019Implicit 3D Orientation Learning for 6D Object Detection from RGB Imagescited by 170
  24. 2019Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchycited by 156
  25. 2017Representation Learning by Learning to Countcited by 120
  26. 2018Boosting Self-Supervised Learning via Knowledge Transfercited by 92
  27. 2019A critical analysis of self-supervision, or what we can learn from a single imagecited by 75
  28. 2018Learning to Segment via Cut-and-Pastecited by 42
  29. 2016How useful is photo-realistic rendering for visual learning?cited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 21; 2023: 20; 2024: 22; 2025: 13; 2026: 13.

100 papers.

Sort:

By area:

  1. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  2. 2022Fake it till you make it: Learning transferable representations from synthetic ImageNet clonescited by 171
  3. 2021PixMix: Dreamlike Pictures Comprehensively Improve Safety Measurescited by 131
  4. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  5. 2021Learning to See by Looking at Noisecited by 92
  6. 2022Towards Adversarial Evaluations for Inexact Machine Unlearningcited by 82
  7. 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
  8. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  9. 2023Predicting Gradient is Better: Exploring Self-Supervised Learning for SAR ATR with a Joint-Embedding Predictive Architecturecited by 43
  10. 2021Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Modelscited by 41
  11. 2024A Vision Check-up for Language Modelscited by 36
  12. 2021Can Vision Transformers Learn without Natural Images?cited by 32
38 more
  1. 2023Asynchronous Federated Continual Learningcited by 29
  2. 2024Learning Universal Predictorscited by 29
  3. 2022Procedural Image Programs for Representation Learningcited by 28
  4. 2021Improving Fractal Pre-trainingcited by 26
  5. 2023On the Connection between Pre-training Data Diversity and Fine-tuning Robustnesscited by 25
  6. 2021Post-training deep neural network pruning via layer-wise calibrationcited by 23
  7. 2022Pre-training Vision Transformers with Formula-driven Supervised Learningcited by 21
  8. 2022CNN Filter DB: An Empirical Investigation of Trained Convolutional Filterscited by 20
  9. 2023Visual Atoms: Pre-training Vision Transformers with Sinusoidal Wavescited by 20
  10. 2023Differentially Private Image Classification by Learning Priors from Random Processescited by 19
  11. 2022ExAug: Robot-Conditioned Navigation Policies via Geometric Experience Augmentationcited by 17
  12. 2023ViP: A Differentially Private Foundation Model for Computer Visioncited by 17
  13. 2022PSP-HDRI+: A Synthetic Dataset Generator for Pre-Training of Human-Centric Computer Vision Modelscited by 15
  14. 2026Training Language Models via Neural Cellular Automatacited by 15
  15. 2023IPMix: Label-Preserving Data Augmentation Method for Training Robust Classifierscited by 14
  16. 2024Uncertainty-guided Contrastive Learning for Single Source Domain Generalisationcited by 14
  17. 2024Scaling Backwards: Minimal Synthetic Pre-training?cited by 14
  18. 2023Pre-training Vision Transformers with Very Limited Synthesized Imagescited by 12
  19. 2022On the Role of Parallel Data in Cross-lingual Transfer Learningcited by 11
  20. 2022FundusQ-Net: a Regression Quality Assessment Deep Learning Algorithm for Fundus Images Quality Gradingcited by 9
  21. 2023Pre-training with Synthetic Data Helps Offline Reinforcement Learningcited by 9
  22. 2022Visual Pre-training for Navigation: What Can We Learn from Noise?cited by 8
  23. 2023SegRCDB: Semantic Segmentation via Formula-Driven Supervised Learningcited by 8
  24. 2025Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformerscited by 8
  25. 2021The Augmented Image Prior: Distilling 1000 Classes by Extrapolating from a Single Imagecited by 7
  26. 2022Palm up: Playing in the Latent Manifold for Unsupervised Pretrainingcited by 7
  27. 2022Fast and High-Quality Image Denoising via Malleable Convolutionscited by 6
  28. 2022Commonality in Natural Images Rescues GANs: Pretraining GANs with Generic and Privacy-free Synthetic Datacited by 6
  29. 2022Self-Similarity Priors: Neural Collages as Differentiable Fractal Representationscited by 6
  30. 2022OOD Augmentation May Be at Odds with Open-Set Recognitioncited by 6
  31. 2024Primitive Geometry Segment Pre-training for 3D Medical Image Segmentationcited by 6
  32. 2022Empirical Evaluation and Theoretical Analysis for Representation Learning: A Surveycited by 5
  33. 2022A Scene-Text Synthesis Engine Achieved Through Learning from Decomposed Real-World Datacited by 5
  34. 2023USIM-DAL: Uncertainty-aware Statistical Image Modeling-based Dense Active Learning for Super-resolutioncited by 4
  35. 2023Harnessing Synthetic Datasets: The Role of Shape Bias in Deep Neural Network Generalizationcited by 4
  36. 2024Content-Based Image Retrieval for Multi-Class Volumetric Radiology Images: A Benchmark Studycited by 4
  37. 2025Torch-Uncertainty: A Deep Learning Framework for Uncertainty Quantificationcited by 4
  38. 2023ExpPoint-MAE: Better interpretability and performance for self-supervised point cloud transformerscited by 3

50 most cited of 100 All 100 (JSON)