# A critical analysis of self-supervision, or what we can learn from a single image

arXiv: https://arxiv.org/abs/1904.13132 · HTML: https://mlsanity.com/paper/1904.13132/

## Key facts

- Cited by 76 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2019.

## Built on by (most cited first, 30 of 76)

- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion](https://mlsanity.com/paper/2210.10716/) (2022)
- [Neighborhood Contrastive Learning for Novel Class Discovery](https://mlsanity.com/paper/2106.10731/) (2021)
- [Are Large-scale Datasets Necessary for Self-Supervised Pre-training?](https://mlsanity.com/paper/2112.10740/) (2021)
- [Pre-training without Natural Images](https://mlsanity.com/paper/2101.08515/) (2021)
- [Learning to See by Looking at Noise](https://mlsanity.com/paper/2106.05963/) (2021)
- [Assessing the State of Self-Supervised Human Activity Recognition using Wearables](https://mlsanity.com/paper/2202.12938/) (2022)
- [Single Motion Diffusion](https://mlsanity.com/paper/2302.05905/) (2023)
- [PASS: An ImageNet replacement for self-supervised pretraining without humans](https://mlsanity.com/paper/2109.13228/) (2021)
- [Understanding Masked Image Modeling via Learning Occlusion Invariant Feature](https://mlsanity.com/paper/2208.04164/) (2022)
- [Negative Data Augmentation](https://mlsanity.com/paper/2102.05113/) (2021)
- [Lightweight Adaptive Feature De-drifting for Compressed Image Classification](https://mlsanity.com/paper/2401.01724/) (2024)
- [Neural Mean Discrepancy for Efficient Out-of-Distribution Detection](https://mlsanity.com/paper/2104.11408/) (2021)
- [SinDiffusion: Learning a Diffusion Model from a Single Natural Image](https://mlsanity.com/paper/2211.12445/) (2022)
- [Contrasting Contrastive Self-Supervised Representation Learning Pipelines](https://mlsanity.com/paper/2103.14005/) (2021)
- [Rethinking Graph Auto-Encoder Models for Attributed Graph Clustering](https://mlsanity.com/paper/2107.08562/) (2021)
- [Revisiting the Transferability of Supervised Pretraining: an MLP Perspective](https://mlsanity.com/paper/2112.00496/) (2021)
- [Space-Time Crop & Attend: Improving Cross-modal Video Representation Learning](https://mlsanity.com/paper/2103.10211/) (2021)
- [Self-supervised Character-to-Character Distillation for Text Recognition](https://mlsanity.com/paper/2211.00288/) (2022)
- [How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?](https://mlsanity.com/paper/2203.14221/) (2022)
- [Self-Supervised Multisensor Change Detection](https://mlsanity.com/paper/2103.05102/) (2021)
- [Acoustic word embeddings for zero-resource languages using self-supervised contrastive learning and multilingual adaptation](https://mlsanity.com/paper/2103.10731/) (2021)
- [Intermediate Layers Matter in Momentum Contrastive Self Supervised Learning](https://mlsanity.com/paper/2110.14805/) (2021)
- [SeamlessGAN: Self-Supervised Synthesis of Tileable Texture Maps](https://mlsanity.com/paper/2201.05120/) (2022)
- [CAiD: Context-Aware Instance Discrimination for Self-supervised Learning in Medical Imaging](https://mlsanity.com/paper/2204.07344/) (2022)
- [Barely-Supervised Learning: Semi-Supervised Learning with very few labeled images](https://mlsanity.com/paper/2112.12004/) (2021)
- [Objectives Matter: Understanding the Impact of Self-Supervised Objectives on Vision Transformer Representations](https://mlsanity.com/paper/2304.13089/) (2023)
- [You Don't Need Domain-Specific Data Augmentations When Scaling Self-Supervised Learning](https://mlsanity.com/paper/2406.09294/) (2024)
- [Scaling Backwards: Minimal Synthetic Pre-training?](https://mlsanity.com/paper/2408.00677/) (2024)
- [Towards Lightweight Black-Box Attacks against Deep Neural Networks](https://mlsanity.com/paper/2209.14826/) (2022)

## Abstract (arXiv)

We look critically at popular self-supervision techniques for learning deep convolutional neural networks without manual labels. We show that three different and representative methods, BiGAN, RotNet and DeepCluster, can learn the first few layers of a convolutional network from a single image as well as using millions of images and manual labels, provided that strong data augmentation is used. However, for deeper layers the gap with manual supervision cannot be closed even if millions of unlabelled images are used for training. We conclude that: (1) the weights of the early layers of deep networks contain limited information about the statistics of natural images, that (2) such low-level statistics can be learned through self-supervision just as well as through strong supervision, and that (3) the low-level statistics can be captured via synthetic transformations instead of using a large image dataset.