mlsanityData to 14 Oct 2026

A critical analysis of self-supervision, or what we can learn from a single image

arXiv 1904.13132 ↗First posted 30 Apr 2019cs.CVcited by 76

Not on the map. Why?

Abstract (arXiv)

We look critically at popular self-supervision techniques for learning deep convolutional neural networks without manual labels. We show that three different and representative methods, BiGAN, RotNet and DeepCluster, can learn the first few layers of a convolutional network from a single image as well as using millions of images and manual labels, provided that strong data augmentation is used. However, for deeper layers the gap with manual supervision cannot be closed even if millions of unlabelled images are used for training. We conclude that: (1) the weights of the early layers of deep networks contain limited information about the statistics of natural images, that (2) such low-level statistics can be learned through self-supervision just as well as through strong supervision, and that (3) the low-level statistics can be captured via synthetic transformations instead of using a large image dataset.

Key facts. Cited by 76 papers in the map (data through 14 Oct 2026). First posted on arXiv 30 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 33; 2022: 17; 2023: 11; 2024: 8; 2025: 4; 2026: 3.

76 papers.

Sort:

By area:

  1. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
  2. 2022CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completioncited by 160
  3. 2021Neighborhood Contrastive Learning for Novel Class Discoverycited by 121
  4. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  5. 2021Pre-training without Natural Imagescited by 100
  6. 2021Learning to See by Looking at Noisecited by 92
  7. 2022Assessing the State of Self-Supervised Human Activity Recognition using Wearablescited by 65
  8. 2023Single Motion Diffusioncited by 65
  9. 2021PASS: An ImageNet replacement for self-supervised pretraining without humanscited by 51
  10. 2022Understanding Masked Image Modeling via Learning Occlusion Invariant Featurecited by 50
  11. 2021Negative Data Augmentationcited by 46
  12. 2024Lightweight Adaptive Feature De-drifting for Compressed Image Classificationcited by 46
38 more
  1. 2021Neural Mean Discrepancy for Efficient Out-of-Distribution Detectioncited by 42
  2. 2022SinDiffusion: Learning a Diffusion Model from a Single Natural Imagecited by 42
  3. 2021Contrasting Contrastive Self-Supervised Representation Learning Pipelinescited by 37
  4. 2021Rethinking Graph Auto-Encoder Models for Attributed Graph Clusteringcited by 37
  5. 2021Revisiting the Transferability of Supervised Pretraining: an MLP Perspectivecited by 36
  6. 2021Space-Time Crop & Attend: Improving Cross-modal Video Representation Learningcited by 31
  7. 2022Self-supervised Character-to-Character Distillation for Text Recognitioncited by 27
  8. 2022How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?cited by 23
  9. 2021Self-Supervised Multisensor Change Detectioncited by 19
  10. 2021Acoustic word embeddings for zero-resource languages using self-supervised contrastive learning and multilingual adaptationcited by 19
  11. 2021Intermediate Layers Matter in Momentum Contrastive Self Supervised Learningcited by 19
  12. 2022SeamlessGAN: Self-Supervised Synthesis of Tileable Texture Mapscited by 19
  13. 2022CAiD: Context-Aware Instance Discrimination for Self-supervised Learning in Medical Imagingcited by 18
  14. 2021Barely-Supervised Learning: Semi-Supervised Learning with very few labeled imagescited by 17
  15. 2023Objectives Matter: Understanding the Impact of Self-Supervised Objectives on Vision Transformer Representationscited by 16
  16. 2024You Don't Need Domain-Specific Data Augmentations When Scaling Self-Supervised Learningcited by 16
  17. 2024Scaling Backwards: Minimal Synthetic Pre-training?cited by 14
  18. 2022Towards Lightweight Black-Box Attacks against Deep Neural Networkscited by 13
  19. 2023R-MAE: Regions Meet Masked Autoencoderscited by 13
  20. 2022On the Role of Parallel Data in Cross-lingual Transfer Learningcited by 11
  21. 2023Learning from One Continuous Video Streamcited by 11
  22. 2023Understanding Masked Autoencoders From a Local Contrastive Perspectivecited by 10
  23. 2021CharacterGAN: Few-Shot Keypoint Character Animation and Reposingcited by 9
  24. 2021Contrastive Neural Processes for Self-Supervised Learningcited by 9
  25. 2022Visual Pre-training for Navigation: What Can We Learn from Noise?cited by 8
  26. 2025Clustering Properties of Self-Supervised Learningcited by 8
  27. 2021Using Self-Supervised Auxiliary Tasks to Improve Fine-Grained Facial Representationcited by 7
  28. 2021Re-using Adversarial Mask Discriminators for Test-time Training under Distribution Shiftscited by 7
  29. 2021The Augmented Image Prior: Distilling 1000 Classes by Extrapolating from a Single Imagecited by 7
  30. 2024Universal Facial Encoding of Codec Avatars from VR Headsetscited by 7
  31. 2021An Empirical Study and Analysis on Open-Set Semi-Supervised Learningcited by 6
  32. 2023Adaptive Data Augmentation for Contrastive Learningcited by 6
  33. 2025RINO: Renormalization Group Invariance with No Labelscited by 6
  34. 2022Spatial Entropy as an Inductive Bias for Vision Transformerscited by 5
  35. 2024MedMerge: Merging Models for Effective Transfer Learning to Medical Imaging Taskscited by 5
  36. 2021Stop Throwing Away Discriminators! Re-using Adversaries for Test-Time Trainingcited by 4
  37. 2022Self-Supervised Pretraining for Differentially Private Learningcited by 4
  38. 2025SEVERE++: Evaluating Benchmark Sensitivity in Generalization of Video Representation Learningcited by 4

50 most cited of 76 All 76 (JSON)