mlsanityData to 7 Oct 2026

How Useful is Self-Supervised Pretraining for Visual Tasks?

arXiv 2003.14323 ↗First posted 31 Mar 2020cs.CV cs.LGcited by 80

Not on the map. Why?

Abstract (arXiv)

Recent advances have spurred incredible progress in self-supervised pretraining for vision. We investigate what factors may play a role in the utility of these pretraining methods for practitioners. To do this, we evaluate various self-supervised algorithms across a comprehensive array of synthetic datasets and downstream tasks. We prepare a suite of synthetic data that enables an endless supply of annotated images as well as full control over dataset difficulty. Our experiments offer insights into how the utility of self-supervision changes as the number of available labels grows as well as how the utility changes as a function of the downstream task and the properties of the training data. We also find that linear evaluation does not correlate with finetuning performance. Code and data is available at hrefhttps://www.github.com/princeton-vl/selfstudygithub.com/princeton-vl/selfstudy.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 25; 2022: 20; 2023: 15; 2024: 10; 2025: 5; 2026: 5.

80 papers.

Sort:

By area:

  1. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  2. 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 224
  3. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  4. 2021Benchmarking Representation Learning for Natural World Image Collectionscited by 167
  5. 2021Self-Supervised Pretraining Improves Self-Supervised Pretrainingcited by 81
  6. 2021Transferable Visual Words: Exploiting the Semantics of Anatomical Patterns for Self-supervised Learningcited by 68
  7. 2021Self-Supervised Training Enhances Online Continual Learningcited by 53
  8. 2021Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learningcited by 47
  9. 2024SpectralEarth: Training Hyperspectral Foundation Models at Scalecited by 42
  10. 2022IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian languagescited by 38
  11. 2023MARBLE: Music Audio Representation Benchmark for Universal Evaluationcited by 38
  12. 2021Contrasting Contrastive Self-Supervised Representation Learning Pipelinescited by 37
38 more
  1. 2021Geographical Knowledge-driven Representation Learning for Remote Sensing Imagescited by 37
  2. 2021A Survey of Self-Supervised and Few-Shot Object Detectioncited by 35
  3. 2021Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretrainingcited by 32
  4. 2021How Transferable Are Self-supervised Features in Medical Image Classification Tasks?cited by 32
  5. 2023Learning Visual Representations via Language-Guided Samplingcited by 32
  6. 2023Overwriting Pretrained Bias with Finetuning Datacited by 32
  7. 2022A Closer Look at Self-Supervised Lightweight Vision Transformerscited by 31
  8. 2021How Well Does Self-Supervised Pre-Training Perform with Streaming Data?cited by 29
  9. 2021Scaled ReLU Matters for Training Vision Transformerscited by 29
  10. 2022Weakly Supervised Semantic Segmentation for Large-Scale Point Cloudcited by 27
  11. 2023The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learningcited by 27
  12. 2022Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Workscited by 24
  13. 2022How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?cited by 23
  14. 2021Object-Aware Cropping for Self-Supervised Learningcited by 22
  15. 2022Representation Uncertainty in Self-Supervised Learning as Variational Inferencecited by 22
  16. 2024A Closer Look at Benchmarking Self-Supervised Pre-training with Image Classificationcited by 19
  17. 2023Transformer-based Self-supervised Multimodal Representation Learning for Wearable Emotion Recognitioncited by 17
  18. 2021Self-supervised deep convolutional neural network for chest X-ray classificationcited by 16
  19. 2021A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective?cited by 14
  20. 2021RIO: Rotation-equivariance supervised learning of robust inertial odometrycited by 14
  21. 2022Self-supervised Learning for Label-Efficient Sleep Stage Classification: A Comprehensive Evaluationcited by 14
  22. 2025With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide Youcited by 13
  23. 2022Autonomy and Intelligence in the Computing Continuum: Challenges, Enablers, and Future Directions for Orchestrationcited by 12
  24. 2022Improving Fine-tuning of Self-supervised Models with Contrastive Initializationcited by 10
  25. 2023A Review of Predictive and Contrastive Self-supervised Learning for Medical Imagescited by 10
  26. 2021Contrastive Neural Processes for Self-Supervised Learningcited by 9
  27. 2023On the Difficulty of Defending Contrastive Learning against Backdoor Attackscited by 8
  28. 2024Probing the Mid-level Vision Capabilities of Self-Supervised Learningcited by 7
  29. 2022Commonality in Natural Images Rescues GANs: Pretraining GANs with Generic and Privacy-free Synthetic Datacited by 6
  30. 2024UDA-Bench: Revisiting Common Assumptions in Unsupervised Domain Adaptation Using a Standardized Frameworkcited by 6
  31. 2021Characterizing the adversarial vulnerability of speech self-supervised learningcited by 5
  32. 2022Empirical Evaluation and Theoretical Analysis for Representation Learning: A Surveycited by 5
  33. 2023On the Effectiveness of Speech Self-supervised Learning for Musiccited by 5
  34. 2024Using Self-supervised Learning Can Improve Model Fairnesscited by 5
  35. 2022Composite Learning for Robust and Effective Dense Predictionscited by 4
  36. 2023CL-MRI: Self-Supervised Contrastive Learning to Improve the Accuracy of Undersampled MRI Reconstructioncited by 4
  37. 2024An Experimental Study on Exploring Strong Lightweight Vision Transformers via Masked Image Modeling Pre-Trainingcited by 4
  38. 2025SEVERE++: Evaluating Benchmark Sensitivity in Generalization of Video Representation Learningcited by 4

50 most cited of 80 All 80 (JSON)