mlsanityData to 7 Oct 2026

Self-Supervised Pretraining Improves Self-Supervised Pretraining

arXiv 2103.12718 ↗First posted 23 Mar 2021cs.CVcited by 81

Not on the map. Why?

Abstract (arXiv)

While self-supervised pretraining has proven beneficial for many computer vision tasks, it requires expensive and lengthy computation, large amounts of data, and is sensitive to data augmentation. Prior work demonstrates that models pretrained on datasets dissimilar to their target data, such as chest X-ray models trained on ImageNet, underperform models trained from scratch. Users that lack the resources to pretrain must use existing models with lower performance. This paper explores Hierarchical PreTraining (HPT), which decreases convergence time and improves accuracy by initializing the pretraining process with an existing pretrained model. Through experimentation on 16 diverse vision datasets, we show HPT converges up to 80x faster, improves accuracy across tasks, and improves the robustness of the self-supervised pretraining process to changes in the image augmentation policy or amount of pretraining data. Taken together, HPT provides a simple framework for obtaining better pretrained representations with less computational resources.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Builds on 55 of its 72 references that are in the map. First posted on arXiv 23 Mar 2021.

Builds on

55 of 72 references.

References · 55

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2017Mask R-CNNcited by 7,259
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
38 more
  1. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  2. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  3. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  4. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  5. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  6. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  7. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  8. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  9. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  10. 2014How transferable are features in deep neural networks?cited by 1,216
  11. 2016Colorful Image Colorizationcited by 1,113
  12. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  13. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  14. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  15. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  16. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  17. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  18. 2018Do Better ImageNet Models Transfer Better?cited by 560
  19. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  20. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 422
  21. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  22. 2018A Survey on Deep Transfer Learningcited by 417
  23. 2019Transfusion: Understanding Transfer Learning for Medical Imagingcited by 366
  24. 2020Keystroke Dynamics as Part of Lifeloggingcited by 334
  25. 2019Semi-supervised Domain Adaptation via Minimax Entropycited by 333
  26. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
  27. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  28. 2020What Should Not Be Contrastive in Contrastive Learningcited by 224
  29. 2017Multi-task Self-Supervised Visual Learningcited by 200
  30. 2020Geography-Aware Self-Supervised Learningcited by 190
  31. 2017Playing for Benchmarkscited by 169
  32. 2018xView: Objects in Context in Overhead Imagerycited by 145
  33. 2020How Useful is Self-Supervised Pretraining for Visual Tasks?cited by 80
  34. 2020Scalable Transfer Learning with Expert Modelscited by 48
  35. 2019In-domain representation learning for remote sensingcited by 41
  36. 2020Remote Sensing Image Scene Classification with Self-Supervised Paradigm under Limited Labeled Samplescited by 36
  37. 2020SelfAugment: Automatic Augmentation Policies for Self-Supervised Learningcited by 32
  38. 2018Domain Adaptive Transfer Learning with Specialist Modelscited by 31

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 17; 2023: 23; 2024: 11; 2025: 9; 2026: 7.

81 papers.

Sort:

  1. 2022Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistencycited by 246
  2. 2023CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoderscited by 167
  3. 2023Towards Geospatial Foundation Models via Continual Pretrainingcited by 109
  4. 2021Recent advances and clinical applications of deep learning in medical image analysiscited by 104
  5. 2021Extending the WILDS Benchmark for Unsupervised Adaptationcited by 85
  6. 2021DETReg: Unsupervised Pretraining with Region Priors for Object Detectioncited by 76
  7. 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
  8. 2023Neuro-GPT: Towards A Foundation Model for EEGcited by 69
  9. 2023A Billion-scale Foundation Model for Remote Sensing Imagescited by 62
  10. 2024PANGAEA: A Global and Inclusive Benchmark for Geospatial Foundation Modelscited by 62
  11. 2021PASS: An ImageNet replacement for self-supervised pretraining without humanscited by 51
  12. 2021Contrasting Contrastive Self-Supervised Representation Learning Pipelinescited by 37
38 more
  1. 2021A Systematic Benchmarking Analysis of Transfer Learning for Medical Image Analysiscited by 31
  2. 2021How Well Does Self-Supervised Pre-Training Perform with Streaming Data?cited by 29
  3. 2024Multi-Label Guided Soft Contrastive Learning for Efficient Earth Observation Pretrainingcited by 27
  4. 2024CrossEarth: Geospatial Vision Foundation Model for Domain Generalizable Remote Sensing Semantic Segmentationcited by 27
  5. 2021Self-supervised Audiovisual Representation Learning for Remote Sensing Datacited by 24
  6. 2022Self-Supervised Pretraining for 2D Medical Image Segmentationcited by 24
  7. 2022Continual Barlow Twins: continual self-supervised learning for remote sensing semantic segmentationcited by 22
  8. 2022Consecutive Pretraining: A Knowledge Transfer Learning Strategy with Relevant Unlabeled Data for Remote Sensing Domaincited by 22
  9. 2023Towards long-tailed, multi-label disease classification from chest X-ray: Overview of the CXR-LT challengecited by 22
  10. 2023PECoP: Parameter Efficient Continual Pretraining for Action Quality Assessmentcited by 19
  11. 2023Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label-Efficient Representationscited by 18
  12. 2023LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speechcited by 17
  13. 2024ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shiftscited by 16
  14. 2023AlignDet: Aligning Pre-training and Fine-tuning in Object Detectioncited by 14
  15. 2021Improving Self-supervised Learning with Hardness-aware Dynamic Curriculum Learning: An Application to Digital Pathologycited by 13
  16. 2022Deep transfer learning for image classification: a surveycited by 13
  17. 2023SIRL: Similarity-based Implicit Representation Learningcited by 13
  18. 2023Aligning Robot and Human Representationscited by 11
  19. 2023A Review of Predictive and Contrastive Self-supervised Learning for Medical Imagescited by 10
  20. 2023RoPAWS: Robust Semi-supervised Representation Learning from Uncurated Datacited by 10
  21. 2025Parameter-Efficient Adaptation of Geospatial Foundation Models through Embedding Deflectioncited by 10
  22. 2022SOS! Self-supervised Learning Over Sets Of Handled Objects In Egocentric Action Recognitioncited by 8
  23. 2021Multi-source Few-shot Domain Adaptationcited by 7
  24. 2022Prior Knowledge-Guided Attention in Self-Supervised Vision Transformerscited by 7
  25. 2022Under the Cover Infant Pose Estimation using Multimodal Datacited by 6
  26. 2024UDA-Bench: Revisiting Common Assumptions in Unsupervised Domain Adaptation Using a Standardized Frameworkcited by 6
  27. 2021Self-supervised Contrastive Learning for Irrigation Detection in Satellite Imagerycited by 5
  28. 2022Refine and Represent: Region-to-Object Representation Learningcited by 5
  29. 2025A survey of datasets for computer vision in agriculturecited by 5
  30. 2021Do we still need ImageNet pre-training in remote sensing scene classification?cited by 4
  31. 2022Can we Adopt Self-supervised Pretraining for Chest X-Rays?cited by 4
  32. 2023Self-Supervised In-Domain Representation Learning for Remote Sensing Image Scene Classificationcited by 4
  33. 2025Bi-Level Optimization for Self-Supervised AI-Generated Face Detectioncited by 4
  34. 2021Auxiliary Learning for Self-Supervised Video Representation via Similarity-based Knowledge Distillationcited by 3
  35. 2023Self-supervised learning-based general laboratory progress pretrained model for cardiovascular event detectioncited by 3
  36. 2023Proposal-Contrastive Pretraining for Object Detection from Fewer Datacited by 3
  37. 2024Cross-sensor self-supervised training and alignment for remote sensingcited by 3
  38. 2025Contrastive Learning from Exploratory Actions: Leveraging Natural Interactions for Preference Elicitationcited by 3

50 most cited of 81 All 81 (JSON)