mlsanityData to 7 Oct 2026

Are Large-scale Datasets Necessary for Self-Supervised Pre-training?

arXiv 2112.10740 ↗First posted 20 Dec 2021cs.CVcited by 117

Not on the map. Why?

Abstract (arXiv)

Pre-training models on large scale datasets, like ImageNet, is a standard practice in computer vision. This paradigm is especially effective for tasks with small training sets, for which high-capacity models tend to overfit. In this work, we consider a self-supervised pre-training scenario that only leverages the target task data. We consider datasets, like Stanford Cars, Sketch or COCO, which are order(s) of magnitude smaller than Imagenet. Our study shows that denoising autoencoders, such as BEiT or a variant that we introduce in this paper, are more robust to the type and size of the pre-training data than popular self-supervised methods trained by comparing image embeddings.We obtain competitive performance compared to ImageNet pre-training on a variety of classification datasets, from different domains. On COCO, when pre-training solely using COCO images, the detection and instance segmentation performance surpasses the supervised ImageNet pre-training in a comparable setting.

Key facts. Cited by 117 papers in the map (data through 7 Oct 2026). Builds on 52 of its 69 references that are in the map. First posted on arXiv 20 Dec 2021.

Builds on

52 of 69 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  11. 2020End-to-End Object Detection with Transformerscited by 7,384
  12. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
38 more
  1. 2017Mask R-CNNcited by 7,259
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  10. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  12. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  13. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  14. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  15. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  16. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  17. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  18. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  19. 2014How transferable are features in deep neural networks?cited by 1,216
  20. 2016Colorful Image Colorizationcited by 1,113
  21. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  22. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  23. 2020Designing Network Design Spacescited by 954
  24. 2015Unsupervised Deep Embedding for Clustering Analysiscited by 763
  25. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  26. 2014Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFscited by 552
  27. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  28. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  29. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 299
  30. 2016Joint Unsupervised Learning of Deep Representations and Image Clusterscited by 205
  31. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 197
  32. 2020CrossTransformers: spatially-aware few-shot transfercited by 174
  33. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  34. 2020Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biasescited by 143
  35. 2021Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samplescited by 114
  36. 2021Efficient Training of Visual Transformers with Small Datasetscited by 107
  37. 2021Pre-training without Natural Imagescited by 100
  38. 2021SiT: Self-supervised vIsion Transformercited by 99

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 50; 2023: 40; 2024: 15; 2025: 6; 2026: 5.

117 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  3. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  4. 2022DeiT III: Revenge of the ViTcited by 417
  5. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  6. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  7. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  8. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  9. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 287
  10. 2023A Cookbook of Self-Supervised Learningcited by 248
  11. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  12. 2024Sapiens: Foundation for Human Vision Modelscited by 229
38 more
  1. 2022CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flowcited by 165
  2. 2022CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completioncited by 158
  3. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
  4. 2022VICRegL: Self-Supervised Learning of Local Visual Featurescited by 118
  5. 20234M: Massively Multimodal Masked Modelingcited by 118
  6. 2024Scalable Pre-training of Large Autoregressive Image Modelscited by 115
  7. 2022OmniMAE: Single Model Masked Pretraining on Images and Videoscited by 99
  8. 2022Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillationcited by 93
  9. 2022Three things everyone should know about Vision Transformerscited by 90
  10. 2022Siamese Image Modeling for Self-Supervised Vision Representation Learningcited by 78
  11. 2022Masked Frequency Modeling for Self-Supervised Visual Pre-Trainingcited by 70
  12. 2022MILAN: Masked Image Pretraining on Language Assisted Representationcited by 69
  13. 2022Corrupted Image Modeling for Self-Supervised Visual Pre-Trainingcited by 68
  14. 2022Bootstrapped Masked Autoencoders for Vision BERT Pretrainingcited by 60
  15. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  16. 2022The Hidden Uniform Cluster Prior in Self-Supervised Learningcited by 57
  17. 2022MARLIN: Masked Autoencoder for facial video Representation LearnINgcited by 55
  18. 2023Masked Scene Contrast: A Scalable Framework for Unsupervised 3D Representation Learningcited by 55
  19. 2022SdAE: Self-distillated Masked Autoencodercited by 52
  20. 20244M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalitiescited by 52
  21. 2022Understanding Masked Image Modeling via Learning Occlusion Invariant Featurecited by 50
  22. 2022Audiovisual Masked Autoencoderscited by 47
  23. 2022On Data Scaling in Masked Image Modelingcited by 45
  24. 2022Revisiting Pretraining Objectives for Tabular Deep Learningcited by 43
  25. 2023Traj-MAE: Masked Autoencoders for Trajectory Predictioncited by 42
  26. 2023Mixed Autoencoder for Self-supervised Visual Representation Learningcited by 38
  27. 2023DreamTeacher: Pretraining Image Backbones with Deep Generative Modelscited by 37
  28. 2022A Unified View of Masked Image Modelingcited by 36
  29. 2022ResFormer: Scaling ViTs with Multi-Resolution Trainingcited by 34
  30. 2023Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priorscited by 33
  31. 2022Image Compression with Product Quantized Masked Image Modelingcited by 32
  32. 2023Overwriting Pretrained Bias with Finetuning Datacited by 32
  33. 2022Architecture-Agnostic Masked Image Modeling -- From ViT back to CNNcited by 31
  34. 2022A Closer Look at Self-Supervised Lightweight Vision Transformerscited by 31
  35. 2022Stare at What You See: Masked Image Modeling without Reconstructioncited by 30
  36. 2023Semantic Image Segmentation: Two Decades of Researchcited by 26
  37. 2022CAE v2: Context Autoencoder with CLIP Targetcited by 25
  38. 2022Masked Modeling Duo: Learning Representations by Encouraging Both Networks to Model the Inputcited by 23

50 most cited of 117 All 117 (JSON)