mlsanityData to 7 Oct 2026

On Pretraining Data Diversity for Self-Supervised Learning

arXiv 2403.13808 ↗First posted 20 Mar 2024cs.CV cs.AI cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

We explore the impact of training with more diverse datasets, characterized by the number of unique samples, on the performance of self-supervised learning (SSL) under a fixed computational budget. Our findings consistently demonstrate that increasing pretraining data diversity enhances SSL performance, albeit only when the distribution distance to the downstream data is minimal. Notably, even with an exceptionally large pretraining data diversity achieved through methods like web crawling or diffusion-generated data, among other ways, the distribution shift remains a challenge. Our experiments are comprehensive with seven SSL methods using large-scale datasets such as ImageNet and YFCC100M amounting to over 200 GPU days. Code and trained models are available at https://github.com/hammoudhasan/DiversitySSL

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 60 of its 73 references that are in the map. First posted on arXiv 20 Mar 2024.

Builds on

60 of 73 references.

References · 60

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2020Exploring Simple Siamese Representation Learningcited by 2,853
38 more
  1. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  2. 2022Emergent Abilities of Large Language Modelscited by 2,483
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  4. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  5. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  6. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  7. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  8. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  9. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  10. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  11. 2019Contrastive Multiview Codingcited by 1,297
  12. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  13. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  14. 2016Colorful Image Colorizationcited by 1,113
  15. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  16. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  17. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  18. 2021Scaling Vision Transformerscited by 993
  19. 2020Designing Network Design Spacescited by 954
  20. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  21. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  22. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  23. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  24. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  25. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  26. 2023A Cookbook of Self-Supervised Learningcited by 248
  27. 2020How Well Do Self-Supervised Models Transfer?cited by 213
  28. 2019Local Aggregation for Unsupervised Learning of Visual Embeddingscited by 204
  29. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  30. 2017Multi-task Self-Supervised Visual Learningcited by 200
  31. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 197
  32. 2021Benchmarking Representation Learning for Natural World Image Collectionscited by 167
  33. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  34. 2021Solo-learn: A Library of Self-supervised Methods for Visual Representation Learningcited by 116
  35. 2020What makes instance discrimination good for transfer learning?cited by 111
  36. 2021When Does Contrastive Visual Representation Learning Work?cited by 92
  37. 2021Divide and Contrast: Self-supervised Learning from Uncurated Datacited by 85
  38. 2022Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervisioncited by 80

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 2.

7 papers.

  1. 2026THOR: A Versatile Foundation Model for Earth Observation Climate and Society Applicationscited by 6
  2. 2024S3PT: Scene Semantics and Structure Guided Clustering to Boost Self-Supervised Pre-Training for Autonomous Drivingcited by 4
  3. 2025Towards Reliable Identification of Diffusion-based Image Manipulationscited by 4
  4. 2024Precision at Scale: Domain-Specific Datasets On-Demandcited by 1
  5. 2025On the Importance of Pretraining Data Alignment for Atomic Property Predictioncited by 0
  6. 2025Understanding Ice Crystal Habit Diversity with Self-Supervised Learningcited by 0
  7. 2026Parameter-Efficient Self-Supervised Adaptation for EEG-FM under Fixed Computational Budgetscited by 0