mlsanityData to 7 Oct 2026

You Don't Need Domain-Specific Data Augmentations When Scaling Self-Supervised Learning

arXiv 2406.09294 ↗First posted 13 Jun 2024cs.LG cs.CVcited by 16

Not on the map. Why?

Abstract (arXiv)

Self-Supervised learning (SSL) with Joint-Embedding Architectures (JEA) has led to outstanding performances. All instantiations of this paradigm were trained using strong and well-established hand-crafted data augmentations, leading to the general belief that they are required for the proper training and performance of such models. On the other hand, generative reconstruction-based models such as BEIT and MAE or Joint-Embedding Predictive Architectures such as I-JEPA have shown strong performance without using data augmentations except masking. In this work, we challenge the importance of invariance and data-augmentation in JEAs at scale. By running a case-study on a recent SSL foundation model - DINOv2 - we show that strong image representations can be obtained with JEAs and only cropping without resizing provided the training data is large enough, reaching state-of-the-art results and using the least amount of augmentation in the literature. Through this study, we also discuss the impact of compute constraints on the outcomes of experimental deep learning research, showing that they can lead to very different conclusions.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 46 of its 54 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

46 of 54 references.

References · 46

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
34 more
  1. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  2. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  3. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  4. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  5. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  6. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  7. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  8. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  9. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  10. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  11. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  12. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  13. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  14. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  15. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  16. 2020Are we done with ImageNet?cited by 329
  17. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  18. 2020VinDr-CXR: An open dataset of chest X-rays with radiologist's annotationscited by 236
  19. 2020What Should Not Be Contrastive in Contrastive Learningcited by 224
  20. 2020Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biasescited by 143
  21. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  22. 2024Scalable Pre-training of Large Autoregressive Image Modelscited by 115
  23. 2017Unsupervised Learning by Predicting Noisecited by 97
  24. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  25. 2020BYOL works even without batch statisticscited by 95
  26. 2019A critical analysis of self-supervision, or what we can learn from a single imagecited by 75
  27. 2020Products-10K: A Large-scale Product Recognition Datasetcited by 42
  28. 2022How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularizationcited by 40
  29. 2022Rethinking the Augmentation Module in Contrastive Learning: Learning Hierarchical Augmentation Invariance with Expanded Viewscited by 37
  30. 2022How to Train Vision Transformer on Small-scale Datasets?cited by 34
  31. 2020RP2K: A Large-Scale Retail Product Dataset for Fine-Grained Image Classificationcited by 27
  32. 2023Self-Supervised Disentanglement by Leveraging Structure in Data Augmentationscited by 16
  33. 2023FireRisk: A Remote Sensing Dataset for Fire Risk Assessment with Benchmarks Using Supervised and Self-supervised Learningcited by 13
  34. 2023No Free Lunch in Self Supervised Representation Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 5.

16 papers.

Sort:

By area:

  1. 2024Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathologycited by 144
  2. 2025Curia: A Multi-Modal Foundation Model for Radiologycited by 14
  3. 2025KerJEPA: Kernel Discrepancies for Euclidean Self-Supervised Learningcited by 6
  4. 2025MoSiC: Optimal-Transport Motion Trajectory for Dense Self-Supervised Learningcited by 5
  5. 2026You Don't Need Strong Assumptions: Visual Representation Learning via Temporal Differencescited by 5
  6. 2025Seeing What Tastes Good: Revisiting Multimodal Distributional Semantics in the Billion Parameter Eracited by 3
  7. 2025Large Vision Models Can Solve Mental Rotation Problemscited by 3
  8. 2025A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomics Representations through Morphological Featurescited by 2
  9. 2025TAP-CT: 3D Task-Agnostic Pretraining of Computed Tomography Foundation Modelscited by 2
  10. 2025Beyond Instance Consistency: Investigating View Diversity in Self-supervised Learningcited by 1
  11. 2025Conjuring Positive Pairs for Efficient Unification of Representation Learning and Image Synthesiscited by 0
  12. 2025The Efficacy of Semantics-Preserving Transformations in Self-Supervised Learning for Medical Ultrasoundcited by 0
4 more
  1. 2026Self-Supervised Learning with a Multi-Task Latent Space Objectivecited by 0
  2. 2026Mixed Magnification Aggregation for Generalizable Region-Level Representations in Computational Pathologycited by 0
  3. 2026Self-Supervised Learning of Plant Image Representationscited by 0
  4. 2026Two Global Crops Suffice: Locating Semantic Emergence in DINO-Style Self-Supervised Learningcited by 0