mlsanityData to 14 Oct 2026

No Free Lunch in Self Supervised Representation Learning

arXiv 2304.11718 ↗First posted 23 Apr 2023cs.CV cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Self-supervised representation learning in computer vision relies heavily on hand-crafted image transformations to learn meaningful and invariant features. However few extensive explorations of the impact of transformation design have been conducted in the literature. In particular, the dependence of downstream performances to transformation design has been established, but not studied in depth. In this work, we explore this relationship, its impact on a domain other than natural images, and show that designing the transformations can be viewed as a form of supervision. First, we demonstrate that not only do transformations have an effect on downstream performance and relevance of clustering, but also that each category in a supervised dataset can be impacted in a different way. Following this, we explore the impact of transformation design on microscopy images, a domain where the difference between classes is more subtle and fuzzy than in natural images. In this case, we observe a greater impact on downstream tasks performances. Finally, we demonstrate that transformation design can be leveraged as a form of supervision, as careful selection of these by a domain expert can lead to a drastic increase in performance on a given downstream task.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 44 of its 62 references that are in the map. First posted on arXiv 23 Apr 2023.

Builds on

44 of 62 references.

References · 44

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  7. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  8. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  9. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  10. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  11. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  12. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
32 more
  1. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  2. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  3. 2018AutoAugment: Learning Augmentation Policies from Datacited by 730
  4. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  5. 2020Hard Negative Mixing for Contrastive Learningcited by 402
  6. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 326
  7. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 312
  8. 2020Whitening for Self-Supervised Representation Learningcited by 279
  9. 2020What Should Not Be Contrastive in Contrastive Learningcited by 226
  10. 2021Decoupled Contrastive Learningcited by 143
  11. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  12. 2022VICRegL: Self-Supervised Learning of Local Visual Featurescited by 118
  13. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  14. 2021Contrastive Learning with Stronger Augmentationscited by 100
  15. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
  16. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  17. 2021ReSSL: Relational Self-Supervised Learning with Weak Augmentationcited by 84
  18. 2022The Effects of Regularization and Data Augmentation are Class Dependentcited by 67
  19. 2020DADA: Differentiable Automatic Data Augmentationcited by 57
  20. 2022Rethinking the Augmentation Module in Contrastive Learning: Learning Hierarchical Augmentation Invariance with Expanded Viewscited by 37
  21. 2020On Compositions of Transformations in Contrastive Self-Supervised Learningcited by 34
  22. 2020SelfAugment: Automatic Augmentation Policies for Self-Supervised Learningcited by 32
  23. 2021Direct Differentiable Augmentation Searchcited by 28
  24. 2023HaLP: Hallucinating Latent Positives for Skeleton-based Self-Supervised Learning of Actionscited by 25
  25. 2022On the Importance of Hyperparameters and Data Augmentation for Self-Supervised Learningcited by 16
  26. 2021Contrastive Learning of Single-Cell Phenotypic Representations for Treatment Classificationcited by 12
  27. 2023Unpaired Image-to-Image Translation with Limited Data to Reveal Subtle Phenotypescited by 7
  28. 2021Computer Vision Self-supervised Learning Methods on Time Seriescited by 4
  29. 2021Distribution Estimation to Automate Transformation Policies for Self-Supervisioncited by 3
  30. 2022Comparison of semi-supervised learning methods for High Content Screening quality controlcited by 3
  31. 2019Towards a Hypothesis on Visual Transformation based Self-Supervisioncited by 2
  32. 2022Optimizing transformations for contrastive learning in a differentiable frameworkcited by 1

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 4; 2025: 1.

7 papers.

  1. 2024InfoNCE: Identifying the Gap Between Theory and Practicecited by 40
  2. 2024You Don't Need Domain-Specific Data Augmentations When Scaling Self-Supervised Learningcited by 16
  3. 2025Clustering Properties of Self-Supervised Learningcited by 8
  4. 2023Weakly supervised cross-modal learning in high-content screeningcited by 4
  5. 2024DDA: Dimensionality Driven Augmentation Search for Contrastive Learning in Laparoscopic Surgerycited by 3
  6. 2023No Data Augmentation? Alternative Regularizations for Effective Training on Small Datasetscited by 1
  7. 2024Unlocking Telemetry Potential: Self-Supervised Learning for Continuous Clinical Electrocardiogram Monitoringcited by 1