# On the Importance of Hyperparameters and Data Augmentation for Self-Supervised Learning

arXiv: https://arxiv.org/abs/2207.07875 · HTML: https://mlsanity.com/paper/2207.07875/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 32 references that are in the map.
- First posted on arXiv 16 Jul 2022.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [Albumentations: fast and flexible image augmentations](https://mlsanity.com/paper/1809.06839/) (2018)
- [TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentation](https://mlsanity.com/paper/2103.10158/) (2021)
- [MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysis](https://mlsanity.com/paper/2010.14925/) (2020)
- [πBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization](https://mlsanity.com/paper/2204.11051/) (2022)
- [SelfAugment: Automatic Augmentation Policies for Self-Supervised Learning](https://mlsanity.com/paper/2009.07724/) (2020)
- [Meta-Learning to Improve Pre-Training](https://mlsanity.com/paper/2111.01754/) (2021)
- [Smart(Sampling)Augment: Optimal and Efficient Data Augmentation for Semantic Segmentation](https://mlsanity.com/paper/2111.00487/) (2021)

## Built on by (most cited first, 16 of 16)

- [InfoNCE: Identifying the Gap Between Theory and Practice](https://mlsanity.com/paper/2407.00143/) (2024)
- [A generalizable 3D framework and model for self-supervised learning in medical imaging](https://mlsanity.com/paper/2501.11755/) (2025)
- [Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model](https://mlsanity.com/paper/2310.17653/) (2023)
- [Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearch](https://mlsanity.com/paper/2603.24647/) (2026)
- [LDReg: Local Dimensionality Regularized Self-Supervised Learning](https://mlsanity.com/paper/2401.10474/) (2024)
- [Amortised Invariance Learning for Contrastive Self-Supervision](https://mlsanity.com/paper/2302.12712/) (2023)
- [Systematic comparison of semi-supervised and self-supervised learning for medical image classification](https://mlsanity.com/paper/2307.08919/) (2023)
- [No Free Lunch in Self Supervised Representation Learning](https://mlsanity.com/paper/2304.11718/) (2023)
- [DDA: Dimensionality Driven Augmentation Search for Contrastive Learning in Laparoscopic Surgery](https://mlsanity.com/paper/2406.00907/) (2024)
- [Augmentation-aware Self-supervised Learning with Conditioned Projector](https://mlsanity.com/paper/2306.06082/) (2023)
- [SASSL: Enhancing Self-Supervised Learning via Neural Style Transfer](https://mlsanity.com/paper/2312.01187/) (2023)
- [Fast Benchmarking of Asynchronous Multi-Fidelity Optimization on Zero-Cost Benchmarks](https://mlsanity.com/paper/2403.01888/) (2024)
- [SRA: A Novel Method to Improve Feature Embedding in Self-supervised Learning for Histopathological Images](https://mlsanity.com/paper/2410.17514/) (2024)
- [Collaborative Unlabeled Data Optimization](https://mlsanity.com/paper/2505.14117/) (2025)
- [Tree-Structured Parzen Estimator Can Solve Black-Box Combinatorial Optimization More Efficiently](https://mlsanity.com/paper/2507.08053/) (2025)
- [Simplifying Knowledge Transfer in Pretrained Models](https://mlsanity.com/paper/2510.22208/) (2025)

## Abstract (arXiv)

Self-Supervised Learning (SSL) has become a very active area of Deep Learning research where it is heavily used as a pre-training method for classification and other tasks. However, the rapid pace of advancements in this area comes at a price: training pipelines vary significantly across papers, which presents a potentially crucial confounding factor. Here, we show that, indeed, the choice of hyperparameters and data augmentation strategies can have a dramatic impact on performance. To shed light on these neglected factors and help maximize the power of SSL, we hyperparameterize these components and optimize them with Bayesian optimization, showing improvements across multiple datasets for the SimSiam SSL approach. Realizing the importance of data augmentations for SSL, we also introduce a new automated data augmentation algorithm, GroupAugment, which considers groups of augmentations and optimizes the sampling across groups. In contrast to algorithms designed for supervised learning, GroupAugment achieved consistently high linear evaluation accuracy across all datasets we considered. Overall, our results indicate the importance and likely underestimated role of data augmentation for SSL.