# Towards Democratizing Joint-Embedding Self-Supervised Learning

arXiv: https://arxiv.org/abs/2303.01986 · HTML: https://mlsanity.com/paper/2303.01986/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 20 references that are in the map.
- First posted on arXiv 3 Mar 2023.

## Builds on (most cited first, 15 of 15)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2104.14548/) (2021)
- [Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods](https://mlsanity.com/paper/2205.11508/) (2022)
- [High Fidelity Visualization of What Your Self-Supervised Representation Knows About](https://mlsanity.com/paper/2112.09164/) (2021)

## Built on by (most cited first, 18 of 18)

- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathology](https://mlsanity.com/paper/2408.00738/) (2024)
- [Large-scale Training of Foundation Models for Wearable Biosignals](https://mlsanity.com/paper/2312.05409/) (2023)
- [Self-Supervised Multimodal Learning: A Survey](https://mlsanity.com/paper/2304.01008/) (2023)
- [Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learning](https://mlsanity.com/paper/2505.12477/) (2025)
- [Do SSL Models Have D'ej`a Vu? A Case of Unintended Memorization in Self-supervised Learning](https://mlsanity.com/paper/2304.13850/) (2023)
- [Multiview Scene Graph](https://mlsanity.com/paper/2410.11187/) (2024)
- [Disentangling Linear Mode-Connectivity](https://mlsanity.com/paper/2312.09832/) (2023)
- [Caregiver Talk Shapes Toddler Vision: A Computational Study of Dyadic Play](https://mlsanity.com/paper/2312.04118/) (2023)
- [Training Dynamics of Deep Network Linear Regions](https://mlsanity.com/paper/2310.12977/) (2023)
- [Random Field Augmentations for Self-Supervised Representation Learning](https://mlsanity.com/paper/2311.03629/) (2023)
- [On The Fairness Impacts of Hardware Selection in Machine Learning](https://mlsanity.com/paper/2312.03886/) (2023)
- [Intra-video Positive Pairs in Self-Supervised Learning for Ultrasound](https://mlsanity.com/paper/2403.07715/) (2024)
- [DailyMAE: Towards Pretraining Masked Autoencoders in One Day](https://mlsanity.com/paper/2404.00509/) (2024)
- [MixDiff: Mixing Natural and Synthetic Images for Robust Self-Supervised Representations](https://mlsanity.com/paper/2406.12368/) (2024)
- [Self-Supervised Learning from Structural Invariance](https://mlsanity.com/paper/2602.02381/) (2026)
- [Learning Beyond Similarities: Incorporating Dissimilarities between Positive Pairs in Self-Supervised Time Series Learning](https://mlsanity.com/paper/2309.07526/) (2023)
- [HSG: Hyperbolic Scene Graph](https://mlsanity.com/paper/2604.17454/) (2026)

## Abstract (arXiv)

Joint Embedding Self-Supervised Learning (JE-SSL) has seen rapid developments in recent years, due to its promise to effectively leverage large unlabeled data. The development of JE-SSL methods was driven primarily by the search for ever increasing downstream classification accuracies, using huge computational resources, and typically built upon insights and intuitions inherited from a close parent JE-SSL method. This has led unwittingly to numerous pre-conceived ideas that carried over across methods e.g. that SimCLR requires very large mini batches to yield competitive accuracies; that strong and computationally slow data augmentations are required. In this work, we debunk several such ill-formed a priori ideas in the hope to unleash the full potential of JE-SSL free of unnecessary limitations. In fact, when carefully evaluating performances across different downstream tasks and properly optimizing hyper-parameters of the methods, we most often -- if not always -- see that these widespread misconceptions do not hold. For example we show that it is possible to train SimCLR to learn useful representations, while using a single image patch as negative example, and simple Gaussian noise as the only data augmentation for the positive pair. Along these lines, in the hope to democratize JE-SSL and to allow researchers to easily make more extensive evaluations of their methods, we introduce an optimized PyTorch library for SSL.