# Towards Good Practices in Self-supervised Representation Learning

arXiv: https://arxiv.org/abs/2012.00868 · HTML: https://mlsanity.com/paper/2012.00868/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Dec 2020.

## Built on by (most cited first, 19 of 19)

- [DocFormer: End-to-End Transformer for Document Understanding](https://mlsanity.com/paper/2106.11539/) (2021)
- [High Fidelity Visualization of What Your Self-Supervised Representation Knows About](https://mlsanity.com/paper/2112.09164/) (2021)
- [DocFormerv2: Local Features for Document Understanding](https://mlsanity.com/paper/2306.01733/) (2023)
- [ReMix: A General and Efficient Framework for Multiple Instance Learning based Whole Slide Image Classification](https://mlsanity.com/paper/2207.01805/) (2022)
- [Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learning](https://mlsanity.com/paper/2206.13378/) (2022)
- [Beyond Supervised vs. Unsupervised: Representative Benchmarking and Analysis of Image Representation Learning](https://mlsanity.com/paper/2206.08347/) (2022)
- [EMP-SSL: Towards Self-Supervised Learning in One Training Epoch](https://mlsanity.com/paper/2304.03977/) (2023)
- [Toward a Geometrical Understanding of Self-supervised Contrastive Learning](https://mlsanity.com/paper/2205.06926/) (2022)
- [All4One: Symbiotic Neighbour Contrastive Learning via Self-Attention and Redundancy Reduction](https://mlsanity.com/paper/2303.09417/) (2023)
- [An Evaluation of Self-Supervised Pre-Training for Skin-Lesion Analysis](https://mlsanity.com/paper/2106.09229/) (2021)
- [Variance Covariance Regularization Enforces Pairwise Independence in Self-Supervised Representations](https://mlsanity.com/paper/2209.14905/) (2022)
- [Learning Data Association for Multi-Object Tracking using Only Coordinates](https://mlsanity.com/paper/2403.08018/) (2024)
- [Evaluating Self-Supervised Learning via Risk Decomposition](https://mlsanity.com/paper/2302.03068/) (2023)
- [Deciphering the Projection Head: Representation Evaluation Self-supervised Learning](https://mlsanity.com/paper/2301.12189/) (2023)
- [Data Compression and Inference in Cosmology with Self-Supervised Machine Learning](https://mlsanity.com/paper/2308.09751/) (2023)
- [Temporal Contrastive Learning with Curriculum](https://mlsanity.com/paper/2209.00760/) (2022)
- [Towards Unsupervised Representation Learning: Learning, Evaluating and Transferring Visual Representations](https://mlsanity.com/paper/2312.00101/) (2023)
- [Show Me What You Don't Know: Efficient Sampling from Invariant Sets for Model Validation](https://mlsanity.com/paper/2603.21782/) (2026)
- [Source-Free Controlled Adaptation of Teachers for Continual Test-Time Adaptation](https://mlsanity.com/paper/2607.23735/) (2026)

## Abstract (arXiv)

Self-supervised representation learning has seen remarkable progress in the last few years. More recently, contrastive instance learning has shown impressive results compared to its supervised learning counterparts. However, even with the ever increased interest in contrastive instance learning, it is still largely unclear why these methods work so well. In this paper, we aim to unravel some of the mysteries behind their success, which are the good practices. Through an extensive empirical analysis, we hope to not only provide insights but also lay out a set of best practices that led to the success of recent work in self-supervised representation learning.