# On the Efficacy of Small Self-Supervised Contrastive Models without Distillation Signals

arXiv: https://arxiv.org/abs/2107.14762 · HTML: https://mlsanity.com/paper/2107.14762/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 40 references that are in the map.
- First posted on arXiv 30 Jul 2021.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [Understanding the Behaviour of Contrastive Loss](https://mlsanity.com/paper/2012.09740/) (2020)
- [Unsupervised Embedding Learning via Invariant and Spreading Instance Feature](https://mlsanity.com/paper/1904.03436/) (2019)
- [SEED: Self-supervised Distillation For Visual Representation](https://mlsanity.com/paper/2101.04731/) (2021)
- [Boosting Contrastive Self-Supervised Learning with False Negative Cancellation](https://mlsanity.com/paper/2011.11765/) (2020)
- [CompRess: Self-Supervised Learning by Compressing Representations](https://mlsanity.com/paper/2010.14713/) (2020)
- [Conditional Negative Sampling for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2010.02037/) (2020)
- [Solving Inefficiency of Self-supervised Representation Learning](https://mlsanity.com/paper/2104.08760/) (2021)

## Built on by (most cited first, 14 of 14)

- [A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithm](https://mlsanity.com/paper/2310.12244/) (2023)
- [A Simple Recipe for Competitive Low-compute Self supervised Vision Models](https://mlsanity.com/paper/2301.09451/) (2023)
- [CORN: Contact-based Object Representation for Nonprehensile Manipulation of General Unseen Objects](https://mlsanity.com/paper/2403.10760/) (2024)
- [Slimmable Networks for Contrastive Self-supervised Learning](https://mlsanity.com/paper/2209.15525/) (2022)
- [Improving Label-Deficient Keyword Spotting Through Self-Supervised Pretraining](https://mlsanity.com/paper/2210.01703/) (2022)
- [Effective Self-supervised Pre-training on Low-compute Networks without Distillation](https://mlsanity.com/paper/2210.02808/) (2022)
- [Establishing a stronger baseline for lightweight contrastive models](https://mlsanity.com/paper/2212.07158/) (2022)
- [Dual Distribution Estimation for Zero-shot Noisy Test-Time Adaptation with VLMs](https://mlsanity.com/paper/2606.25758/) (2026)
- [On Improving the Algorithm-, Model-, and Data- Efficiency of Self-Supervised Learning](https://mlsanity.com/paper/2404.19289/) (2024)
- [Relational Self-supervised Distillation with Compact Descriptors for Image Copy Detection](https://mlsanity.com/paper/2405.17928/) (2024)
- [Lightweight Model Pre-training via Language Guided Knowledge Distillation](https://mlsanity.com/paper/2406.11689/) (2024)
- [Dist2ill: Distributional Distillation for One-Pass Uncertainty Estimation in Large Language Models](https://mlsanity.com/paper/2505.11731/) (2025)
- [Retrieving Patient-Specific Radiomic Feature Sets for Transparent Knee MRI Assessment](https://mlsanity.com/paper/2603.02367/) (2026)
- [Contrastive Self-Supervised Learning at the Edge: An Energy Perspective](https://mlsanity.com/paper/2510.08374/) (2025)

## Abstract (arXiv)

It is a consensus that small models perform quite poorly under the paradigm of self-supervised contrastive learning. Existing methods usually adopt a large off-the-shelf model to transfer knowledge to the small one via distillation. Despite their effectiveness, distillation-based methods may not be suitable for some resource-restricted scenarios due to the huge computational expenses of deploying a large model. In this paper, we study the issue of training self-supervised small models without distillation signals. We first evaluate the representation spaces of the small models and make two non-negligible observations: (i) the small models can complete the pretext task without overfitting despite their limited capacity and (ii) they universally suffer the problem of over clustering. Then we verify multiple assumptions that are considered to alleviate the over-clustering phenomenon. Finally, we combine the validated techniques and improve the baseline performances of five small architectures with considerable margins, which indicates that training small self-supervised contrastive models is feasible even without distillation signals. The code is available at textithttps://github.com/WOWNICE/ssl-small.