# DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learning

arXiv: https://arxiv.org/abs/2104.09124 · HTML: https://mlsanity.com/paper/2104.09124/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 51 references that are in the map.
- First posted on arXiv 19 Apr 2021.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Do Better ImageNet Models Transfer Better?](https://mlsanity.com/paper/1805.08974/) (2018)

## Built on by (most cited first, 26 of 26)

- [PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining](https://mlsanity.com/paper/2204.14095/) (2022)
- [Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Training](https://mlsanity.com/paper/2305.13689/) (2023)
- [A Closer Look at Self-Supervised Lightweight Vision Transformers](https://mlsanity.com/paper/2205.14443/) (2022)
- [Bag of Instances Aggregation Boosts Self-supervised Distillation](https://mlsanity.com/paper/2107.01691/) (2021)
- [Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/2304.06461/) (2023)
- [On the Efficacy of Small Self-Supervised Contrastive Models without Distillation Signals](https://mlsanity.com/paper/2107.14762/) (2021)
- [Learning Common Rationale to Improve Self-Supervised Representation for Fine-Grained Visual Recognition Problems](https://mlsanity.com/paper/2303.01669/) (2023)
- [A Simple Recipe for Competitive Low-compute Self supervised Vision Models](https://mlsanity.com/paper/2301.09451/) (2023)
- [Self-Contrastive Learning: Single-viewed Supervised Contrastive Framework using Sub-network](https://mlsanity.com/paper/2106.15499/) (2021)
- [Pixel-Wise Contrastive Distillation](https://mlsanity.com/paper/2211.00218/) (2022)
- [Simple Unsupervised Knowledge Distillation With Space Similarity](https://mlsanity.com/paper/2409.13939/) (2024)
- [Slimmable Networks for Contrastive Self-supervised Learning](https://mlsanity.com/paper/2209.15525/) (2022)
- [SDCL: Self-Distillation Contrastive Learning for Chinese Spell Checking](https://mlsanity.com/paper/2210.17168/) (2022)
- [Learning Task-Agnostic Representations through Multi-Teacher Distillation](https://mlsanity.com/paper/2510.18680/) (2025)
- [Weak Augmentation Guided Relational Self-Supervised Learning](https://mlsanity.com/paper/2203.08717/) (2022)
- [Effective Self-supervised Pre-training on Low-compute Networks without Distillation](https://mlsanity.com/paper/2210.02808/) (2022)
- [Establishing a stronger baseline for lightweight contrastive models](https://mlsanity.com/paper/2212.07158/) (2022)
- [On Good Practices for Task-Specific Distillation of Large Pretrained Visual Models](https://mlsanity.com/paper/2402.11305/) (2024)
- [Relational Self-supervised Distillation with Compact Descriptors for Image Copy Detection](https://mlsanity.com/paper/2405.17928/) (2024)
- [Lightweight Model Pre-training via Language Guided Knowledge Distillation](https://mlsanity.com/paper/2406.11689/) (2024)
- [DSPNet: Towards Slimmable Pretrained Networks based on Discriminative Self-supervised Learning](https://mlsanity.com/paper/2207.06075/) (2022)
- [LowDINO -- A Low Parameter Self Supervised Learning Model](https://mlsanity.com/paper/2305.17791/) (2023)
- [On the Surprising Efficacy of Distillation as an Alternative to Pre-Training Small Models](https://mlsanity.com/paper/2404.03263/) (2024)
- [ProFeAT: Projected Feature Adversarial Training for Self-Supervised Learning of Robust Representations](https://mlsanity.com/paper/2406.05796/) (2024)
- [Efficient Building Roof Type Classification: A Domain-Specific Self-Supervised Approach](https://mlsanity.com/paper/2503.22251/) (2025)
- [TinySSL: Distilled Self-Supervised Pretraining for Sub-Megabyte MCU Models](https://mlsanity.com/paper/2605.08241/) (2026)

## Abstract (arXiv)

While self-supervised representation learning (SSL) has received widespread attention from the community, recent research argue that its performance will suffer a cliff fall when the model size decreases. The current method mainly relies on contrastive learning to train the network and in this work, we propose a simple yet effective Distilled Contrastive Learning (DisCo) to ease the issue by a large margin. Specifically, we find the final embedding obtained by the mainstream SSL methods contains the most fruitful information, and propose to distill the final embedding to maximally transmit a teacher's knowledge to a lightweight model by constraining the last embedding of the student to be consistent with that of the teacher. In addition, in the experiment, we find that there exists a phenomenon termed Distilling BottleNeck and present to enlarge the embedding dimension to alleviate this problem. Our method does not introduce any extra parameter to lightweight models during deployment. Experimental results demonstrate that our method achieves the state-of-the-art on all lightweight models. Particularly, when ResNet-101/ResNet-50 is used as teacher to teach EfficientNet-B0, the linear result of EfficientNet-B0 on ImageNet is very close to ResNet-101/ResNet-50, but the number of parameters of EfficientNet-B0 is only 9.4%/16.3% of ResNet-101/ResNet-50. Code is available at https://github. com/Yuting-Gao/DisCo-pytorch.