mlsanityData to 7 Oct 2026

Simple Distillation Baselines for Improving Small Self-supervised Models

arXiv 2106.11304 ↗First posted 21 Jun 2021cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

While large self-supervised models have rivalled the performance of their supervised counterparts, small models still struggle. In this report, we explore simple baselines for improving small self-supervised models via distillation, called SimDis. Specifically, we present an offline-distillation baseline, which establishes a new state-of-the-art, and an online-distillation baseline, which achieves similar performance with minimal computational overhead. We hope these baselines will provide useful experience for relevant future research. Code is available at: https://github.com/JindongGu/SimDis/

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 14 of its 15 references that are in the map. First posted on arXiv 21 Jun 2021.

Builds on

14 of 15 references.

References · 14

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  7. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  8. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  9. 2019Contrastive Multiview Codingcited by 1,297
  10. 2017Deep Mutual Learningcited by 621
  11. 2013Do Deep Nets Really Need to be Deep?cited by 466
  12. 2021SEED: Self-supervised Distillation For Visual Representationcited by 132
2 more
  1. 2020CompRess: Self-Supervised Learning by Compressing Representationscited by 61
  2. 2020Search for Better Students to Learn Distilled Knowledgecited by 11

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 2; 2023: 1; 2024: 1.

5 papers.

  1. 2021On the Efficacy of Small Self-Supervised Contrastive Models without Distillation Signalscited by 14
  2. 2022Slimmable Networks for Contrastive Self-supervised Learningcited by 4
  3. 2022Improving Label-Deficient Keyword Spotting Through Self-Supervised Pretrainingcited by 3
  4. 2024Relational Self-supervised Distillation with Compact Descriptors for Image Copy Detectioncited by 1
  5. 2023Explainability and Robustness of Deep Visual Classification Modelscited by 0