mlsanityData to 7 Oct 2026

SEED: Self-supervised Distillation For Visual Representation

arXiv 2101.04731 ↗First posted 12 Jan 2021cs.CV cs.AIcited by 132

Not on the map. Why?

Abstract (arXiv)

This paper is concerned with self-supervised learning for small models. The problem is motivated by our empirical studies that while the widely used contrastive self-supervised learning method has shown great progress on large model training, it does not work well for small models. To address this problem, we propose a new learning paradigm, named SElf-SupErvised Distillation (SEED), where we leverage a larger network (as Teacher) to transfer its representational knowledge into a smaller architecture (as Student) in a self-supervised fashion. Instead of directly learning from unlabeled data, we train a student encoder to mimic the similarity score distribution inferred by a teacher over a set of instances. We show that SEED dramatically boosts the performance of small networks on downstream tasks. Compared with self-supervised baselines, SEED improves the top-1 accuracy from 42.2% to 67.6% on EfficientNet-B0 and from 36.3% to 68.2% on MobileNet-v3-Large on the ImageNet-1k dataset.

Key facts. Cited by 132 papers in the map (data through 7 Oct 2026). Builds on 47 of its 63 references that are in the map. First posted on arXiv 12 Jan 2021.

Builds on

47 of 63 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2017Mask R-CNNcited by 7,259
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  11. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
35 more
  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  2. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  4. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  5. 2019Searching for MobileNetV3cited by 1,984
  6. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  7. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  8. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  9. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  10. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  11. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  12. 2019Contrastive Multiview Codingcited by 1,297
  13. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  14. 2016Colorful Image Colorizationcited by 1,113
  15. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  16. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  17. 2019Contrastive Representation Distillationcited by 734
  18. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  19. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  20. 2016MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognitioncited by 621
  21. 2018Do Better ImageNet Models Transfer Better?cited by 560
  22. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  23. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  24. 2019On the Efficacy of Knowledge Distillationcited by 377
  25. 2019S4L: Self-Supervised Semi-Supervised Learningcited by 335
  26. 2019Variational Information Distillation for Knowledge Transfercited by 321
  27. 2018Model compression via distillation and quantizationcited by 287
  28. 2019Structured Knowledge Distillation for Dense Predictioncited by 277
  29. 2019Revisiting Self-Supervised Visual Representation Learningcited by 250
  30. 2016Joint Unsupervised Learning of Deep Representations and Image Clusterscited by 205
  31. 2016Range Loss for Deep Face Recognition with Long-tailcited by 111
  32. 2017Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracycited by 100
  33. 2019AET vs. AED: Unsupervised Representation Learning by Auto-Encoding Transformations rather than Datacited by 77
  34. 2020CompRess: Self-Supervised Learning by Compressing Representationscited by 61
  35. 2020MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Trickscited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 31; 2022: 36; 2023: 27; 2024: 25; 2025: 10; 2026: 3.

132 papers.

Sort:

By area:

  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  2. 2021Co2L: Contrastive Continual Learningcited by 243
  3. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  4. 2022Cross-Image Relational Knowledge Distillation for Semantic Segmentationcited by 117
  5. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
  6. 2021Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillationcited by 104
  7. 2022PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretrainingcited by 104
  8. 2022Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learningcited by 90
  9. 2023CLIP-KD: An Empirical Study of CLIP Model Distillationcited by 87
  10. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 85
  11. 2021ReSSL: Relational Self-Supervised Learning with Weak Augmentationcited by 83
  12. 2023Online Prototype Learning for Online Continual Learningcited by 53
38 more
  1. 2021A Fast Knowledge Distillation Framework for Visual Recognitioncited by 50
  2. 2021Improving Transferability of Representations via Augmentation-Aware Self-Supervisioncited by 48
  3. 2022Online Continual Learning for Embedded Devicescited by 48
  4. 2021Mutual Contrastive Learning for Visual Representation Learningcited by 42
  5. 2023A General-Purpose Self-Supervised Model for Computational Pathologycited by 42
  6. 2022What Makes Good Contrastive Learning on Small-Scale Wearable-based Tasks?cited by 39
  7. 2023Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Trainingcited by 39
  8. 2022CMD: Self-supervised 3D Action Representation Learning with Cross-modal Mutual Distillationcited by 38
  9. 2022Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognitioncited by 36
  10. 2021Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillationcited by 35
  11. 2021Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learningcited by 34
  12. 2022Contrastive Positive Mining for Unsupervised 3D Action Representation Learningcited by 34
  13. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 33
  14. 2023Distilling Large Vision-Language Model with Out-of-Distribution Generalizabilitycited by 33
  15. 2022Revisiting Label Smoothing and Knowledge Distillation Compatibility: What was Missing?cited by 32
  16. 2022A Closer Look at Self-Supervised Lightweight Vision Transformerscited by 31
  17. 2021A Practical Contrastive Learning Framework for Single-Image Super-Resolutioncited by 30
  18. 2024Federated Graph Semantic and Structural Learningcited by 30
  19. 2021Enhancing Self-supervised Video Representation Learning via Multi-level Feature Optimizationcited by 27
  20. 2021DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learningcited by 26
  21. 2022Modality-Aware Contrastive Instance Learning with Self-Distillation for Weakly-Supervised Audio-Visual Violence Detectioncited by 26
  22. 2023TinyMIM: An Empirical Study of Distilling MIM Pre-trained Modelscited by 24
  23. 2023SIESTA: Efficient Online Continual Learning with Sleepcited by 24
  24. 2021Distill on the Go: Online knowledge distillation in self-supervised learningcited by 22
  25. 2021Bag of Instances Aggregation Boosts Self-supervised Distillationcited by 22
  26. 2022Multimodal Adaptive Distillation for Leveraging Unimodal Encoders for Vision-Language Taskscited by 22
  27. 2022SimReg: Regression as a Simple Yet Effective Tool for Self-supervised Knowledge Distillationcited by 21
  28. 2023Extending global-local view alignment for self-supervised learning with remote sensing imagerycited by 20
  29. 2023Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learningcited by 20
  30. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  31. 2024Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillationcited by 18
  32. 2023COMEDIAN: Self-Supervised Learning and Knowledge Distillation for Action Spotting using Transformerscited by 17
  33. 2022MimCo: Masked Image Modeling Pre-training with Contrastive Teachercited by 16
  34. 2022Attention Distillation: self-supervised vision transformer students need more guidancecited by 16
  35. 2023New Insights for the Stability-Plasticity Dilemma in Online Continual Learningcited by 16
  36. 2021S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibrationcited by 15
  37. 2024Wearable Accelerometer Foundation Models for Health via Knowledge Distillationcited by 15
  38. 2021On the Efficacy of Small Self-Supervised Contrastive Models without Distillation Signalscited by 14

50 most cited of 132 All 132 (JSON)