mlsanityData to 7 Oct 2026

SimReg: Regression as a Simple Yet Effective Tool for Self-supervised Knowledge Distillation

arXiv 2201.05131 ↗First posted 13 Jan 2022cs.CVcited by 21

Not on the map. Why?

Abstract (arXiv)

Feature regression is a simple way to distill large neural network models to smaller ones. We show that with simple changes to the network architecture, regression can outperform more complex state-of-the-art approaches for knowledge distillation from self-supervised models. Surprisingly, the addition of a multi-layer perceptron head to the CNN backbone is beneficial even if used only during distillation and discarded in the downstream task. Deeper non-linear projections can thus be used to accurately mimic the teacher without changing inference architecture and time. Moreover, we utilize independent projection heads to simultaneously distill multiple teacher networks. We also find that using the same weakly augmented image as input for both teacher and student networks aids distillation. Experiments on ImageNet dataset demonstrate the efficacy of the proposed changes in various self-supervised distillation settings.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 40 of its 54 references that are in the map. First posted on arXiv 13 Jan 2022.

Builds on

40 of 54 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2016Densely Connected Convolutional Networkscited by 6,691
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  10. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
28 more
  1. 2016Wide Residual Networkscited by 3,057
  2. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  4. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  5. 2013Describing Textures in the Wildcited by 2,110
  6. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  7. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  8. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  9. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  10. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  11. 2019Contrastive Multiview Codingcited by 1,297
  12. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  13. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  14. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  15. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  16. 2019Relational Knowledge Distillationcited by 873
  17. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  18. 2019Contrastive Representation Distillationcited by 734
  19. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  20. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  21. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 235
  22. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 209
  23. 2020Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biasescited by 143
  24. 2021SEED: Self-supervised Distillation For Visual Representationcited by 132
  25. 2017Representation Learning by Learning to Countcited by 120
  26. 2018Boosting Self-Supervised Learning via Knowledge Transfercited by 92
  27. 2021Mean Shift for Self-Supervised Learningcited by 65
  28. 2020CompRess: Self-Supervised Learning by Compressing Representationscited by 61

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 5; 2024: 7; 2025: 4; 2026: 2.

21 papers.

Sort:

By area:

  1. 2023Understanding the Role of the Projector in Knowledge Distillationcited by 31
  2. 2024VkD: Improving Knowledge Distillation using Orthogonal Projectionscited by 24
  3. 2023Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learningcited by 20
  4. 2022Attention Distillation: self-supervised vision transformer students need more guidancecited by 16
  5. 2024On the Surprising Effectiveness of Attention Transfer for Vision Transformerscited by 16
  6. 2023Plasticity-Optimized Complementary Networks for Unsupervised Continual Learningcited by 14
  7. 2024Improving Text-guided Object Inpainting with Semantic Pre-inpaintingcited by 12
  8. 2023A Simple Recipe for Competitive Low-compute Self supervised Vision Modelscited by 10
  9. 2022Towards Sustainable Self-supervised Learningcited by 8
  10. 2024Simple Unsupervised Knowledge Distillation With Space Similaritycited by 5
  11. 2022Effective Self-supervised Pre-training on Low-compute Networks without Distillationcited by 3
  12. 2024On Good Practices for Task-Specific Distillation of Large Pretrained Visual Modelscited by 3
9 more
  1. 2024Multi-Token Enhancing for Vision Representation Learningcited by 3
  2. 2025Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimizationcited by 1
  3. 2023Unifying Synergies between Self-supervised Learning and Dynamic Computationcited by 0
  4. 2024ProFeAT: Projected Feature Adversarial Training for Self-Supervised Learning of Robust Representationscited by 0
  5. 2025MoKD: Multi-Task Optimization for Knowledge Distillationcited by 0
  6. 2025PCoreSet: Effective Active Learning through Knowledge Distillation from Vision-Language Modelscited by 0
  7. 2025PSMamba: Progressive Self-supervised Vision Mamba for Plant Disease Recognitioncited by 0
  8. 2026HQ-JEPA: Hybrid Quantum Joint-Embedding Predictive Architecture for Cross-Modal Remote Sensing Representation Learningcited by 0
  9. 2026When Diffusion Models Forget Who You Are: Identity Preservation in Face Inpainting under Large Occlusionscited by 0