mlsanityData to 14 Oct 2026

DSPNet: Towards Slimmable Pretrained Networks based on Discriminative Self-supervised Learning

arXiv 2207.06075 ↗First posted 13 Jul 2022cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) has achieved promising downstream performance. However, when facing various resource budgets in real-world applications, it costs a huge computation burden to pretrain multiple networks of various sizes one by one. In this paper, we propose Discriminative-SSL-based Slimmable Pretrained Networks (DSPNet), which can be trained at once and then slimmed to multiple sub-networks of various sizes, each of which faithfully learns good representation and can serve as good initialization for downstream tasks with various resource budgets. Specifically, we extend the idea of slimmable networks to a discriminative SSL paradigm, by integrating SSL and knowledge distillation gracefully. We show comparable or improved performance of DSPNet on ImageNet to the networks individually pretrained one by one under the linear evaluation and semi-supervised evaluation protocols, while reducing large training cost. The pretrained models also generalize well on downstream detection and segmentation tasks. Code will be made public.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 46 of its 54 references that are in the map. First posted on arXiv 13 Jul 2022.

Builds on

46 of 54 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2017Mask R-CNNcited by 7,277
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  11. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  12. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
34 more
  1. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  3. 2016Wide Residual Networkscited by 3,070
  4. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  5. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  7. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  8. 2019Searching for MobileNetV3cited by 1,994
  9. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  10. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  11. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  12. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  13. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  14. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  15. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  16. 2016Colorful Image Colorizationcited by 1,114
  17. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  18. 2019Relational Knowledge Distillationcited by 889
  19. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  20. 2019Contrastive Representation Distillationcited by 748
  21. 2017Large Batch Training of Convolutional Networkscited by 535
  22. 2018Slimmable Neural Networkscited by 305
  23. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 300
  24. 2019Universally Slimmable Networks and Improved Training Techniquescited by 221
  25. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 188
  26. 2020Knowledge Distillation Meets Self-Supervisioncited by 142
  27. 2021SEED: Self-supervised Distillation For Visual Representationcited by 134
  28. 2019Knowledge Adaptation for Efficient Semantic Segmentationcited by 105
  29. 2017MegDet: A Large Mini-Batch Object Detectorcited by 66
  30. 2020CompRess: Self-Supervised Learning by Compressing Representationscited by 62
  31. 2019MutualNet: Adaptive ConvNet via Mutual Learning from Network Width and Resolutioncited by 36
  32. 2021DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learningcited by 26
  33. 2021Distill on the Go: Online knowledge distillation in self-supervised learningcited by 22
  34. 2021Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learningcited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Slimmable Networks for Contrastive Self-supervised Learning
  2. 2023A Simple Recipe for Competitive Low-compute Self supervised Vision Models
  3. 2024A Review on Discriminative Self-supervised Learning Methods in Computer Vision
  4. 2022Effective Self-supervised Pre-training on Low-compute Networks without Distillation
  5. 2023Unifying Synergies between Self-supervised Learning and Dynamic Computation
  6. 2025Unsupervised Transformer Pre-Training for Images: Self-Distillation, Mean Teachers, and Random Crops
  7. 2024On Improving the Algorithm-, Model-, and Data- Efficiency of Self-Supervised Learning
  8. 2021Self-Distilled Self-Supervised Representation Learning