mlsanityData to 14 Oct 2026

Self-Knowledge Distillation via Dropout

arXiv 2208.05642 ↗First posted 11 Aug 2022cs.CVcited by 18

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

To boost the performance, deep neural networks require deeper or wider network structures that involve massive computational and memory costs. To alleviate this issue, the self-knowledge distillation method regularizes the model by distilling the internal knowledge of the model itself. Conventional self-knowledge distillation methods require additional trainable parameters or are dependent on the data. In this paper, we propose a simple and effective self-knowledge distillation method using a dropout (SD-Dropout). SD-Dropout distills the posterior distributions of multiple models through a dropout sampling. Our method does not require any additional trainable modules, does not rely on data, and requires only simple operations. Furthermore, this simple method can be easily combined with various self-knowledge distillation approaches. We provide a theoretical and experimental analysis of the effect of forward and reverse KL-divergences in our work. Extensive experiments on various vision tasks, i.e., image classification, object detection, and distribution shift, demonstrate that the proposed method can effectively improve the generalization of a single network. Further experiments show that the proposed method also improves calibration performance, adversarial robustness, and out-of-distribution detection ability.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 38 of its 50 references that are in the map. First posted on arXiv 11 Aug 2022.

Builds on

38 of 50 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  9. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  10. 2017Mask R-CNNcited by 7,277
  11. 2016Densely Connected Convolutional Networkscited by 6,720
  12. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
26 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  2. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  3. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  4. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  6. 2013Describing Textures in the Wildcited by 2,125
  7. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  8. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,355
  9. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,328
  10. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,057
  11. 2017Deep Mutual Learningcited by 628
  12. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  13. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  14. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 472
  15. 2019On the Efficacy of Knowledge Distillationcited by 381
  16. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 213
  17. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 195
  18. 2015TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Trackingcited by 165
  19. 2019Ensemble Distribution Distillationcited by 158
  20. 2020Regularizing Class-wise Predictions via Self-knowledge Distillationcited by 144
  21. 2019Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustnesscited by 104
  22. 2020OBoW: Online Bag-of-Visual-Words Generation for Self-Supervised Learningcited by 65
  23. 2018Knowledge Distillation with Adversarial Samples Supporting Decision Boundarycited by 49
  24. 2018Dropout Distillation for Efficiently Estimating Model Confidencecited by 9
  25. 2018Improving Confidence Estimates for Unfamiliar Examplescited by 2
  26. 2019Efficient Evaluation-Time Uncertainty Estimation by Improved Distillationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 5; 2026: 7.

18 papers.

Sort:

By area:

  1. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 139
  2. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Modelscited by 65
  3. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 56
  4. 2025ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergencecited by 23
  5. 2023Masking meets Supervision: A Strong Learning Alliancecited by 3
  6. 2024SalNAS: Efficient Saliency-prediction Neural Architecture Search with self-knowledge distillationcited by 1
  7. 2025Smooth-Distill: A Self-distillation Framework for Multitask Learning with Wearable Sensor Datacited by 1
  8. 2025From Deferral to Learning: Online In-Context Knowledge Distillation for LLM Cascadescited by 1
  9. 2023Self-distilled Masked Attention guided masked image modeling with noise Regularized Teacher (SMART) for medical image analysiscited by 0
  10. 2025Knowledge Distillation for Variational Quantum Convolutional Neural Networks on Heterogeneous Datacited by 0
  11. 2025Winning the Lottery by Preserving Network Training Dynamics with Concrete Ticket Searchcited by 0
  12. 2026SGD-Based Knowledge Distillation with Bayesian Teachers: Theory and Guidelinescited by 0
6 more
  1. 2026A Multi-Prototype-Guided Federated Knowledge Distillation Approach in AI-RAN Enabled Multi-Access Edge Computing Systemcited by 0
  2. 2026HEART-PFL: Stable Personalized Federated Learning under Heterogeneity with Hierarchical Directional Alignment and Adversarial Knowledge Transfercited by 0
  3. 2026Co-distilled attention guided masked image modeling with noisy teacher for self-supervised learning on medical imagescited by 0
  4. 2026The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Workscited by 0
  5. 2026A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactionscited by 0
  6. 2026KDFP: A first-principles approach to knowledge distillation in large language modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021A New Training Framework for Deep Neural Network
  2. 2022AI-KD: Adversarial learning and Implicit regularization for self-Knowledge Distillation
  3. 2024Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-Training of Deep Networks
  4. 2024A Comprehensive Review of Knowledge Distillation in Computer Vision
  5. 2022Self-Distillation from the Last Mini-Batch for Consistency Regularization
  6. 2021Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation
  7. 2025Intra-class Patch Swap for Self-Distillation
  8. 2021Distill on the Go: Online knowledge distillation in self-supervised learning