mlsanityData to 14 Oct 2026

Revisiting Self-Distillation

arXiv 2206.08491 ↗First posted 17 Jun 2022cs.LGcited by 24

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Knowledge distillation is the procedure of transferring "knowledge" from a large model (the teacher) to a more compact one (the student), often being used in the context of model compression. When both models have the same architecture, this procedure is called self-distillation. Several works have anecdotally shown that a self-distilled student can outperform the teacher on held-out data. In this work, we systematically study self-distillation in a number of settings. We first show that even with a highly accurate teacher, self-distillation allows a student to surpass the teacher in all cases. Secondly, we revisit existing theoretical explanations of (self) distillation and identify contradicting examples, revealing possible drawbacks of these explanations. Finally, we provide an alternative explanation for the dynamics of self-distillation through the lens of loss landscape geometry. We conduct extensive experiments to show that self-distillation leads to flatter minima, thereby resulting in better generalization.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 46 of its 66 references that are in the map. First posted on arXiv 17 Jun 2022.

Builds on

46 of 66 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data27 Oct
  2. 2021Does Knowledge Distillation Really Work?10 Jun
  3. 2021Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation25 Feb

Other references · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  7. 2015SSD: Single Shot MultiBox Detectorcited by 3,130
  8. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,917
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  10. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  11. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  12. 2020Knowledge Distillation: A Surveycited by 1,603
31 more
  1. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  2. 2019Natural Adversarial Examplescited by 1,408
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  4. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  5. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  6. 2019Relational Knowledge Distillationcited by 889
  7. 2019Contrastive Representation Distillationcited by 748
  8. 2018AutoAugment: Learning Augmentation Policies from Datacited by 730
  9. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  10. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  11. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  12. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  13. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  14. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 423
  15. 2019On the Efficacy of Knowledge Distillationcited by 381
  16. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
  17. 2019Variational Information Distillation for Knowledge Transfercited by 324
  18. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
  19. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  20. 2018Model compression via distillation and quantizationcited by 289
  21. 2015Unifying distillation and privileged informationcited by 233
  22. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 173
  23. 2017Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracycited by 101
  24. 2020Self-Distillation as Instance-Specific Label Smoothingcited by 71
  25. 2017DarkRank: Accelerating Deep Metric Learning via Cross Sample Similarities Transfercited by 56
  26. 2020Deep Semi-supervised Knowledge Distillation for Overlapping Cervical Cell Instance Segmentationcited by 30
  27. 2019Distillation ≈ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Networkcited by 28
  28. 2020DE-RRD: A Knowledge Distillation Framework for Recommender Systemcited by 28
  29. 2020Entropic gradient descent algorithms and wide flat minimacited by 25
  30. 2021Topology Distillation for Recommender Systemcited by 24
  31. 2017Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networkscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 5; 2024: 5; 2025: 5; 2026: 6.

24 papers.

Sort:

By area:

  1. 2023Understanding Self-Distillation in the Presence of Label Noisecited by 21
  2. 2025PRIME: Large Language Model Personalization with Cognitive Dual-Memory and Personalized Thought Processcited by 18
  3. 2023On student-teacher deviations in distillation: does it pay to disobey?cited by 16
  4. 2024Towards a theory of model distillationcited by 15
  5. 2024Align and Distill: Unifying and Improving Domain Adaptive Object Detectioncited by 12
  6. 2026Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Modelscited by 9
  7. 2022Fast Yet Effective Speech Emotion Recognition with Self-distillationcited by 7
  8. 2022Beyond Invariance: Test-Time Label-Shift Adaptation for Distributions with "Spurious" Correlationscited by 7
  9. 2023Simulated Annealing in Early Layers Leads to Better Generalizationcited by 6
  10. 2025SDRT: Enhance Vision-Language Models by Self-Distillation with Diverse Reasoning Tracescited by 5
  11. 2024FedDistill: Global Model Distillation for Local Model De-Biasing in Non-IID Federated Learningcited by 4
  12. 2024Quantifying Context Bias in Domain Adaptation for Object Detectioncited by 2
12 more
  1. 2025ATAS: Any-to-Any Self-Distillation for Enhanced Open-Vocabulary Dense Predictioncited by 2
  2. 2026SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoningcited by 2
  3. 2025Smooth-Distill: A Self-distillation Framework for Multitask Learning with Wearable Sensor Datacited by 1
  4. 2026On the Generalization of Knowledge Distillation: An Information-Theoretic Viewcited by 1
  5. 2022SADT: Combining Sharpness-Aware Minimization with Self-Distillation for Improved Model Generalizationcited by 0
  6. 2023UNFUSED: UNsupervised Finetuning Using SElf supervised Distillationcited by 0
  7. 2023Bayesian Optimization Meets Self-Distillationcited by 0
  8. 2024An Efficient Self-Learning Framework For Interactive Spoken Dialog Systemscited by 0
  9. 2025Mitigating Context Bias in Domain Adaptation for Object Detection using Mask Poolingcited by 0
  10. 2026HAD: Heterogeneity-Aware Distillation for Lifelong Heterogeneous Learningcited by 0
  11. 2026Generative Diffusion Prior Distillation for Long-Context Knowledge Transfercited by 0
  12. 2026Misanthrope: A Privacy-Preserving Keypoint Detectorcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Disentangling Self-Distillation: Measuring and Modeling Acquisition and Retention
  2. 2026LoOPSD: On-Policy Self-Distillation with Looped Teachers
  3. 2025UNDO: Understanding Distillation as Optimization
  4. 2026Randomly Initialized Networks Can Learn from Peer-to-Peer Consensus
  5. 2025A Functional Perspective on Knowledge Distillation in Neural Networks
  6. 2023Random Teachers are Good Teachers
  7. 2026Geometric Self-Distillation for Reasoning Generalization
  8. 2021Knowledge Distillation as Semiparametric Inference