mlsanityData to 7 Oct 2026

Regularizing Class-wise Predictions via Self-knowledge Distillation

arXiv 2003.13964 ↗First posted 31 Mar 2020cs.LG cs.CV stat.MLcited by 144

Not on the map. Why?

Abstract (arXiv)

Deep neural networks with millions of parameters may suffer from poor generalization due to overfitting. To mitigate the issue, we propose a new regularization method that penalizes the predictive distribution between similar samples. In particular, we distill the predictive distribution between different samples of the same label during training. This results in regularizing the dark knowledge (i.e., the knowledge on wrong predictions) of a single network (i.e., a self-knowledge distillation) by forcing it to produce more meaningful and consistent predictions in a class-wise manner. Consequently, it mitigates overconfident predictions and reduces intra-class variations. Our experimental results on various image classification tasks demonstrate that the simple yet powerful method can significantly improve not only the generalization ability but also the calibration performance of modern convolutional neural networks.

Key facts. Cited by 144 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 33; 2022: 36; 2023: 30; 2024: 18; 2025: 12; 2026: 15.

144 papers.

Sort:

By area:

  1. 2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Trainingcited by 91
  2. 2023From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labelscited by 69
  3. 2022Robust Cross-Modal Representation Learning with Progressive Self-Distillationcited by 55
  4. 2021Excavating the Potential Capacity of Self-Supervised Monocular Depth Estimationcited by 48
  5. 2021Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillationcited by 46
  6. 2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 44
  7. 2023Rethinking Confidence Calibration for Failure Predictioncited by 43
  8. 2021Semantic Concentration for Domain Adaptationcited by 42
  9. 2022Bi-directional Weakly Supervised Knowledge Distillation for Whole Slide Image Classificationcited by 42
  10. 2021Confidence-Aware Multi-Teacher Knowledge Distillationcited by 39
  11. 2022Fortuitous Forgetting in Connectionist Networkscited by 39
  12. 2021Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Modelcited by 37
38 more
  1. 2021Undistillable: Making A Nasty Teacher That CANNOT teach studentscited by 36
  2. 2022Self-Distilled Vision Transformer for Domain Generalizationcited by 35
  3. 2022CD2-pFed: Cyclic Distillation-guided Channel Decoupling for Model Personalization in Federated Learningcited by 34
  4. 2021Student Network Learning via Evolutionary Knowledge Distillationcited by 30
  5. 2022MixSKD: Self-Knowledge Distillation from Mixup for Image Recognitioncited by 30
  6. 2022Self-Distillation from the Last Mini-Batch for Consistency Regularizationcited by 29
  7. 2023A Machine Learning-oriented Survey on Tiny Machine Learningcited by 26
  8. 2021Semi-supervised Domain Adaptation via Sample-to-Sample Self-Distillationcited by 23
  9. 2021Embedding Transfer with Label Relaxation for Improved Metric Learningcited by 21
  10. 2022Cross-Domain Ensemble Distillation for Domain Generalizationcited by 21
  11. 2021LGD: Label-guided Self-distillation for Object Detectioncited by 19
  12. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 19
  13. 2021Knowledge Evolution in Neural Networkscited by 18
  14. 2022Self-Knowledge Distillation via Dropoutcited by 18
  15. 2022Respecting Transfer Gap in Knowledge Distillationcited by 18
  16. 2024Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillationcited by 18
  17. 2021Self-distillation with Batch Knowledge Ensembling Improves ImageNet Classificationcited by 16
  18. 2021A Short Study on Compressing Decoder-Based Language Modelscited by 16
  19. 2021Pro-KD: Progressive Distillation by Following the Footsteps of the Teachercited by 16
  20. 2023KD-DLGAN: Data Limited Image Generation via Knowledge Distillationcited by 16
  21. 2024PracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalizationcited by 15
  22. 2022Label Semantic Knowledge Distillation for Unbiased Scene Graph Generationcited by 14
  23. 2022Efficient One Pass Self-distillation with Zipf's Label Smoothingcited by 12
  24. 2021Federated Few-Shot Learning with Adversarial Learningcited by 11
  25. 2021Distilling Self-Knowledge From Contrastive Links to Classify Graph Nodes Without Passing Messagescited by 11
  26. 2022Learning to Purification for Unsupervised Person Re-identificationcited by 11
  27. 2022Multi-Faceted Distillation of Base-Novel Commonality for Few-shot Object Detectioncited by 11
  28. 2023Temporal Knowledge Sharing enable Spiking Neural Network Learning from Past and Futurecited by 10
  29. 2024Weakly Supervised Monocular 3D Detection with a Single-View Imagecited by 10
  30. 2021Deep Hash Distillation for Image Retrievalcited by 9
  31. 2022Meta-Learning with Self-Improving Momentum Targetcited by 9
  32. 2022Adaptive Label Smoothing with Self-Knowledge in Natural Language Generationcited by 9
  33. 2023Self-distillation for surgical action recognitioncited by 9
  34. 2023Learning Contrastive Self-Distillation for Ultra-Fine-Grained Visual Categorization Targeting Limited Samplescited by 9
  35. 2024Revisiting Confidence Estimation: Towards Reliable Failure Predictioncited by 9
  36. 2021Distilling EEG Representations via Capsules for Affective Computingcited by 8
  37. 2021Semi-Online Knowledge Distillationcited by 8
  38. 2023GIST: Improving Parameter Efficient Fine Tuning via Knowledge Interactioncited by 8

50 most cited of 144 All 144 (JSON)