mlsanityData to 7 Oct 2026

Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation

arXiv 1905.08094 ↗First posted 17 May 2019cs.LG stat.MLcited by 491

Not on the map. Why?

Abstract (arXiv)

Convolutional neural networks have been widely deployed in various application scenarios. In order to extend the applications' boundaries to some accuracy-crucial domains, researchers have been investigating approaches to boost accuracy through either deeper or wider network structures, which brings with them the exponential increment of the computational and storage cost, delaying the responding time. In this paper, we propose a general training framework named self distillation, which notably enhances the performance (accuracy) of convolutional neural networks through shrinking the size of the network rather than aggrandizing it. Different from traditional knowledge distillation - a knowledge transformation methodology among networks, which forces student neural networks to approximate the softmax layer outputs of pre-trained teacher neural networks, the proposed self distillation framework distills knowledge within network itself. The networks are firstly divided into several sections. Then the knowledge in the deeper portion of the networks is squeezed into the shallow ones. Experiments further prove the generalization of the proposed self distillation framework: enhancement of accuracy at average level is 2.65%, varying from 0.61% in ResNeXt as minimum to 4.07% in VGG19 as maximum. In addition, it can also provide flexibility of depth-wise scalable inference on resource-limited edge devices.Our codes will be released on github soon.

Key facts. Cited by 491 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 76; 2022: 98; 2023: 102; 2024: 81; 2025: 80; 2026: 54.

491 papers.

Sort:

By area:

  1. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  2. 2022Large Language Models Can Self-Improvecited by 656
  3. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  4. 2021Training Graph Neural Networks with 1000 Layerscited by 169
  5. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 151
  6. 2022Self-supervised Learning in Remote Sensing: A Reviewcited by 137
  7. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 131
  8. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  9. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  10. 2022Class-Incremental Continual Learning into the eXtended DER-versecited by 121
  11. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 121
  12. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 89
38 more
  1. 2023Hierarchical Dense Correlation Distillation for Few-Shot Segmentationcited by 84
  2. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decodingcited by 80
  3. 2021Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Bettercited by 77
  4. 2023One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillationcited by 77
  5. 2021Adaptive Inference through Early-Exit Networks: Design, Challenges and Directionscited by 74
  6. 2024D-FINE: Redefine Regression Task in DETRs as Fine-grained Distribution Refinementcited by 73
  7. 2023From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labelscited by 69
  8. 2021Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecturecited by 63
  9. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillationcited by 58
  10. 2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 58
  11. 2021Localization Distillation for Dense Object Detectioncited by 55
  12. 2024Large Language Models Can Self-Improve At Web Agent Taskscited by 54
  13. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 51
  14. 2021Farewell to Mutual Information: Variational Distillation for Cross-Modal Person Re-Identificationcited by 50
  15. 2021A Fast Knowledge Distillation Framework for Visual Recognitioncited by 50
  16. 2021Instance-Dependent Partial Label Learningcited by 49
  17. 2021Excavating the Potential Capacity of Self-Supervised Monocular Depth Estimationcited by 48
  18. 2023CMID: A Unified Self-Supervised Learning Framework for Remote Sensing Image Understandingcited by 48
  19. 2024LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendationcited by 47
  20. 2021Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillationcited by 46
  21. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  22. 2025Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Modelscited by 44
  23. 2021Efficient Medical Image Segmentation Based on Knowledge Distillationcited by 43
  24. 2024Not All Voxels Are Equal: Hardness-Aware Semantic Scene Completion with Self-Distillationcited by 43
  25. 2021Generate, Annotate, and Learn: NLP with Synthetic Textcited by 42
  26. 2021Semantic Concentration for Domain Adaptationcited by 42
  27. 2023Feature Alignment and Uniformity for Test Time Adaptationcited by 42
  28. 2023Towards Safe Self-Distillation of Internet-Scale Text-to-Image Diffusion Modelscited by 41
  29. 2021Improving Multi-Modal Learning with Uni-Modal Teacherscited by 40
  30. 2022Wavelet Knowledge Distillation: Towards Efficient Image-to-Image Translationcited by 39
  31. 2023Knowledge Distillation on Graphs: A Surveycited by 38
  32. 2024Lightweight Deep Learning for Resource-Constrained Environments: A Surveycited by 38
  33. 2022Rethinking the Augmentation Module in Contrastive Learning: Learning Hierarchical Augmentation Invariance with Expanded Viewscited by 37
  34. 2022Efficient Knowledge Distillation from Model Checkpointscited by 37
  35. 2023A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversationscited by 37
  36. 2021Undistillable: Making A Nasty Teacher That CANNOT teach studentscited by 36
  37. 2024Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languagescited by 36
  38. 2022How many Observations are Enough? Knowledge Distillation for Trajectory Forecastingcited by 35

50 most cited of 491 All 491 (JSON)