mlsanityData to 14 Oct 2026

Improve Knowledge Distillation via Label Revision and Data Selection

arXiv 2404.03693 ↗First posted 3 Apr 2024cs.LG cs.AIcited by 5

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Knowledge distillation (KD) has become a widely used technique in the field of model compression, which aims to transfer knowledge from a large teacher model to a lightweight student model for efficient network development. In addition to the supervision of ground truth, the vanilla KD method regards the predictions of the teacher as soft labels to supervise the training of the student model. Based on vanilla KD, various approaches have been developed to further improve the performance of the student model. However, few of these previous methods have considered the reliability of the supervision from teacher models. Supervision from erroneous predictions may mislead the training of the student model. This paper therefore proposes to tackle this problem from two aspects: Label Revision to rectify the incorrect supervision and Data Selection to select appropriate samples for distillation to reduce the impact of erroneous supervision. In the former, we propose to rectify the teacher's inaccurate predictions using the ground truth. In the latter, we introduce a data selection technique to choose suitable training samples to be supervised by the teacher, thereby reducing the impact of incorrect predictions to some extent. Experiment results demonstrate the effectiveness of our proposed method, and show that our method can be combined with other distillation approaches, improving their performance.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 52 of its 75 references that are in the map. First posted on arXiv 3 Apr 2024.

Builds on

52 of 75 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Sampling to Distill: Knowledge Transfer from Open-World Data31 Jul
  2. 2022Curriculum Temperature for Knowledge Distillation29 Nov
  3. 2022Knowledge Distillation with the Reused Teacher Classifier26 Mar
  4. 2022Decoupled Knowledge Distillation16 Mar
  5. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation19 May
  6. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective1 Feb

Other references · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  5. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  7. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  8. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  9. 2020Knowledge Distillation: A Surveycited by 1,603
  10. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  11. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  12. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
34 more
  1. 2019Relational Knowledge Distillationcited by 889
  2. 2019Contrastive Representation Distillationcited by 748
  3. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  4. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 629
  5. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  6. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  7. 2019Similarity-Preserving Knowledge Distillationcited by 516
  8. 2023Studying Large Language Model Generalization with Influence Functionscited by 326
  9. 2019Variational Information Distillation for Knowledge Transfercited by 324
  10. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 316
  11. 2019Variational Adversarial Active Learningcited by 284
  12. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 260
  13. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 213
  14. 2020Cross-Layer Distillation with Semantic Calibrationcited by 143
  15. 2020Knowledge Distillation Meets Self-Supervisioncited by 142
  16. 2022Training Data Influence Analysis and Estimation: A Surveycited by 135
  17. 2019Data Valuation using Reinforcement Learningcited by 116
  18. 2021Influence Selection for Active Learningcited by 84
  19. 2019Knowledge Distillation via Route Constrained Optimizationcited by 75
  20. 2020Heterogeneous Knowledge Distillation using Information Flow Modelingcited by 73
  21. 2017Learning to select data for transfer learning with Bayesian Optimizationcited by 64
  22. 2022PUMA: Performance Unchanged Model Augmentation for Training Data Removalcited by 58
  23. 2021Reliable Adversarial Distillation with Unreliable Teacherscited by 56
  24. 2022Measuring the Effect of Training Data on Deep Learning Predictions via Randomized Experimentscited by 50
  25. 2018Self-supervised Knowledge Distillation Using Singular Value Decompositioncited by 48
  26. 2019Less Is Better: Unweighted Data Subsampling via Influence Functioncited by 30
  27. 2019Preparing Lessons: Improve Knowledge Distillation with Better Supervisioncited by 27
  28. 2019Highlight Every Step: Knowledge Distillation via Collaborative Teachingcited by 21
  29. 2019Variational Student: Learning Compact and Sparser Networks in Knowledge Distillation Frameworkcited by 8
  30. 2022Complexity-Driven CNN Compression for Resource-constrained Edge AIcited by 8
  31. 2023Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillationcited by 7
  32. 2021A Self-Distillation Embedded Supervised Affinity Attention Model for Few-Shot Segmentationcited by 6
  33. 2020Generating Long Financial Report using Conditional Variational Autoencoders with Knowledge Distillationcited by 3
  34. 2022Deep Insights of Learning based Micro Expression Recognition: A Perspective on Promises, Challenges and Research Needscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 1.

5 papers.

  1. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  2. 2024Knowledge Distillation with Refined Logitscited by 9
  3. 2025Progressive Class-level Distillationcited by 0
  4. 2025Parameter-Free Logit Distillation via Sorting Mechanismcited by 0
  5. 2026REDistill: Robust Estimator Distillation for Balancing Robustness and Efficiencycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Improved knowledge distillation by utilizing backward pass knowledge in neural networks
  2. 2021Knowledge Distillation as Semiparametric Inference
  3. 2025HPM-KD: Hierarchical Progressive Multi-Teacher Framework for Knowledge Distillation and Efficient Model Compression
  4. 2022Parameter-Efficient and Student-Friendly Knowledge Distillation
  5. 2021Dynamic Knowledge Distillation for Pre-trained Language Models
  6. 2023From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels
  7. 2026Progressive2: A Teacher-Student Progressive Co-Evolving Knowledge Distillation Method for Substantial Model Compression
  8. 2021Augmenting Knowledge Distillation With Peer-To-Peer Mutual Learning For Model Compression