mlsanityData to 7 Oct 2026

Explaining Knowledge Distillation by Quantifying the Knowledge

arXiv 2003.03622 ↗First posted 7 Mar 2020cs.LG cs.CV stat.MLcited by 65

Not on the map. Why?

Abstract (arXiv)

This paper presents a method to interpret the success of knowledge distillation by quantifying and analyzing task-relevant and task-irrelevant visual concepts that are encoded in intermediate layers of a deep neural network (DNN). More specifically, three hypotheses are proposed as follows. 1. Knowledge distillation makes the DNN learn more visual concepts than learning from raw data. 2. Knowledge distillation ensures that the DNN is prone to learning various visual concepts simultaneously. Whereas, in the scenario of learning from raw data, the DNN learns visual concepts sequentially. 3. Knowledge distillation yields more stable optimization directions than learning from raw data. Accordingly, we design three types of mathematical metrics to evaluate feature representations of the DNN. In experiments, we diagnosed various DNNs, and above hypotheses were verified.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 13; 2023: 15; 2024: 12; 2025: 5; 2026: 6.

65 papers.

Sort:

By area:

  1. 2022Decoupled Knowledge Distillationcited by 380
  2. 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AIcited by 299
  3. 2022Class-Incremental Continual Learning into the eXtended DER-versecited by 121
  4. 2021General Instance Distillation for Object Detectioncited by 113
  5. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  6. 2021Instance-Conditional Knowledge Distillation for Object Detectioncited by 42
  7. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillationcited by 41
  8. 2022Explaining Deepfake Detection by Analysing Image Matchingcited by 34
  9. 2021Student Network Learning via Evolutionary Knowledge Distillationcited by 30
  10. 2021Knowledge Distillation as Semiparametric Inferencecited by 29
  11. 2021Embracing the Dark Knowledge: Domain Generalization Using Regularized Knowledge Distillationcited by 28
  12. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
38 more
  1. 2021Semi-supervised Domain Adaptation via Sample-to-Sample Self-Distillationcited by 23
  2. 2022Asymmetric Temperature Scaling Makes Larger Networks Teach Well Againcited by 23
  3. 2022Teacher-Student Architecture for Knowledge Learning: A Surveycited by 23
  4. 2023Understanding Self-Distillation in the Presence of Label Noisecited by 20
  5. 2022Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classificationcited by 19
  6. 2022Compressing Deep Graph Neural Networks via Adversarial Knowledge Distillationcited by 18
  7. 2023Computation-efficient Deep Learning for Computer Vision: A Surveycited by 14
  8. 2022Narrowing the Coordinate-frame Gap in Behavior Prediction Models: Distillation for Efficient and Accurate Scene-centric Motion Forecastingcited by 10
  9. 2023Dividing and Conquering a BlackBox to a Mixture of Interpretable Models: Route, Interpret, Repeatcited by 10
  10. 2023EA-KD: Entropy-based Adaptive Knowledge Distillationcited by 8
  11. 2021Visualizing the Emergence of Intermediate Visual Patterns in DNNscited by 7
  12. 2023Zone Evaluation: Revealing Spatial Bias in Object Detectioncited by 7
  13. 2023Graph-based Knowledge Distillation: A survey and experimental evaluationcited by 6
  14. 2023Towards a Smaller Student: Capacity Dynamic Distillation for Efficient Image Retrievalcited by 6
  15. 2024Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformerscited by 6
  16. 2024CKD: Contrastive Knowledge Distillation from A Sample-wise Perspectivecited by 6
  17. 2024Decoupling Dark Knowledge via Block-wise Logit Distillation for Feature-level Alignmentcited by 6
  18. 2024Data Laundering: Artificially Boosting Benchmark Results through Knowledge Distillationcited by 6
  19. 2023Towards the Fundamental Limits of Knowledge Transfer over Finite Domainscited by 4
  20. 2024Exploring Dark Knowledge under Various Teacher Capacities and Addressing Capacity Mismatchcited by 4
  21. 2022Towards Feature Distribution Alignment and Diversity Enhancement for Data-Free Quantizationcited by 3
  22. 2024Multi-Level Feature Distillation of Joint Teachers Trained on Distinct Image Datasetscited by 3
  23. 2023Frameless Graph Knowledge Distillationcited by 2
  24. 2025A Survey on Interpretability in Visual Recognitioncited by 2
  25. 2026EGAD: Entropy-Guided Adaptive Distillation for Token-Level Knowledge Transfercited by 2
  26. 2021Hands-on Guidance for Distilling Object Detectorscited by 1
  27. 2021Efficient Action Recognition Using Confidence Distillationcited by 1
  28. 2021Partial to Whole Knowledge Distillation: Progressive Distilling Decomposed Knowledge Boosts Student Bettercited by 1
  29. 2021Boosting Unsupervised Domain Adaptation with Soft Pseudo-label and Curriculum Learningcited by 1
  30. 2022On effects of Knowledge Distillation on Transfer Learningcited by 1
  31. 2023Practical Knowledge Distillation: Using DNNs to Beat DNNscited by 1
  32. 2023Leveraging Synthetic Targets for Machine Translationcited by 1
  33. 2023Exploring the Lottery Ticket Hypothesis with Explainability Methods: Insights into Sparse Network Performancecited by 1
  34. 2024Multi-perspective Contrastive Logit Distillationcited by 1
  35. 2021Spectral Roll-off Points Variations: Exploring Useful Information in Feature Maps by Its Variationscited by 0
  36. 2021Dynamic Knowledge Distillation With Noise Elimination for RGB-D Salient Object Detectioncited by 0
  37. 2021Visualizing the embedding space to explain the effect of knowledge distillationcited by 0
  38. 2022A General Multiple Data Augmentation Based Framework for Training Deep Neural Networkscited by 0

50 most cited of 65 All 65 (JSON)