mlsanityData to 14 Oct 2026

Spot-adaptive Knowledge Distillation

arXiv 2205.02399 ↗First posted 5 May 2022cs.CV cs.AI cs.LGcited by 22

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Knowledge distillation (KD) has become a well established paradigm for compressing deep neural networks. The typical way of conducting knowledge distillation is to train the student network under the supervision of the teacher network to harness the knowledge at one or multiple spots (i.e., layers) in the teacher network. The distillation spots, once specified, will not change for all the training samples, throughout the whole distillation process. In this work, we argue that distillation spots should be adaptive to training samples and distillation epochs. We thus propose a new distillation strategy, termed spot-adaptive KD (SAKD), to adaptively determine the distillation spots in the teacher network per sample, at every training iteration during the whole distillation period. As SAKD actually focuses on "where to distill" instead of "what to distill" that is widely investigated by most existing works, it can be seamlessly integrated into existing distillation methods to further improve their performance. Extensive experiments with 10 state-of-the-art distillers are conducted to demonstrate the effectiveness of SAKD for improving their distillation performance, under both homogeneous and heterogeneous distillation settings. Code is available at https://github.com/zju-vipa/spot-adaptive-pytorch

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 38 of its 53 references that are in the map. First posted on arXiv 5 May 2022.

Builds on

38 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data27 Oct

Other references · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  6. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2016Wide Residual Networkscited by 3,070
  9. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  10. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  11. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  12. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
25 more
  1. 2019Relational Knowledge Distillationcited by 889
  2. 2019Contrastive Representation Distillationcited by 748
  3. 2017Deep Mutual Learningcited by 628
  4. 2019Similarity-Preserving Knowledge Distillationcited by 516
  5. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  6. 2019On the Efficacy of Knowledge Distillationcited by 381
  7. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 364
  8. 2019Variational Information Distillation for Knowledge Transfercited by 324
  9. 2019Correlation Congruence for Knowledge Distillationcited by 285
  10. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 260
  11. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 238
  12. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 213
  13. 2019Data-Free Learning of Student Networkscited by 204
  14. 2018SpotTune: Transfer Learning through Adaptive Fine-tuningcited by 168
  15. 2015Conditional Computation in Neural Networks for faster modelscited by 167
  16. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 126
  17. 2019Data-Free Adversarial Distillationcited by 90
  18. 2018Knowledge Transfer with Jacobian Matchingcited by 76
  19. 2019Knowledge Distillation via Route Constrained Optimizationcited by 75
  20. 2020Self-Distillation as Instance-Specific Label Smoothingcited by 71
  21. 2018Modular Networks: Learning to Decompose Neural Computationcited by 62
  22. 2019Routing Networks and the Challenges of Modular and Compositional Computationcited by 50
  23. 2018Amalgamating Knowledge towards Comprehensive Classificationcited by 38
  24. 2020Knowledge Transfer via Dense Cross-Layer Mutual-Distillationcited by 32
  25. 2019Student Becoming the Master: Knowledge Amalgamation for Joint Scene Parsing, Depth Estimation, and Morecited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 5; 2024: 7; 2025: 3; 2026: 3.

22 papers.

Sort:

By area:

  1. 2022Source-Free Unsupervised Domain Adaptation: A Surveycited by 80
  2. 2022Cross-Architecture Knowledge Distillationcited by 41
  3. 2024Bootstrapping Chest CT Image Understanding by Distilling Knowledge from X-ray Expert Modelscited by 24
  4. 2023Deep Graph Reprogrammingcited by 22
  5. 2024Federated Distillation: A Surveycited by 15
  6. 2023Exploring Multi-Modal Contextual Knowledge for Open-Vocabulary Object Detectioncited by 7
  7. 2024CKD: Contrastive Knowledge Distillation from A Sample-wise Perspectivecited by 7
  8. 2025Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Modelscited by 6
  9. 2022Deeply Supervised Skin Lesions Diagnosis with Stage and Branch Attentioncited by 4
  10. 2024Privacy-Preserving Synthetic Continual Semantic Segmentation for Robotic Surgerycited by 4
  11. 2025Local Dense Logit Relations for Enhanced Knowledge Distillationcited by 4
  12. 2023RdimKD: Generic Distillation Paradigm by Dimensionality Reductioncited by 3
10 more
  1. 2023Towards Few-Call Model Stealing via Active Self-Paced Knowledge Distillation and Diffusion-Based Image Generationcited by 2
  2. 2024Efficient and Robust Knowledge Distillation from A Stronger Teacher Based on Correlation Matchingcited by 2
  3. 2026UETrack: A Unified and Efficient Framework for Single Object Trackingcited by 2
  4. 2023Understanding the Effects of Projectors in Knowledge Distillationcited by 1
  5. 2024Harmonizing knowledge Transfer in Neural Network with Unified Distillationcited by 1
  6. 2022Hint-dynamic Knowledge Distillationcited by 0
  7. 2024Leveraging Knowledge Distillation for Lightweight Skin Cancer Classification: Balancing Accuracy and Computational Efficiencycited by 0
  8. 2025PLD: A Choice-Theoretic List-Wise Knowledge Distillationcited by 0
  9. 2026Integrating Knowledge Distillation Methods: A Sequential Multi-Stage Frameworkcited by 0
  10. 2026Enhanced Knowledge Distillation for Detection Transformer via Teacher Prediction Refinementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Comprehensive Survey on Knowledge Distillation
  2. 2021Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher
  3. 2023Improved knowledge distillation by utilizing backward pass knowledge in neural networks
  4. 2024Neural Collapse Inspired Knowledge Distillation
  5. 2021Adaptive Distillation: Aggregating Knowledge from Multiple Paths for Efficient Distillation
  6. 2025Intra-class Patch Swap for Self-Distillation
  7. 2022Dynamic Rectification Knowledge Distillation
  8. 2023Improving Knowledge Distillation with Teacher's Explanation