mlsanityData to 14 Oct 2026

Multi-granularity for knowledge distillation

arXiv 2108.06681 ↗First posted 15 Aug 2021cs.CVcited by 0

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Considering the fact that students have different abilities to understand the knowledge imparted by teachers, a multi-granularity distillation mechanism is proposed for transferring more understandable knowledge for student networks. A multi-granularity self-analyzing module of the teacher network is designed, which enables the student network to learn knowledge from different teaching patterns. Furthermore, a stable excitation scheme is proposed for robust supervision for the student training. The proposed distillation mechanism can be embedded into different distillation frameworks, which are taken as baselines. Experiments show the mechanism improves the accuracy by 0.58% on average and by 1.08% in the best over the baselines, which makes its performance superior to the state-of-the-arts. It is also exploited that the student's ability of fine-tuning and robustness to noisy inputs can be improved via the proposed mechanism. The code is available at https://github.com/shaoeric/multi-granularity-distillation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 36 of its 54 references that are in the map. First posted on arXiv 15 Aug 2021.

Builds on

36 of 54 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matching5 Feb
  2. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective1 Feb
  3. 2021Knowledge Distillation Methods for Efficient Unsupervised Adaptation Across Multiple Domains18 Jan

Other references · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  5. 2016Wide Residual Networkscited by 3,070
  6. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  8. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  9. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  10. 2020Knowledge Distillation: A Surveycited by 1,603
  11. 2014How transferable are features in deep neural networks?cited by 1,225
  12. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
21 more
  1. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  2. 2019Relational Knowledge Distillationcited by 889
  3. 2019Contrastive Representation Distillationcited by 748
  4. 2017Deep Mutual Learningcited by 628
  5. 2019Similarity-Preserving Knowledge Distillationcited by 516
  6. 2017Beyond Part Models: Person Retrieval with Refined Part Pooling (and a Strong Convolutional Baseline)cited by 473
  7. 2020Circle Loss: A Unified Perspective of Pair Similarity Optimizationcited by 403
  8. 2019A Comprehensive Overhaul of Feature Distillationcited by 348
  9. 2019Variational Information Distillation for Knowledge Transfercited by 324
  10. 2019Omni-Scale Feature Learning for Person Re-Identificationcited by 295
  11. 2019Correlation Congruence for Knowledge Distillationcited by 285
  12. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 260
  13. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 238
  14. 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 215
  15. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 213
  16. 2018Harmonious Attention Network for Person Re-Identificationcited by 211
  17. 2020Cross-Layer Distillation with Semantic Calibrationcited by 143
  18. 2018Multi-Level Factorisation Net for Person Re-Identificationcited by 58
  19. 2020PyTorch Metric Learningcited by 57
  20. 2019Torchreid: A Library for Deep Learning Person Re-Identification in Pytorchcited by 41
  21. 2020Faster Person Re-Identificationcited by 15

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Adaptive Multi-Teacher Multi-level Knowledge Distillation
  2. 2022Knowledge Condensation Distillation
  3. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillation
  4. 2021Multi-Modality Distillation via Learning the teacher's modality-level Gram Matrix
  5. 2021Partial to Whole Knowledge Distillation: Progressive Distilling Decomposed Knowledge Boosts Student Better
  6. 2021Fixing the Teacher-Student Knowledge Discrepancy in Distillation
  7. 2025Efficient Knowledge Distillation via Curriculum Extraction
  8. 2023Education distillation:getting student models to learn in shcools