mlsanityData to 7 Oct 2026

Extracting knowledge from features with multilevel abstraction

arXiv 2112.13642 ↗First posted 4 Dec 2021cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation aims at transferring the knowledge from a large teacher model to a small student model with great improvements of the performance of the student model. Therefore, the student network can replace the teacher network to deploy on low-resource devices since the higher performance, lower number of parameters and shorter inference time. Self-knowledge distillation (SKD) attracts a great attention recently that a student model itself is a teacher model distilling knowledge from. To the best of our knowledge, self knowledge distillation can be divided into two main streams: data augmentation and refined knowledge auxiliary. In this paper, we purpose a novel SKD method in a different way from the main stream methods. Our method distills knowledge from multilevel abstraction features. Experiments and ablation studies show its great effectiveness and generalization on various kinds of tasks with various kinds of model structures. Our codes have been released on GitHub.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 17 of its 24 references that are in the map. First posted on arXiv 4 Dec 2021.

Builds on

17 of 24 references.

References · 17

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  4. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  5. 2018YOLOv3: An Incremental Improvementcited by 2,652
  6. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  7. 2017Deep Mutual Learningcited by 621
  8. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
  9. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  10. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  11. 2017Visualizing Deep Neural Network Decisions: Prediction Difference Analysiscited by 166
  12. 2019Learning Lightweight Lane Detection CNNs by Self Attention Distillationcited by 162
5 more
  1. 2021Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillationcited by 46
  2. 20163D Deeply Supervised Network for Automatic Liver Segmentation from CT Volumescited by 43
  3. 2020EventSR: From Asynchronous Events to Image Reconstruction, Restoration, and Super-Resolution via End-to-End Adversarial Learningcited by 40
  4. 2020Channel Distillation: Channel-Wise Attention for Knowledge Distillationcited by 20
  5. 2019Deeply-supervised Knowledge Synergycited by 18

Built on by

Not cited yet.