mlsanityData to 7 Oct 2026

Distilling Knowledge via Knowledge Review

arXiv 2104.09044 ↗First posted 19 Apr 2021cs.CVcited by 285

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation transfers knowledge from the teacher network to the student one, with the goal of greatly improving the performance of the student network. Previous methods mostly focus on proposing feature transformation and loss functions between the same level's features to improve the effectiveness. We differently study the factor of connection path cross levels between teacher and student networks, and reveal its great importance. For the first time in knowledge distillation, cross-stage connection paths are proposed. Our new review mechanism is effective and structurally simple. Our finally designed nested and compact framework requires negligible computation overhead, and outperforms other methods on a variety of tasks. We apply our method to classification, object detection, and instance segmentation tasks. All of them witness significant student network performance improvement. Code is available at https://github.com/Jia-Research-Lab/ReviewKD

Key facts. Cited by 285 papers in the map (data through 7 Oct 2026). Builds on 33 of its 41 references that are in the map. First posted on arXiv 19 Apr 2021.

Builds on

33 of 41 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2017Mask R-CNNcited by 7,259
  8. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  9. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  10. 2017Squeeze-and-Excitation Networkscited by 4,763
  11. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  12. 2016Wide Residual Networkscited by 3,057
21 more
  1. 2016Pyramid Scene Parsing Networkcited by 2,472
  2. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  3. 2019Searching for MobileNetV3cited by 1,984
  4. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  5. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  7. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  8. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  9. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  10. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  11. 2019Relational Knowledge Distillationcited by 873
  12. 2017Focal Loss for Dense Object Detectioncited by 749
  13. 2019Contrastive Representation Distillationcited by 734
  14. 2018Rethinking the Value of Network Pruningcited by 595
  15. 2019Similarity-Preserving Knowledge Distillationcited by 507
  16. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
  17. 2019A Comprehensive Overhaul of Feature Distillationcited by 341
  18. 2019Variational Information Distillation for Knowledge Transfercited by 321
  19. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 255
  20. 2019Distilling Object Detectors with Fine-grained Feature Imitationcited by 188
  21. 2018Quantization Mimic: Towards Very Tiny CNN for Object Detectioncited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 43; 2023: 74; 2024: 75; 2025: 59; 2026: 30.

285 papers.

Sort:

By area:

  1. 2022Anomaly Detection via Reverse Distillation from One-Class Embeddingcited by 419
  2. 2022Decoupled Knowledge Distillationcited by 380
  3. 2022Knowledge Distillation from A Stronger Teachercited by 204
  4. 2022Knowledge Distillation with the Reused Teacher Classifiercited by 140
  5. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  6. 2022Curriculum Temperature for Knowledge Distillationcited by 110
  7. 2022Masked Generative Distillationcited by 107
  8. 2024Logit Standardization in Knowledge Distillationcited by 101
  9. 2023One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillationcited by 77
  10. 2022Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillationcited by 72
  11. 2022Knowledge Distillation via the Target-aware Transformercited by 70
  12. 2023From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labelscited by 69
38 more
  1. 2022MonoDistill: Learning Spatial Features for Monocular 3D Object Detectioncited by 68
  2. 2023Class Attention Transfer Based Knowledge Distillationcited by 61
  3. 2023Decoupled Kullback-Leibler Divergence Losscited by 58
  4. 2023Knowledge Diffusion for Distillationcited by 51
  5. 2022Parameter-Efficient and Student-Friendly Knowledge Distillationcited by 47
  6. 2024Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitatecited by 46
  7. 2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 44
  8. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillationcited by 41
  9. 2023Leveraging Vision-Language Models for Improving Domain Generalization in Image Classificationcited by 41
  10. 2022Cross-Architecture Knowledge Distillationcited by 40
  11. 2024Lightweight Deep Learning for Resource-Constrained Environments: A Surveycited by 38
  12. 2023DreamTeacher: Pretraining Image Backbones with Deep Generative Modelscited by 37
  13. 2024Knowledge Distillation Based on Transformed Teacher Matchingcited by 37
  14. 2022DETRDistill: A Universal Knowledge Distillation Framework for DETR-familiescited by 36
  15. 2022ViTKD: Practical Guidelines for ViT feature knowledge distillationcited by 34
  16. 2023Visual Tuningcited by 34
  17. 2023Distilling Large Vision-Language Model with Out-of-Distribution Generalizabilitycited by 33
  18. 2022Switchable Online Knowledge Distillationcited by 29
  19. 2024Vision Superalignment: Weak-to-Strong Generalization for Vision Foundation Modelscited by 28
  20. 2022Improved Feature Distillation via Projector Ensemblecited by 27
  21. 2023Stitchable Neural Networkscited by 26
  22. 2023Subclass-balancing Contrastive Learning for Long-tailed Recognitioncited by 26
  23. 2024Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Informationcited by 25
  24. 2022Difficulty-Aware Simulator for Open Set Recognitioncited by 24
  25. 2024VkD: Improving Knowledge Distillation using Orthogonal Projectionscited by 24
  26. 2022Asymmetric Temperature Scaling Makes Larger Networks Teach Well Againcited by 23
  27. 2023Do Not Blindly Imitate the Teacher: Using Perturbed Loss for Knowledge Distillationcited by 23
  28. 2022Boosting 3D Object Detection by Simulating Multimodality on Point Cloudscited by 22
  29. 2024Frequency Attention for Knowledge Distillationcited by 22
  30. 2023Function-Consistent Feature Distillationcited by 21
  31. 2023Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?cited by 21
  32. 2022Knowledge Condensation Distillationcited by 20
  33. 2022PROD: Progressive Distillation for Dense Retrievalcited by 20
  34. 2021Information Theoretic Representation Distillationcited by 19
  35. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 19
  36. 2022Respecting Transfer Gap in Knowledge Distillationcited by 18
  37. 2023Generic-to-Specific Distillation of Masked Autoencoderscited by 18
  38. 2023Improving Knowledge Distillation via Regularizing Feature Norm and Directioncited by 18

50 most cited of 285 All 285 (JSON)