mlsanityData to 7 Oct 2026

A Comprehensive Overhaul of Feature Distillation

arXiv 1904.01866 ↗First posted 3 Apr 2019cs.CV cs.LGcited by 341

Not on the map. Why?

Abstract (arXiv)

We investigate the design aspects of feature distillation methods achieving network compression and propose a novel feature distillation method in which the distillation loss is designed to make a synergy among various aspects: teacher transform, student transform, distillation feature position and distance function. Our proposed distillation loss includes a feature transform with a newly designed margin ReLU, a new distillation feature position, and a partial L2 distance function to skip redundant information giving adverse effects to the compression of student. In ImageNet, our proposed method achieves 21.65% of top-1 error with ResNet50, which outperforms the performance of the teacher network, ResNet152. Our proposed method is evaluated on various tasks such as image classification, object detection and semantic segmentation and achieves a significant performance improvement in all tasks. The code is available at https://sites.google.com/view/byeongho-heo/overhaul

Key facts. Cited by 341 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 42; 2022: 63; 2023: 75; 2024: 82; 2025: 50; 2026: 29.

341 papers.

Sort:

By area:

  1. 2022Decoupled Knowledge Distillationcited by 380
  2. 2021Distilling Knowledge via Knowledge Reviewcited by 285
  3. 2022Knowledge Distillation from A Stronger Teachercited by 204
  4. 2022Unifying Voxel-based Representation with Transformer for 3D Object Detectioncited by 202
  5. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 195
  6. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 165
  7. 2021Focal and Global Knowledge Distillation for Detectorscited by 138
  8. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 131
  9. 2021Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labelscited by 117
  10. 2022Cross-Image Relational Knowledge Distillation for Semantic Segmentationcited by 117
  11. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  12. 2022Masked Generative Distillationcited by 107
38 more
  1. 2021Distilling Object Detectors via Decoupled Featurescited by 105
  2. 2024Logit Standardization in Knowledge Distillationcited by 101
  3. 2023One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillationcited by 77
  4. 2023Decoupled Multimodal Distilling for Emotion Recognitioncited by 76
  5. 2024Continual Learning and Catastrophic Forgettingcited by 73
  6. 2022Knowledge Distillation via the Target-aware Transformercited by 70
  7. 2021LIGA-Stereo: Learning LiDAR Geometry Aware Representations for Stereo-based 3D Detectorcited by 69
  8. 2023From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labelscited by 69
  9. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  10. 2023Class Attention Transfer Based Knowledge Distillationcited by 61
  11. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillationcited by 58
  12. 2023Decoupled Kullback-Leibler Divergence Losscited by 58
  13. 2022LiDAR Distillation: Bridging the Beam-Induced Domain Gap for 3D Object Detectioncited by 54
  14. 2023Knowledge Diffusion for Distillationcited by 51
  15. 2023CrossKD: Cross-Head Knowledge Distillation for Object Detectioncited by 50
  16. 2021Excavating the Potential Capacity of Self-Supervised Monocular Depth Estimationcited by 48
  17. 2023FeatureNeRF: Learning Generalizable NeRFs by Distilling Foundation Modelscited by 47
  18. 2021Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillationcited by 46
  19. 2021EvDistill: Asynchronous Events to End-task Learning via Bidirectional Reconstruction-guided Cross-modal Knowledge Distillationcited by 46
  20. 2022Exploring Inter-Channel Correlation for Diversity-preserved KnowledgeDistillationcited by 44
  21. 2022Transfer without Forgettingcited by 44
  22. 2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 44
  23. 2021Multi-Target Domain Adaptation with Collaborative Consistency Learningcited by 43
  24. 2023Masked Image Modeling with Local Multi-Scale Reconstructioncited by 43
  25. 2024Not All Voxels Are Equal: Hardness-Aware Semantic Scene Completion with Self-Distillationcited by 43
  26. 2022Bi-directional Weakly Supervised Knowledge Distillation for Whole Slide Image Classificationcited by 42
  27. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillationcited by 41
  28. 2022Cross-Architecture Knowledge Distillationcited by 40
  29. 2023A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidancecited by 40
  30. 2021PP-OCRv2: Bag of Tricks for Ultra Lightweight OCR Systemcited by 39
  31. 2021Distilling Object Detectors with Feature Richnesscited by 38
  32. 2022Masked Autoencoders Enable Efficient Knowledge Distillerscited by 36
  33. 2022ViTKD: Practical Guidelines for ViT feature knowledge distillationcited by 34
  34. 2022PointDistiller: Structured Knowledge Distillation Towards Efficient and Compact 3D Detectioncited by 33
  35. 2022Few-shot Image Generation via Adaptation-Aware Kernel Modulationcited by 33
  36. 2022Revisiting Label Smoothing and Knowledge Distillation Compatibility: What was Missing?cited by 32
  37. 2021Dual Transfer Learning for Event-based End-task Prediction via Pluggable Event to Image Translationcited by 31
  38. 2021Data-Free Knowledge Transfer: A Surveycited by 31

50 most cited of 341 All 341 (JSON)