mlsanityData to 7 Oct 2026

Decoupled Knowledge Distillation

arXiv 2203.08679 ↗First posted 16 Mar 2022cs.CV cs.AIcited by 380

Not on the map. Why?

Abstract (arXiv)

State-of-the-art distillation methods are mainly based on distilling deep features from intermediate layers, while the significance of logit distillation is greatly overlooked. To provide a novel viewpoint to study logit distillation, we reformulate the classical KD loss into two parts, i.e., target class knowledge distillation (TCKD) and non-target class knowledge distillation (NCKD). We empirically investigate and prove the effects of the two parts: TCKD transfers knowledge concerning the "difficulty" of training samples, while NCKD is the prominent reason why logit distillation works. More importantly, we reveal that the classical KD loss is a coupled formulation, which (1) suppresses the effectiveness of NCKD and (2) limits the flexibility to balance these two parts. To address these issues, we present Decoupled Knowledge Distillation (DKD), enabling TCKD and NCKD to play their roles more efficiently and flexibly. Compared with complex feature-based methods, our DKD achieves comparable or even better results and has better training efficiency on CIFAR-100, ImageNet, and MS-COCO datasets for image classification and object detection tasks. This paper proves the great potential of logit distillation, and we hope it will be helpful for future research. The code is available at https://github.com/megvii-research/mdistiller.

Key facts. Cited by 380 papers in the map (data through 7 Oct 2026). Builds on 36 of its 42 references that are in the map. First posted on arXiv 16 Mar 2022.

Builds on

36 of 42 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2017Mask R-CNNcited by 7,259
  9. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  10. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  11. 2017Squeeze-and-Excitation Networkscited by 4,763
  12. 2016Wide Residual Networkscited by 3,057
24 more
  1. 2016Pyramid Scene Parsing Networkcited by 2,472
  2. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  3. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  4. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  5. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
  6. 2019Relational Knowledge Distillationcited by 873
  7. 2019Contrastive Representation Distillationcited by 734
  8. 2017Deep Mutual Learningcited by 621
  9. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  10. 2018Born Again Neural Networkscited by 539
  11. 2019Similarity-Preserving Knowledge Distillationcited by 507
  12. 2019On the Efficacy of Knowledge Distillationcited by 377
  13. 2019A Comprehensive Overhaul of Feature Distillationcited by 341
  14. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 312
  15. 2021Distilling Knowledge via Knowledge Reviewcited by 285
  16. 2019Correlation Congruence for Knowledge Distillationcited by 280
  17. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 255
  18. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 235
  19. 2019Distilling Object Detectors with Fine-grained Feature Imitationcited by 188
  20. 2021Towards Understanding Knowledge Distillationcited by 158
  21. 2015Learning with Symmetric Label Noise: The Importance of Being Unhingedcited by 120
  22. 2018Snapshot Distillation: Teacher-Student Optimization in One Generationcited by 99
  23. 2020Explaining Knowledge Distillation by Quantifying the Knowledgecited by 65
  24. 2021Online Knowledge Distillation for Efficient Pose Estimationcited by 55

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 29; 2023: 101; 2024: 108; 2025: 83; 2026: 59.

380 papers.

Sort:

By area:

  1. 2023Effective Whole-body Pose Estimation with Two-stages Distillationcited by 307
  2. 2023A Survey on Model Compression for Large Language Modelscited by 263
  3. 2023EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anythingcited by 187
  4. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  5. 2022Curriculum Temperature for Knowledge Distillationcited by 110
  6. 2024Logit Standardization in Knowledge Distillationcited by 101
  7. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  8. 2023One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillationcited by 77
  9. 2023Decoupled Multimodal Distilling for Emotion Recognitioncited by 76
  10. 2024PromptKD: Unsupervised Prompt Distillation for Vision-Language Modelscited by 72
  11. 2023From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labelscited by 69
  12. 2023Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matchingcited by 69
38 more
  1. 2023Class Attention Transfer Based Knowledge Distillationcited by 61
  2. 2023Decoupled Kullback-Leibler Divergence Losscited by 58
  3. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  4. 2023Knowledge Diffusion for Distillationcited by 51
  5. 2023CrossKD: Cross-Head Knowledge Distillation for Object Detectioncited by 50
  6. 2023EVC: Towards Real-Time Neural Image Compression with Mask Decaycited by 46
  7. 2024Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitatecited by 46
  8. 2024Not All Voxels Are Equal: Hardness-Aware Semantic Scene Completion with Self-Distillationcited by 43
  9. 2022PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficientcited by 41
  10. 2024Tiny Machine Learning: Progress and Futurescited by 40
  11. 2023Generalizable Heterogeneous Federated Cross-Correlation and Instance Similarity Learningcited by 39
  12. 2023Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Modelscited by 37
  13. 2024Knowledge Distillation Based on Transformed Teacher Matchingcited by 37
  14. 2024Gradient Reweighting: Towards Imbalanced Class-Incremental Learningcited by 37
  15. 2022Masked Autoencoders Enable Efficient Knowledge Distillerscited by 36
  16. 2022ViTKD: Practical Guidelines for ViT feature knowledge distillationcited by 34
  17. 2023Visual Tuningcited by 34
  18. 2024Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalitiescited by 32
  19. 2023DisWOT: Student Architecture Search for Distillation WithOut Trainingcited by 29
  20. 2023CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservationcited by 29
  21. 2025AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineagecited by 29
  22. 2025Decoupled Distillation to Erase: A General Unlearning Method for Any Class-centric Taskscited by 28
  23. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  24. 2023Dynamic Residual Classifier for Class Incremental Learningcited by 26
  25. 2023Lifting the Curse of Capacity Gap in Distilling Language Modelscited by 25
  26. 2024Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Informationcited by 25
  27. 2024VkD: Improving Knowledge Distillation using Orthogonal Projectionscited by 24
  28. 2023Do Not Blindly Imitate the Teacher: Using Perturbed Loss for Knowledge Distillationcited by 23
  29. 2023Class-relation Knowledge Distillation for Novel Class Discoverycited by 23
  30. 2023Long-Tailed Visual Recognition via Self-Heterogeneous Integration with Knowledge Excavationcited by 22
  31. 2023NAYER: Noisy Layer Data Generation for Efficient and Effective Data-free Knowledge Distillationcited by 22
  32. 2024Frequency Attention for Knowledge Distillationcited by 22
  33. 2023Function-Consistent Feature Distillationcited by 21
  34. 2023Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?cited by 21
  35. 2023DomainAdaptor: A Novel Approach to Test-time Adaptationcited by 20
  36. 2023SlimSAM: 0.1% Data Makes Segment Anything Slimcited by 20
  37. 2024Small Scale Data-Free Knowledge Distillationcited by 20
  38. 2024DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillationcited by 20

50 most cited of 380 All 380 (JSON)