mlsanityData to 14 Oct 2026

General knowledge distillation methods

900 papers3 this weekLow confidence in this grouping: the papers may belong to more than one topic, or the group may changeLow confidence in this grouping: the papers may belong to more than one topic, or the group may change · About

Keywords: knowledge, distillation, teacher, student, knowledge distillation

Key facts. General knowledge distillation methods: 900 papers on arXiv since 2021 (data through 14 Oct 2026). 160 of its 900 mapped papers (18%) cite “Decoupled Knowledge Distillation” (2022, arXiv 2203.08679). 14 new papers 25 Sep–14 Oct.

What’s new

25 Sep – 14 Oct 2026

14 new papers, 25 Sep – 14 Oct; 3 this week.

  1. This week · 3
  2. CARD: Closing the Teacher-Student Loop for Adversarial Robustness Distillation also fits: On-policy and self-distillation for LLMs
  3. Data Density Scaling Laws for Image Self-Distillation
  4. DisaggKD: Efficient Knowledge Distillation through Teacher-Student Disaggregation and a Fused Loss
  5. Earlier · 11
  6. Representational Difference Distillation
  7. RINO: Renormalization Group Distillation with No Labels
  8. The Gradient That Top-k Distillation Never Computes also fits: On-policy and self-distillation for LLMs
  9. Collaboratively Guided Adversarial Robust Distillation with Teacher-Favorable Examples also fits: Federated, robust and private distillation
  10. Decoupled Optimization for Teacher-Student Semi-Supervised Learning via a Pioneer Student also fits: On-policy and self-distillation for LLMs
  11. Distillation of Tabular Foundation Models into Efficient Predictors also fits: Knowledge distillation for LLMs
  12. Gromov-Wasserstein Distillation for Inductive Multi-View Embedding also fits: Multimodal distillation
All 14
  1. Can Representation Learning Decouple from Loss Minimization? Polar Updates Have an Answer
  2. When Less Data Favors Smaller Teachers: Rethinking Teacher Capacity and Data Selection for Knowledge Distillation
  3. An End-to-End Latent-Rollout Approach for Pushing Few-Step ImageNet-256 Generation to FID 1.11 without Fr'echet Losses also fits: Multimodal distillation
  4. A Flatness-Generalization Relation in the Teacher-Student Tree-Committee Machine

Foundations

Most cited by papers here

  1. Decoupled Knowledge Distillation2022cited by 160 here
  2. Distilling Knowledge via Knowledge Review2021cited by 146 here
  3. Knowledge Distillation from A Stronger Teacher2022cited by 91 here
  4. Knowledge Distillation with the Reused Teacher Classifier2022cited by 69 here
  5. Does Knowledge Distillation Really Work?2021cited by 65 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 53 here
  2. 2022Cross-Image Relational Knowledge Distillation for Semantic Segmentationcited by 31 here
  3. 2022Knowledge Distillation from A Stronger Teachercited by 91 here
  4. 2024Logit Standardization in Knowledge Distillationcited by 49 here
20 more it builds on, not in this chain
  1. 2021Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matchingcited by 30 here
  2. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillationcited by 29 here
  3. 2021Distilling Knowledge via Knowledge Reviewcited by 146 here
  4. 2021Contrastive Model Inversion for Data-Free Knowledge Distillationcited by 36 here
  5. 2021Towards Understanding Knowledge Distillationcited by 50 here
  6. 2021Knowledge distillation: A good teacher is patient and consistentcited by 63 here
  7. 2021Does Knowledge Distillation Really Work?cited by 65 here
  8. 2021Hierarchical Self-supervised Augmented Knowledge Distillationcited by 22 here
  9. 2021Focal and Global Knowledge Distillation for Detectorscited by 32 here
  10. 2022Exploring Inter-Channel Correlation for Diversity-preserved KnowledgeDistillationcited by 23 here
  11. 2022Decoupled Knowledge Distillationcited by 160 here
  12. 2022Knowledge Distillation with the Reused Teacher Classifiercited by 69 here
  13. 2022DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformerscited by 22 here
  14. 2022Masked Generative Distillationcited by 31 here
  15. 2022Knowledge Distillation via the Target-aware Transformercited by 32 here
  16. 2022Curriculum Temperature for Knowledge Distillationcited by 56 here
  17. 2023From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labelscited by 35 here
  18. 2023Class Attention Transfer Based Knowledge Distillationcited by 43 here
  19. 2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 25 here
  20. 2023One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillationcited by 30 here

The 24 most cited of 607.

Browse

30 of 900, ranked 6–35

  1. 6Knowledge distillation: A good teacher is patient and consistent202163
  2. 7Curriculum Temperature for Knowledge Distillation202256
  3. 8Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective202153
  4. 9Towards Understanding Knowledge Distillation202150
  5. 10Logit Standardization in Knowledge Distillation202449
  6. 11Class Attention Transfer Based Knowledge Distillation202343
  7. 12Contrastive Model Inversion for Data-Free Knowledge Distillation202136
  8. 13From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels202335
  9. 14Knowledge Distillation via the Target-aware Transformer202232
  10. 15Masked Generative Distillation202231
All 30
  1. 16One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation202330
  2. 17Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matching202130
  3. 18Learning Student-Friendly Teacher Networks for Knowledge Distillation202129
  4. 19NORM: Knowledge Distillation via N-to-One Representation Matching202325
  5. 20Exploring Inter-Channel Correlation for Diversity-preserved KnowledgeDistillation202223
  6. 21Knowledge Distillation Based on Transformed Teacher Matching202422
  7. 22DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformers202222
  8. 23Hierarchical Self-supervised Augmented Knowledge Distillation202122
  9. 24Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation202121
  10. 25Adaptive Multi-Teacher Multi-level Knowledge Distillation202120
  11. 26Complementary Relation Contrastive Distillation202120
  12. 27Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation202218
  13. 28Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study202117
  14. 29Understanding the Role of the Projector in Knowledge Distillation202316
  15. 30Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data202116
  16. 31Function-Consistent Feature Distillation202315
  17. 32Robust and Resource-Efficient Data-Free Knowledge Distillation by Generative Pseudo Replay202215
  18. 33Efficient Knowledge Distillation from Model Checkpoints202215
  19. 34Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation202115
  20. 35Learning to Retain while Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillation202314