# General knowledge distillation methods

Area: [Knowledge distillation](https://mlsanity.com/distill/index.md). HTML: https://mlsanity.com/distill/general-knowledge-distillation-methods/

## Key facts

- General knowledge distillation methods: 900 papers on arXiv since 2021 (data through 14 Oct 2026).
- 160 of its 900 mapped papers (18%) cite “Decoupled Knowledge Distillation” (2022, arXiv 2203.08679).
- 146 of its 900 mapped papers (16%) cite “Distilling Knowledge via Knowledge Review” (2021, arXiv 2104.09044).
- 14 new papers 25 Sep–14 Oct.

## Most cited here

- [Decoupled Knowledge Distillation](https://mlsanity.com/paper/2203.08679/) (2022), cited by 160 papers here
- [Distilling Knowledge via Knowledge Review](https://mlsanity.com/paper/2104.09044/) (2021), cited by 146 papers here
- [Knowledge Distillation from A Stronger Teacher](https://mlsanity.com/paper/2205.10536/) (2022), cited by 91 papers here
- [Knowledge Distillation with the Reused Teacher Classifier](https://mlsanity.com/paper/2203.14001/) (2022), cited by 69 papers here
- [Does Knowledge Distillation Really Work?](https://mlsanity.com/paper/2106.05945/) (2021), cited by 65 papers here
- [Knowledge distillation: A good teacher is patient and consistent](https://mlsanity.com/paper/2106.05237/) (2021), cited by 63 papers here
- [Curriculum Temperature for Knowledge Distillation](https://mlsanity.com/paper/2211.16231/) (2022), cited by 56 papers here
- [Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective](https://mlsanity.com/paper/2102.00650/) (2021), cited by 53 papers here
- [Towards Understanding Knowledge Distillation](https://mlsanity.com/paper/2105.13093/) (2021), cited by 50 papers here
- [Logit Standardization in Knowledge Distillation](https://mlsanity.com/paper/2403.01427/) (2024), cited by 49 papers here

## New (25 Sep–14 Oct)

- [CARD: Closing the Teacher-Student Loop for Adversarial Robustness Distillation](https://mlsanity.com/paper/2610.19911/) (2026-10-14)
- [Data Density Scaling Laws for Image Self-Distillation](https://mlsanity.com/paper/2610.19013/) (2026-10-13)
- [DisaggKD: Efficient Knowledge Distillation through Teacher-Student Disaggregation and a Fused Loss](https://mlsanity.com/paper/2610.17644/) (2026-10-13)
- [Representational Difference Distillation](https://mlsanity.com/paper/2610.15997/) (2026-10-11)
- [RINO: Renormalization Group Distillation with No Labels](https://mlsanity.com/paper/2610.15919/) (2026-10-11)
- [The Gradient That Top-k Distillation Never Computes](https://mlsanity.com/paper/2610.15589/) (2026-10-11)
- [Collaboratively Guided Adversarial Robust Distillation with Teacher-Favorable Examples](https://mlsanity.com/paper/2610.11306/) (2026-10-08)
- [Decoupled Optimization for Teacher-Student Semi-Supervised Learning via a Pioneer Student](https://mlsanity.com/paper/2610.09609/) (2026-10-07)
- [Distillation of Tabular Foundation Models into Efficient Predictors](https://mlsanity.com/paper/2610.01435/) (2026-10-01)
- [Gromov-Wasserstein Distillation for Inductive Multi-View Embedding](https://mlsanity.com/paper/2609.40047/) (2026-09-30)
- [Can Representation Learning Decouple from Loss Minimization? Polar Updates Have an Answer](https://mlsanity.com/paper/2609.36240/) (2026-09-28)
- [When Less Data Favors Smaller Teachers: Rethinking Teacher Capacity and Data Selection for Knowledge Distillation](https://mlsanity.com/paper/2609.34489/) (2026-09-28)
- [An End-to-End Latent-Rollout Approach for Pushing Few-Step ImageNet-256 Generation to FID 1.11 without Fr'echet Losses](https://mlsanity.com/paper/2609.32376/) (2026-09-26)
- [A Flatness-Generalization Relation in the Teacher-Student Tree-Committee Machine](https://mlsanity.com/paper/2609.31101/) (2026-09-25)