# Knowledge distillation for LLMs

Area: [Knowledge distillation](https://mlsanity.com/distill/index.md). HTML: https://mlsanity.com/distill/knowledge-distillation-for-llms/

## Key facts

- Knowledge distillation for LLMs: 623 papers on arXiv since 2021 (data through 14 Oct 2026).
- 93 of its 623 mapped papers (15%) cite “On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes” (2023, arXiv 2306.13649).
- 56 of its 623 mapped papers (9%) cite “f-Divergence Minimization for Sequence-Level Knowledge Distillation” (2023, arXiv 2307.15190).
- 28 new papers 25 Sep–14 Oct.

## Most cited here

- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023), cited by 93 papers here
- [f-Divergence Minimization for Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/2307.15190/) (2023), cited by 56 papers here
- [Distilling Reasoning Capabilities into Smaller Language Models](https://mlsanity.com/paper/2212.00193/) (2022), cited by 35 papers here
- [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024), cited by 33 papers here
- [DistiLLM: Towards Streamlined Distillation for Large Language Models](https://mlsanity.com/paper/2402.03898/) (2024), cited by 32 papers here
- [Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step](https://mlsanity.com/paper/2306.14050/) (2023), cited by 26 papers here
- [Selective Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2105.12967/) (2021), cited by 22 papers here
- [Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs](https://mlsanity.com/paper/2402.12030/) (2024), cited by 21 papers here
- [SCOTT: Self-Consistent Chain-of-Thought Distillation](https://mlsanity.com/paper/2305.01879/) (2023), cited by 19 papers here
- [Revisiting Knowledge Distillation for Autoregressive Language Models](https://mlsanity.com/paper/2402.11890/) (2024), cited by 19 papers here

## New (25 Sep–14 Oct)

- [CoTranslator: Reverse to Forward Chain-of-Thought Translator for Reasoning Distillation](https://mlsanity.com/paper/2610.20828/) (2026-10-14)
- [Qwen-MT-Lumen at WMT2026: Instruction-Specialized Translation via Quality-Aware Training and On-Policy Distillation](https://mlsanity.com/paper/2610.19758/) (2026-10-14)
- [PA1-OPD: One-Step Prefix-Augmented On-Policy Distillation for Language Model Pretraining](https://mlsanity.com/paper/2610.19601/) (2026-10-14)
- [Learning from Self-Generated Responses: Reliability--Uncertainty Self-Distillation for Reasoning](https://mlsanity.com/paper/2610.19176/) (2026-10-13)
- [Distilling Text LMs into Speech-to-Speech LMs by Matching Internal Text Predictions](https://mlsanity.com/paper/2610.17461/) (2026-10-12)
- [Avoiding What Your Teacher Already Knows: Entity Familiarity for Deep-Search Distillation](https://mlsanity.com/paper/2610.16251/) (2026-10-12)
- [CUSE: Fine-Grained Multimodal Unlearning with Counterfactual Guidance and On-Policy Self-Distillation](https://mlsanity.com/paper/2610.14763/) (2026-10-10)
- [BoundaryEvo: Data-Model Co-Evolution at the Capability Boundary for Long-Chain Reasoning Distillation](https://mlsanity.com/paper/2610.13285/) (2026-10-09)
- [Mitigating Capability Degradation in LLM Self-Improvement via Representation Geometry Regularization](https://mlsanity.com/paper/2610.12855/) (2026-10-09)
- [COFFEQA: Counterfactual-Guided Few-Shot Distillation for LLM Question Answering](https://mlsanity.com/paper/2610.12837/) (2026-10-09)
- [Universal Textual Teaching for LLMs](https://mlsanity.com/paper/2610.12114/) (2026-10-08)
- [Lapras: Latent Reasoning for Time Series Language Models](https://mlsanity.com/paper/2610.11111/) (2026-10-08)
- [KDFP: A first-principles approach to knowledge distillation in large language models](https://mlsanity.com/paper/2610.10854/) (2026-10-07)
- [MIRROR: From Imitation to Internalization in LLM Personalization](https://mlsanity.com/paper/2610.09795/) (2026-10-07)
- [Collaborative Reasoning Distillation via Cross-Feedback and Coherent Curation](https://mlsanity.com/paper/2610.09587/) (2026-10-07)
- [OnlineQAT: On-Policy Distillation for Ultra-Low-Bit Large Language Models](https://mlsanity.com/paper/2610.09346/) (2026-10-07)
- [WASD: Wasserstein-based Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2610.07706/) (2026-10-06)
- [Learning What to Distill: Bilevel Top-K Token Selection for Self-Distillation in Large Language Models](https://mlsanity.com/paper/2610.07247/) (2026-10-05)
- [Sharpen Without Search: On-Policy Distillation of Sequence-Level Power Distribution](https://mlsanity.com/paper/2610.06804/) (2026-10-05)
- [Investigating Model Compression for Neural Machine Translation in the Biomedical Domain](https://mlsanity.com/paper/2610.07032/) (2026-10-04)
- [Distilling Directional Verification](https://mlsanity.com/paper/2610.00997/) (2026-10-01)
- [K2P: Label-Free Knowledge to Prompt Distillation](https://mlsanity.com/paper/2609.38898/) (2026-09-30)
- [Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models](https://mlsanity.com/paper/2609.36734/) (2026-09-29)
- [Dual-Vocabulary Language Model for Cross-Tokenizer Distillation](https://mlsanity.com/paper/2609.33816/) (2026-09-27)
- [A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensions](https://mlsanity.com/paper/2609.33727/) (2026-09-27)
- [SoFT: Soft Targets for Generalizable LLM Fine-Tuning](https://mlsanity.com/paper/2609.32493/) (2026-09-26)
- [User Model Extraction via Belief Self-Distillation](https://mlsanity.com/paper/2609.31603/) (2026-09-25)
- [Understanding the Role of Prompt Template in Knowledge Distillation for Safety Alignment](https://mlsanity.com/paper/2609.30802/) (2026-09-25)