mlsanityData to 14 Oct 2026

Knowledge distillation for LLMs

623 papers6 this weekHigh confidence in this grouping: the papers form one clear topicHigh confidence in this grouping: the papers form one clear topic · About

Keywords: language, distillation, knowledge, reasoning, llms

Key facts. Knowledge distillation for LLMs: 623 papers on arXiv since 2021 (data through 14 Oct 2026). 93 of its 623 mapped papers (15%) cite “On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes” (2023, arXiv 2306.13649). 28 new papers 25 Sep–14 Oct.

What’s new

25 Sep – 14 Oct 2026

28 new papers, 25 Sep – 14 Oct; 6 this week.

  1. This week · 6
  2. CoTranslator: Reverse to Forward Chain-of-Thought Translator for Reasoning Distillation
  3. Qwen-MT-Lumen at WMT2026: Instruction-Specialized Translation via Quality-Aware Training and On-Policy Distillation
  4. PA1-OPD: One-Step Prefix-Augmented On-Policy Distillation for Language Model Pretraining also fits: On-policy and self-distillation for LLMs
  5. Learning from Self-Generated Responses: Reliability--Uncertainty Self-Distillation for Reasoning
  6. Distilling Text LMs into Speech-to-Speech LMs by Matching Internal Text Predictions
  7. Avoiding What Your Teacher Already Knows: Entity Familiarity for Deep-Search Distillation
  8. Earlier · 22
  9. CUSE: Fine-Grained Multimodal Unlearning with Counterfactual Guidance and On-Policy Self-Distillation
  10. BoundaryEvo: Data-Model Co-Evolution at the Capability Boundary for Long-Chain Reasoning Distillation
  11. Mitigating Capability Degradation in LLM Self-Improvement via Representation Geometry Regularization also fits: On-policy and self-distillation for LLMs
  12. COFFEQA: Counterfactual-Guided Few-Shot Distillation for LLM Question Answering
All 28
  1. Universal Textual Teaching for LLMs
  2. Lapras: Latent Reasoning for Time Series Language Models
  3. KDFP: A first-principles approach to knowledge distillation in large language models
  4. MIRROR: From Imitation to Internalization in LLM Personalization
  5. Collaborative Reasoning Distillation via Cross-Feedback and Coherent Curation
  6. OnlineQAT: On-Policy Distillation for Ultra-Low-Bit Large Language Models
  7. WASD: Wasserstein-based Knowledge Distillation for Large Language Models
  8. Learning What to Distill: Bilevel Top-K Token Selection for Self-Distillation in Large Language Models also fits: On-policy and self-distillation for LLMs
  9. Sharpen Without Search: On-Policy Distillation of Sequence-Level Power Distribution also fits: On-policy and self-distillation for LLMs
  10. Investigating Model Compression for Neural Machine Translation in the Biomedical Domain
  11. Distilling Directional Verification
  12. K2P: Label-Free Knowledge to Prompt Distillation
  13. Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models
  14. Dual-Vocabulary Language Model for Cross-Tokenizer Distillation also fits: On-policy and self-distillation for LLMs
  15. A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensions also fits: General knowledge distillation methods
  16. SoFT: Soft Targets for Generalizable LLM Fine-Tuning
  17. User Model Extraction via Belief Self-Distillation
  18. Understanding the Role of Prompt Template in Knowledge Distillation for Safety Alignment

Foundations

Most cited by papers here

  1. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes2023cited by 93 here
  2. f-Divergence Minimization for Sequence-Level Knowledge Distillation2023cited by 56 here
  3. Distilling Reasoning Capabilities into Smaller Language Models2022cited by 35 here
  4. A Survey on Knowledge Distillation of Large Language Models2024cited by 33 here
  5. DistiLLM: Towards Streamlined Distillation for Large Language Models2024cited by 32 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 11 here
  2. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 93 here
  3. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 33 here
  4. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 16 here
20 more it builds on, not in this chain
  1. 2021Annealing Knowledge Distillationcited by 21 here
  2. 2021Selective Knowledge Distillation for Neural Machine Translationcited by 22 here
  3. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 16 here
  4. 2021Does Knowledge Distillation Really Work?cited by 17 here
  5. 2021Distilling Linguistic Context for Language Model Compressioncited by 16 here
  6. 2021Dynamic Knowledge Distillation for Pre-trained Language Modelscited by 16 here
  7. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 17 here
  8. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 35 here
  9. 2023SCOTT: Self-Consistent Chain-of-Thought Distillationcited by 19 here
  10. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 26 here
  11. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 56 here
  12. 2023Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penaltycited by 11 here
  13. 2023MCC-KD: Multi-CoT Consistent Knowledge Distillationcited by 16 here
  14. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 32 here
  15. 2024Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillationcited by 12 here
  16. 2024Revisiting Knowledge Distillation for Autoregressive Language Modelscited by 19 here
  17. 2024Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMscited by 21 here
  18. 2024PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuningcited by 11 here
  19. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Modelscited by 19 here
  20. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 14 here

The 24 most cited of 441.

Browse

30 of 623, ranked 6–35

  1. 6Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step202326
  2. 7Selective Knowledge Distillation for Neural Machine Translation202122
  3. 8Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs202421
  4. 9Revisiting Knowledge Distillation for Autoregressive Language Models202419
  5. 10Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models202419
  6. 11SCOTT: Self-Consistent Chain-of-Thought Distillation202319
  7. 12Symbolic Knowledge Distillation: from General Language Models to Commonsense Models202117
  8. 13MCC-KD: Multi-CoT Consistent Knowledge Distillation202316
  9. 14Distilling Linguistic Context for Language Model Compression202116
  10. 15Compact Language Models via Pruning and Knowledge Distillation202414
All 30
  1. 16Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillation202412
  2. 17PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning202411
  3. 18Dual-Space Knowledge Distillation for Large Language Models202411
  4. 19MiniLLM: On-Policy Distillation of Large Language Models202311
  5. 20Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty202311
  6. 21Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application202410
  7. 22Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks202310
  8. 23MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillation20219
  9. 24Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning20258
  10. 25AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression20238
  11. 26Lifting the Curse of Capacity Gap in Distilling Language Models20238
  12. 27Pre-training Distillation for Large Language Models: A Design Space Exploration20247
  13. 28Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models20247
  14. 29Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation20237
  15. 30Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching20256
  16. 31ToDi: Token-wise Distillation via Fine-Grained Divergence Control20256
  17. 32Learning to Maximize Mutual Information for Chain-of-Thought Distillation20246
  18. 33Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarios20246
  19. 34Mentor-KD: Making Small Language Models Better Multi-step Reasoners20246
  20. 35MiniPLM: Knowledge Distillation for Pre-Training Language Models20246