mlsanityData to 14 Oct 2026

MiniLLM: On-Policy Distillation of Large Language Models

arXiv 2306.08543 ↗First posted 14 Jun 2023cs.CL cs.AIcited by 139

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Knowledge Distillation (KD) is a promising technique for reducing the high computational demand of large language models (LLMs). However, previous KD methods are primarily applied to white-box classification models or training small models to imitate black-box model APIs like ChatGPT. How to effectively distill the knowledge of white-box LLMs into small models is still under-explored, which becomes more important with the prosperity of open-source LLMs. In this work, we propose a KD approach that distills LLMs into smaller language models. We first replace the forward Kullback-Leibler divergence (KLD) objective in the standard KD approaches with reverse KLD, which is more suitable for KD on generative language models, to prevent the student model from overestimating the low-probability regions of the teacher distribution. Then, we derive an effective on-policy optimization approach to learn this objective. The student models are named MiniLLM. Extensive experiments in the instruction-following setting show that MiniLLM generates more precise responses with higher overall quality, lower exposure bias, better calibration, and higher long-text generation performance than the baselines. Our method is scalable for different model families with 120M to 13B parameters. Our code, data, and model checkpoints can be found in https://github.com/microsoft/LMOps/tree/main/minillm.

Key facts. Cited by 139 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 63 of its 81 references that are in the map. First posted on arXiv 14 Jun 2023.

Builds on

63 of 81 references.

References · 63

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
38 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  6. 2022Emergent Abilities of Large Language Modelscited by 2,512
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  11. 2020Knowledge Distillation: A Surveycited by 1,603
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  13. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  14. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  15. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  16. 2023PaLM 2 Technical Reportcited by 1,162
  17. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  18. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  19. 2016Sequence-Level Knowledge Distillationcited by 801
  20. 2023Instruction Tuning with GPT-4cited by 686
  21. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  22. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  23. 2018Born Again Neural Networkscited by 546
  24. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  25. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 506
  26. 2018Behavioral Cloning from Observationcited by 492
  27. 2019Measuring Calibration in Deep Learningcited by 446
  28. 2015Policy Distillationcited by 440
  29. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  30. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  31. 2023The False Promise of Imitating Proprietary LLMscited by 263
  32. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 254
  33. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  34. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  35. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 108
  36. 2019Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustnesscited by 104
  37. 2018Language GANs Falling Shortcited by 97
  38. 2019Distilling Policy Distillationcited by 84

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 19; 2024: 40; 2025: 26; 2026: 54.

139 papers.

Sort:

By area:

  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  2. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  3. 2023A Survey on Model Compression for Large Language Modelscited by 269
  4. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  5. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  6. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  7. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  8. 2023Efficient Large Language Models: A Surveycited by 186
  9. 2026On-Policy Context Distillation for Language Modelscited by 186
  10. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 165
  11. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 163
  12. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
38 more
  1. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 126
  2. 2024Knowledge Fusion of Large Language Modelscited by 115
  3. 2023Online Speculative Decodingcited by 113
  4. 2026TIP: Token Importance in On-Policy Distillationcited by 104
  5. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 94
  6. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 90
  7. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  8. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Timecited by 78
  9. 2024CLLMs: Consistency Large Language Modelscited by 72
  10. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Modelscited by 65
  11. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  12. 2024FuseChat: Knowledge Fusion of Chat Modelscited by 55
  13. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 55
  14. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 49
  15. 2024A Survey on Transformer Compressioncited by 46
  16. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 45
  17. 2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 44
  18. 2024Everybody Prune Now: Structured Pruning of LLMs with only Forward Passescited by 43
  19. 2024Accurate LoRA-Finetuning Quantization of LLMs via Information Retentioncited by 41
  20. 2025Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokenscited by 41
  21. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Modelscited by 38
  22. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Modelscited by 38
  23. 2025Position: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!cited by 37
  24. 2023Integration of Large Language Models and Federated Learningcited by 31
  25. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  26. 2024STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMscited by 31
  27. 2024ARB-LLM: Alternating Refined Binarizations for Large Language Modelscited by 31
  28. 2026When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoningcited by 30
  29. 2026PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competencecited by 28
  30. 2026Online Experiential Learning for Language Modelscited by 28
  31. 2023LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compressioncited by 26
  32. 2024LLAVADI: What Matters For Multimodal Large Language Models Distillationcited by 26
  33. 2024Q-Sparse: All Large Language Models can be Fully Sparsely-Activatedcited by 25
  34. 2025Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasonerscited by 22
  35. 2024Online Cascade Learning for Efficient Inference over Streamscited by 20
  36. 2024Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQAcited by 19
  37. 2024Distillation of Diffusion Features for Semantic Correspondencecited by 17
  38. 2026Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifierscited by 15

50 most cited of 139 All 139 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024DistiLLM: Towards Streamlined Distillation for Large Language Models
  2. 2026Hybrid Policy Distillation for LLMs
  3. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application
  4. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Models
  5. 2026Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?
  6. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
  7. 2026KDFP: A first-principles approach to knowledge distillation in large language models
  8. 2024A Survey on Knowledge Distillation of Large Language Models