mlsanityData to 14 Oct 2026

On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes

arXiv 2306.13649 ↗First posted 23 Jun 2023cs.LG cs.AI cs.CLcited by 939

On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training

also in 1 more
Abstract (arXiv)

Knowledge distillation (KD) is widely used for compressing a teacher model to reduce its inference cost and memory footprint, by training a smaller student model. However, current KD methods for auto-regressive sequence models suffer from distribution mismatch between output sequences seen during training and those generated by the student during inference. To address this issue, we introduce Generalized Knowledge Distillation (GKD). Instead of solely relying on a fixed set of output sequences, GKD trains the student on its self-generated output sequences by leveraging feedback from the teacher on such sequences. Unlike supervised KD approaches, GKD also offers the flexibility to employ alternative loss functions between the student and teacher, which can be useful when the student lacks the expressivity to mimic the teacher's distribution. Furthermore, GKD facilitates the seamless integration of distillation with RL fine-tuning (RLHF). We demonstrate the efficacy of GKD for distilling auto-regressive language models on summarization, translation, and arithmetic reasoning tasks, and task-agnostic distillation for instruction-tuning.

Key facts. Cited by 939 papers in the map (data through 14 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 38 of its 51 references that are in the map. First posted on arXiv 23 Jun 2023.

Builds on

38 of 51 references.

References · 38

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
26 more
  1. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  2. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  3. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  4. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  5. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  6. 2016Sequence-Level Knowledge Distillationcited by 801
  7. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  8. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  9. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
  10. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  11. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  12. 2022Large Language Models Are Reasoning Teacherscited by 329
  13. 2022Teaching Small Language Models to Reasoncited by 284
  14. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 266
  15. 2015Actor-Mimic: Deep Multitask and Transfer Reinforcement Learningcited by 178
  16. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 139
  17. 2022Explanations from Large Language Models Make Small Reasoners Bettercited by 124
  18. 2019Bridging the Gap between Training and Inference for Neural Machine Translationcited by 123
  19. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 108
  20. 2015How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?cited by 78
  21. 2018A Study of Reinforcement Learning for Neural Machine Translationcited by 74
  22. 2020Autoregressive Knowledge Distillation through Imitation Learningcited by 74
  23. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 72
  24. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 65
  25. 2019Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation?cited by 35
  26. 2021Relating Neural Text Degeneration to Exposure Biascited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 52; 2025: 99; 2026: 785.

939 papers.

Sort:

By area:

  1. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  2. 2025Gemma 3 Technical Reportcited by 1,824
  3. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  4. 2026Reinforcement Learning via Self-Distillationcited by 332
  5. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 299
  6. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  7. 2026Self-Distillation Enables Continual Learningcited by 247
  8. 2026MiMo-V2-Flash Technical Reportcited by 191
  9. 2023Efficient Large Language Models: A Surveycited by 186
  10. 2026On-Policy Context Distillation for Language Modelscited by 186
  11. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 184
  12. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 175
38 more
  1. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 165
  2. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 163
  3. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 163
  4. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 148
  5. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 145
  6. 2026Self-Distilled RLVRcited by 131
  7. 2025Small Models Struggle to Learn from Strong Reasonerscited by 113
  8. 2024Approaching Human-Level Forecasting with Language Modelscited by 107
  9. 2026TIP: Token Importance in On-Policy Distillationcited by 104
  10. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 90
  11. 2026Privileged Information Distillation for Language Modelscited by 90
  12. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 90
  13. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agentscited by 90
  14. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  15. 2025Universal Model Routing for Efficient LLM Inferencecited by 86
  16. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 84
  17. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 82
  18. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 79
  19. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 78
  20. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Timecited by 78
  21. 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillationcited by 77
  22. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 76
  23. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 76
  24. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 75
  25. 2026MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Trainingcited by 72
  26. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 71
  27. 2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 68
  28. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Modelscited by 65
  29. 2025Black-Box On-Policy Distillation of Large Language Modelscited by 59
  30. 2024FuseChat: Knowledge Fusion of Chat Modelscited by 55
  31. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 55
  32. 2026Self-Distilled Agentic Reinforcement Learningcited by 55
  33. 2026Stable On-Policy Distillation through Adaptive Target Reformulationcited by 53
  34. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 53
  35. 2026Fast and Effective On-policy Distillation from Reasoning Prefixescited by 53
  36. 2026Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillationcited by 51
  37. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 50
  38. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 49

50 most cited of 939 All 939 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Revisiting Knowledge Distillation for Autoregressive Language Models
  2. 2022Gradient Knowledge Distillation for Pre-trained Language Models
  3. 2026KDFP: A first-principles approach to knowledge distillation in large language models
  4. 2024Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Models
  5. 2024Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation
  6. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
  7. 2024SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Models
  8. 2023AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression