mlsanityData to 14 Oct 2026

One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation

arXiv 2608.25936 ↗First posted 26 Aug 2026cs.LG cs.AI cs.CLcited by 0

On the map: Knowledge distillation → On-policy and self-distillation for LLMs

also in 1 more
Abstract (arXiv)

On-policy distillation trains a language model on its own generations while a teacher scores them token by token. It combines the dense supervision of imitation learning with the on-policy sampling of reinforcement learning. But it requires a second, larger model to act as teacher. On-Policy Self-Distillation (OPSD) removes that cost. The teacher is the model itself, conditioned on privileged information the student will not have at test time, such as a reference solution, a plan, or environment feedback. The teacher is no stronger than the student, only better informed. Early results were promising, with accuracy comparable to reinforcement learning at a fraction of the generated tokens. But the same asymmetry that produces the signal also biases it. One failure mode now dominates the field: collapse, the progressive narrowing of the set of reasoning paths the model can produce. Collapse is not specific to OPSD, though privileged information aggravates it. This review treats collapse as a symptom governed by three levers: (i) where the signal is applied, that is, how tokens are weighted; (ii) what the teacher is shown, that is, the nature of the privileged information; and (iii) when the signal changes, that is, the teacher's dynamics and the decay of guidance. We restrict our scope to mathematical reasoning, where the method originated and where its failure modes are best documented. We report no new experiments. The contribution is structural: a shared vocabulary for phenomena named differently across papers, and a clear line between what is settled and what is still disputed.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 56 of its 62 references that are in the map. First posted on arXiv 26 Aug 2026.

Builds on

56 of 62 references.

Closest ancestors · 25

Same direction, newest first

  1. 2026Rethinking On-Policy Self-Distillation for Thinking Models6 Jul
  2. 2026DemoPSD: Disagreement-Modulated Policy Self-Distillation2 Jul
  3. 2026Purified OPSD: On-Policy Self-Distillation Without Losing How to Think2 Jul
  4. 2026Denser neq Better: Limits of On-Policy Self-Distillation for Continual Post-Training2 Jul
  5. 2026DOPD: Dual On-policy Distillation29 Jun
  6. 2026MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training29 Jun
  7. 2026On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversity24 Jun
  8. 2026A Formula-Driven Survey and Research Agenda for On-Policy Distillation22 Jun
  9. 2026Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation17 Jun
  10. 2026Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoning14 Jun
  11. 2026The Role of Feedback Alignment in Self-Distillation9 Jun
  12. 2026Beyond Absolute Imitation: Anchored Residual Guidance for Privileged On-Policy Distillation9 Jun
13 more
  1. 2026When Should the Teacher Move? Temporal Coupling and Stability in Self On-Policy Distillation2 Jun
  2. 2026Trust Region On-Policy Distillation31 May
  3. 2026Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservation26 May
  4. 2026Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information12 May
  5. 2026Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning12 May
  6. 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why11 May
  7. 2026PAINT: Partial-Solution Adaptive Interpolated Training for Self-Distilled Reasoners29 Apr
  8. 2026A Survey of On-Policy Distillation for Large Language Models1 Apr
  9. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?25 Mar
  10. 2026Entropy-Aware On-Policy Distillation of Language Models7 Mar
  11. 2026Reinforcement Learning via Self-Distillation28 Jan
  12. 2026Self-Distillation Enables Continual Learning27 Jan
  13. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models26 Jan

Other references · 31

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  4. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  6. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  7. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  8. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  9. 2018Born Again Neural Networkscited by 546
  10. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 508
  11. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  12. 2015Policy Distillationcited by 440
19 more
  1. 2019Learning by Cheatingcited by 383
  2. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 297
  3. 2015Unifying distillation and privileged informationcited by 233
  4. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 209
  5. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 176
  6. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 173
  7. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 163
  8. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 151
  9. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 113
  10. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 93
  11. 2025Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contaminationcited by 82
  12. 2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learningcited by 65
  13. 2024Are Your LLMs Capable of Stable Reasoning?cited by 65
  14. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 60
  15. 2025UFT: Unifying Supervised and Reinforcement Fine-Tuningcited by 59
  16. 2020Bridging the Imitation Gap by Adaptive Insubordinationcited by 38
  17. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 37
  18. 2023How to Scale Your EMAcited by 24
  19. 2026A Brief Overview: On-Policy Self-Distillation In Large Language Modelscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
  2. 2026Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement
  3. 2026Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective
  4. 2026Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals
  5. 2026Recursive Self-Improvement via On-Policy Distillation for Reasoning
  6. 2026Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations
  7. 2026Demystifying On-Policy Distillation: Where Does the Teacher's Advantage Live?
  8. 2026Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation