mlsanityData to 14 Oct 2026

Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models

arXiv 2601.18734 ↗First posted 26 Jan 2026cs.LG cs.CLcited by 455

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

also in 1 more
Abstract (arXiv)

Knowledge distillation improves large language model (LLM) reasoning by compressing the knowledge of a teacher LLM to train smaller LLMs. On-policy distillation advances this approach by having the student sample its own trajectories while a teacher LLM provides dense token-level supervision, addressing the distribution mismatch between training and inference in off-policy distillation methods. However, on-policy distillation typically requires a separate, often larger, teacher LLM and does not explicitly leverage ground-truth solutions available in reasoning datasets. Inspired by the intuition that a sufficiently capable LLM can rationalize external privileged reasoning traces and teach its weaker self, we introduce On-Policy Self-Distillation (OPSD), a learning algorithm where a single LLM acts as both teacher and student with different contexts. The teacher policy conditions on privileged information (e.g., verified reasoning traces) while the student policy sees only the question; training minimizes the per-token divergence between these distributions over the student's own rollouts. We demonstrate the efficacy of our method on multiple mathematical reasoning benchmarks, achieving superior token efficiency compared to reinforcement learning methods and better performance over off-policy distillation methods. Code repo: https://github.com/siyan-zhao/OPSD.

Key facts. Cited by 455 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 49 references that are in the map. First posted on arXiv 26 Jan 2026.

Builds on

34 of 49 references.

References · 34

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  10. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  11. 2025Group Sequence Policy Optimizationcited by 835
  12. 2016Sequence-Level Knowledge Distillationcited by 801
22 more
  1. 2024Self-Rewarding Language Modelscited by 681
  2. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  3. 2025Kimi K2: Open Agentic Intelligencecited by 567
  4. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  5. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  6. 2025LIMO: Less is More for Reasoningcited by 498
  7. 2015Policy Distillationcited by 440
  8. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 415
  9. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  10. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
  11. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Taskscited by 285
  12. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  13. 2022Learning by Distilling Contextcited by 138
  14. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 106
  15. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 96
  16. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 96
  17. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 90
  18. 2025Magistralcited by 60
  19. 2025AMO-Bench: Large Language Models Still Struggle in High School Math Competitionscited by 38
  20. 2024In-Context Editing: Learning Knowledge from Self-Induced Distributionscited by 34
  21. 2025Inpainting-Guided Policy Optimization for Diffusion Large Language Modelscited by 16
  22. 2025Semantic Soft Bootstrapping: Long Context Reasoning in LLMs without Reinforcement Learningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 455.

455 papers.

Sort:

By area:

  1. 2026Reinforcement Learning via Self-Distillationcited by 332
  2. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 299
  3. 2026Self-Distillation Enables Continual Learningcited by 247
  4. 2026On-Policy Context Distillation for Language Modelscited by 186
  5. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 184
  6. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 163
  7. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 145
  8. 2026Self-Distilled RLVRcited by 131
  9. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?cited by 103
  10. 2026Privileged Information Distillation for Language Modelscited by 90
  11. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agentscited by 90
  12. 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillationcited by 77
38 more
  1. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 76
  2. 2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 68
  3. 2026CRISP: Compressed Reasoning via Iterative Self-Policy Distillationcited by 58
  4. 2026Self-Distilled Agentic Reinforcement Learningcited by 55
  5. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 53
  6. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 49
  7. 2026TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agentscited by 47
  8. 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillationcited by 45
  9. 2026SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weightingcited by 43
  10. 2026HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillationcited by 42
  11. 2026Embarrassingly Simple Self-Distillation Improves Code Generationcited by 40
  12. 2026Rethinking On-Policy Self-Distillation for Thinking Modelscited by 39
  13. 2026GATES: Self-Distillation under Privileged Context with Consensus Gatingcited by 35
  14. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationcited by 33
  15. 2026Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillationcited by 32
  16. 2026OPID: On-Policy Skill Distillation for Agentic Reinforcement Learningcited by 32
  17. 2026SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learningcited by 32
  18. 2026Purified OPSD: On-Policy Self-Distillation Without Losing How to Thinkcited by 31
  19. 2026Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Modelscited by 30
  20. 2026OPSDL: On-Policy Self-Distillation for Long-Context Language Modelscited by 30
  21. 2026When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoningcited by 30
  22. 2026PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competencecited by 28
  23. 2026Aligning Language Models from User Interactionscited by 28
  24. 2026AnyFlow: Any-Step Video Diffusion Model with On-Policy Flow Map Distillationcited by 26
  25. 2026On the Position Bias of On-Policy Distillationcited by 26
  26. 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Whycited by 25
  27. 2026Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoningcited by 23
  28. 2026D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Modelscited by 22
  29. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 21
  30. 2026GenEvolve: Self-Evolving Image Generation Agents via Tool-Orchestrated Visual Experience Distillationcited by 19
  31. 2026Pass the Baton: Trajectory-Relayed On-Policy Distillationcited by 18
  32. 2026Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillationcited by 18
  33. 2026MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debatecited by 17
  34. 2026ROSD: Reflective On-Policy Self-Distillation for Language Model Reasoning across Domainscited by 17
  35. 2026TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Trainingcited by 17
  36. 2026On-Policy Self-Distillation without Any Supervisioncited by 17
  37. 2026STEPS: Selective On-Policy Self-Distillation for Reasoningcited by 16
  38. 2026OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoningcited by 16

50 most cited of 455 All 455 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026A Brief Overview: On-Policy Self-Distillation In Large Language Models
  2. 2026Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models
  3. 2025Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning
  4. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
  5. 2026Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation
  6. 2026Distribution Corrected Offline Data Distillation for Large Language Models
  7. 2026SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning
  8. 2026KL for a KL: On-Policy Distillation with Control Variate Baseline