mlsanityData to 14 Oct 2026

Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning

arXiv 2609.37915 ↗First posted 29 Sep 2026cs.LG cs.CLcited by 1

On the map: LLM reasoning → Prompting and chain-of-thought reasoning also fits: Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

On-policy self-distillation (OPSD) trains a student to match a privileged teacher distribution along its own sampled trajectory. Standard OPSD applies this supervision to unverified student rollouts while conditioning the teacher on privileged context, typically a reference solution. We separate these roles in a factorial analysis and find that scaffold correctness has a stronger effect on downstream accuracy than context correctness. Unverified scaffolds create an imitation gap because the teacher can use information unavailable to the student. This gap shrinks with model scale, yet OPSD continues to supervise mostly unverified trajectories. In contrast, verified scaffolds remain effective even when the teacher is conditioned on the student's own unsuccessful rollout. Based on this finding, we introduce OASIS, which retains the OPSD objective but supervises mostly verified by label on-policy trajectories and replaces written solutions with unverified model-generated attempts as the teacher context. OASIS therefore requires only final-answer labels. Across Qwen3-1.7B, 4B, and 8B on AIME 2024, AIME 2025, and HMMT 2025, OASIS improves over the base model by 3.2--3.8 points on average, while OPSD's gain falls from 3.05 points at 1.7B to 0.14 at 8B. At 8B, OASIS improves over OPSD by 3.05 points, showing that verified on-policy scaffolds preserve the effectiveness of self-distillation as models scale.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 26 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

24 of 26 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models6 Aug
  2. 2026Purified OPSD: On-Policy Self-Distillation Without Losing How to Think2 Jul
  3. 2026Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning13 May
  4. 2026CRISP: Compressed Reasoning via Iterative Self-Policy Distillation5 Mar
  5. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models26 Jan
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar

Other references · 18

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2023Let's Verify Step by Stepcited by 3,789
  6. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  7. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  8. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  9. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  10. 2016Sequence-Level Knowledge Distillationcited by 801
  11. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  12. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 304
6 more
  1. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 252
  2. 2020Bridging the Imitation Gap by Adaptive Insubordinationcited by 38
  3. 2022Sequence Model Imitation Learning with Unobserved Contextscited by 34
  4. 2026Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillationcited by 12
  5. 2026AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signalscited by 7
  6. 2025A Theoretical Justification for Asymmetric Actor-Critic Algorithmscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Learning from Self-Generated Responses: Reliability--Uncertainty Self-Distillation for Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
  2. 2026Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation
  3. 2026Towards Understanding On-Policy Distillation through the Lens of Test-Time Scaling
  4. 2026LoOPSD: On-Policy Self-Distillation with Looped Teachers
  5. 2026OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning
  6. 2026Diagnosing On-Policy Self-Distillation for Reasoning Language Models
  7. 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why
  8. 2026Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations