Learning from Think-Mode Advantage via On-Policy Distillation
On the map: LLM reasoning → Prompting and chain-of-thought reasoning
Abstract (arXiv)
Explicit intermediate reasoning gives large language models (LLMs) a stronger problem-solving mode. We study learning from this think-mode advantage via on-policy distillation (OPD). OPD preserves student-generated trajectories and provides dense token-level teacher targets at student-visited prefixes. Privileged reasoning is used during distillation rather than student inference. Uniform ThinkOPD, a natural think-enabled OPD baseline, conditions a fixed teacher on one shared think trace and uniformly distills every sibling student response. Although its prefixes are on-policy, the trace need not follow a route compatible with every complete response: the same privileged trace can induce different teacher-student discrepancies even when responses reach the same outcome. We summarize this interaction with trace-response divergence (TRD) and introduce ThinkOPD, which routes supervision at the response level by combining group-relative reward gain with a TRD-based compatibility proxy. Final response weights are normalized within each rollout group. Across mathematical reasoning and code generation, ThinkOPD outperforms Uniform ThinkOPD in both same-model settings and both cross-model teacher-student pairs, and it exceeds representative rationale and self-distillation baselines in a controlled comparison. Controlled interventions show that outcome benefit and the TRD-based proxy provide complementary routing signals in this setting. Think-enabled OPD provides a controlled setting for studying how teacher advantage becomes transferable along student responses.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 15 of its 33 references that are in the map. First posted on arXiv 29 Sep 2026.
Builds on
15 of 33 references.
Closest ancestors · 4
Same direction, newest first
- 2026Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance29 May
- 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe14 Apr
- 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models26 Jan
- 2024Distilling System 2 into System 18 Jul
Other references · 11
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2025Qwen3 Technical Reportcited by 8,195
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
- 2026Reinforcement Learning via Self-Distillationcited by 285
- 2026Self-Distillation Enables Continual Learningcited by 212
- 2025Skywork Open Reasoner 1 Technical Reportcited by 161
- 2026On-Policy Context Distillation for Language Modelscited by 161
- 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
- 2026Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillationcited by 42
- 2026When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoningcited by 24
- 2026On-Policy Distillation with Best-of-N Teacher Rollout Selectioncited by 7
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress
- 2026Rethinking On-Policy Self-Distillation for Thinking Models
- 2026What Does Privileged Information Add to On-Policy Self-Distillation?
- 2026Diagnosing On-Policy Self-Distillation for Reasoning Language Models
- 2026Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models
- 2026Reward-Aligned Reweighting for On-Policy Distillation
- 2026Rethinking Privileged Information in On-Policy Self-Distillation
- 2026Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation