# Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation

arXiv: https://arxiv.org/abs/2608.09228 · HTML: https://mlsanity.com/paper/2608.09228/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 25 of its 27 references that are in the map.
- First posted on arXiv 10 Aug 2026.

## Placed in

- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 25 of 25)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [Reinforced Self-Training (ReST) for Language Modeling](https://mlsanity.com/paper/2308.08998/) (2023)
- [Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models](https://mlsanity.com/paper/2601.18734/) (2026)
- [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026)
- [MathArena: Evaluating LLMs on Uncontaminated Math Competitions](https://mlsanity.com/paper/2505.23281/) (2025)
- [Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models](https://mlsanity.com/paper/2410.07985/) (2024)
- [Self-Distillation Enables Continual Learning](https://mlsanity.com/paper/2601.19897/) (2026)
- [Unifying distillation and privileged information](https://mlsanity.com/paper/1511.03643/) (2015)
- [On-Policy Context Distillation for Language Models](https://mlsanity.com/paper/2602.12275/) (2026)
- [DistiLLM: Towards Streamlined Distillation for Large Language Models](https://mlsanity.com/paper/2402.03898/) (2024)
- [Learning by Distilling Context](https://mlsanity.com/paper/2209.15189/) (2022)
- [Privileged Information Distillation for Language Models](https://mlsanity.com/paper/2602.04942/) (2026)
- [Rethinking On-Policy Self-Distillation for Thinking Models](https://mlsanity.com/paper/2607.05184/) (2026)
- [In-Context Editing: Learning Knowledge from Self-Induced Distributions](https://mlsanity.com/paper/2406.11194/) (2024)
- [Purified OPSD: On-Policy Self-Distillation Without Losing How to Think](https://mlsanity.com/paper/2607.02234/) (2026)
- [When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning](https://mlsanity.com/paper/2605.21606/) (2026)
- [ROSD: Reflective On-Policy Self-Distillation for Language Model Reasoning across Domains](https://mlsanity.com/paper/2605.28014/) (2026)
- [Self-Supervised On-Policy Distillation for Reasoning Language Models](https://mlsanity.com/paper/2605.17497/) (2026)
- [Semantic Soft Bootstrapping: Long Context Reasoning in LLMs without Reinforcement Learning](https://mlsanity.com/paper/2512.05105/) (2025)

## Built on by (most cited first, 12 of 12)

- [Rethinking Privileged Information in On-Policy Self-Distillation](https://mlsanity.com/paper/2608.18271/) (2026)
- [What Does Privileged Information Add to On-Policy Self-Distillation?](https://mlsanity.com/paper/2609.20612/) (2026)
- [What Should a Self-Teacher See? Privileged Context Design for On-Policy Self-Distillation](https://mlsanity.com/paper/2609.25623/) (2026)
- [SCOPE-OPSD: Fisher-Conditioned Privileged Subspaces for On-Policy Self-Distillation](https://mlsanity.com/paper/2609.12579/) (2026)
- [Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning](https://mlsanity.com/paper/2609.37915/) (2026)
- [When Clipping Reverses Correction: Failure Dynamics of Pointwise Forward-KL On-Policy Self-Distillation](https://mlsanity.com/paper/2609.38995/) (2026)
- [Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresight](https://mlsanity.com/paper/2610.08077/) (2026)
- [Learning from a Thoughtful Teacher: Adaptive On-Policy Self-Distillation for Mathematical Reasoning](https://mlsanity.com/paper/2609.32667/) (2026)
- [Does Learning to Predict the World Help Agents Act? Auditing World-Model Post-Training](https://mlsanity.com/paper/2609.33335/) (2026)
- [Privileged Context as Drift in On-Policy Self-Distillation](https://mlsanity.com/paper/2610.07842/) (2026)
- [Does self-distillation improve reasoning?](https://mlsanity.com/paper/2610.18613/) (2026)
- [Learning from Self-Generated Responses: Reliability--Uncertainty Self-Distillation for Reasoning](https://mlsanity.com/paper/2610.19176/) (2026)

## Abstract (arXiv)

On-Policy Self-Distillation (OPSD) is commonly interpreted as the transfer of privileged information: a teacher observes the verified solution to the target problem and supervises the student's trajectory. However, this interpretation conflates two effects. The reference solution not only reveals the answer to the current instance but also changes the context under which the teacher provides token-level supervision. We investigate the role of target-specific privilege with OP2SD (On-Policy Self-Distillation from Other Problems), which replaces the paired reference with a problem and solution from a different example, while preserving the student rollout, teacher, and distillation objective. Across three models and three mathematics benchmarks, OP2SD improves over the base model, remains competitive with OPSD. The success of OP2SD implies that OPSD gains do not necessarily come from access to the reference solution, and that the teacher's context-induced behavior is an important factor.