mlsanityData to 7 Oct 2026

Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation

arXiv 2608.09228 ↗First posted 10 Aug 2026cs.LG cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

On-Policy Self-Distillation (OPSD) is commonly interpreted as the transfer of privileged information: a teacher observes the verified solution to the target problem and supervises the student's trajectory. However, this interpretation conflates two effects. The reference solution not only reveals the answer to the current instance but also changes the context under which the teacher provides token-level supervision. We investigate the role of target-specific privilege with OP2SD (On-Policy Self-Distillation from Other Problems), which replaces the paired reference with a problem and solution from a different example, while preserving the student rollout, teacher, and distillation objective. Across three models and three mathematics benchmarks, OP2SD improves over the base model, remains competitive with OPSD. The success of OP2SD implies that OPSD gains do not necessarily come from access to the reference solution, and that the teacher's context-induced behavior is an important factor.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 25 of its 27 references that are in the map. First posted on arXiv 10 Aug 2026.

Builds on

25 of 27 references.

References · 25

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  4. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  5. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  6. 2016Sequence-Level Knowledge Distillationcited by 761
  7. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  8. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  9. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  10. 2026Reinforcement Learning via Self-Distillationcited by 285
  11. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 277
  12. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 275
13 more
  1. 2015Unifying distillation and privileged informationcited by 218
  2. 2026Self-Distillation Enables Continual Learningcited by 212
  3. 2026On-Policy Context Distillation for Language Modelscited by 161
  4. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 145
  5. 2022Learning by Distilling Contextcited by 128
  6. 2026Privileged Information Distillation for Language Modelscited by 79
  7. 2024In-Context Editing: Learning Knowledge from Self-Induced Distributionscited by 33
  8. 2026Rethinking On-Policy Self-Distillation for Thinking Modelscited by 28
  9. 2026When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoningcited by 24
  10. 2026Purified OPSD: On-Policy Self-Distillation Without Losing How to Thinkcited by 23
  11. 2026ROSD: Reflective On-Policy Self-Distillation for Language Model Reasoning across Domainscited by 17
  12. 2026Self-Supervised On-Policy Distillation for Reasoning Language Modelscited by 10
  13. 2025Semantic Soft Bootstrapping: Long Context Reasoning in LLMs without Reinforcement Learningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Rethinking Privileged Information in On-Policy Self-Distillationcited by 6
  2. 2026What Should a Self-Teacher See? Privileged Context Design for On-Policy Self-Distillationcited by 2
  3. 2026SCOPE-OPSD: Fisher-Conditioned Privileged Subspaces for On-Policy Self-Distillationcited by 1
  4. 2026What Does Privileged Information Add to On-Policy Self-Distillation?cited by 1
  5. 2026Learning from a Thoughtful Teacher: Adaptive On-Policy Self-Distillation for Mathematical Reasoningcited by 0
  6. 2026Does Learning to Predict the World Help Agents Act? Auditing World-Model Post-Trainingcited by 0
  7. 2026Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoningcited by 0
  8. 2026When Clipping Reverses Correction: Failure Dynamics of Pointwise Forward-KL On-Policy Self-Distillationcited by 0