mlsanityData to 14 Oct 2026

What Should a Self-Teacher See? Privileged Context Design for On-Policy Self-Distillation

arXiv 2609.25623 ↗First posted 22 Sep 2026cs.LG cs.AIcited by 2

On the map: Knowledge distillation → On-policy and self-distillation for LLMs

Abstract (arXiv)

More privileged information does not always make a better teacher. We study this tension in on-policy self-distillation (OPSD), where a self-teacher scores the student's own rollouts under privileged context, conventionally a complete reference solution that bundles the final answer with one particular reasoning path. Holding the student view and training fixed within each scale, we compare that default against three abstractions compiled offline, a named strategy, a method-independent framing, and a problem category, and against an answer-only control that keeps the destination but removes the path. In the primary runs on competition mathematics, the best intermediate contexts improve the in-domain peak mean over the full solution by 1.4 points at 4B and 1.6 at 8B, while storing an order of magnitude fewer hint tokens. Comparisons across three seeds also show positive mean gains for the framing and category contexts at both scales. Answer-only conditioning remains competitive in the primary runs, within 0.2 points of the full solution at these scales. The preferred context varies with student scale and task. Initial teacher-student KL does not order downstream performance. What a self-teacher should see is therefore not everything it could, but the level of abstraction its student can still act on.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 39 of its 44 references that are in the map. First posted on arXiv 22 Sep 2026.

Builds on

39 of 44 references.

Closest ancestors · 19

Same direction, newest first

  1. 2026Rethinking Privileged Information in On-Policy Self-Distillation18 Aug
  2. 2026Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation10 Aug
  3. 2026On-Policy Self-Distillation without Any Supervision6 Aug
  4. 2026Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation5 Aug
  5. 2026Is More Privileged Information Better? From Solution Traces to Problem-Solving Structure in Self-Distilled Reasoning3 Aug
  6. 2026Rethinking On-Policy Self-Distillation for Thinking Models6 Jul
  7. 2026Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning12 May
  8. 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why11 May
  9. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe5 May
  10. 2026The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillation18 Apr
  11. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe14 Apr
  12. 2026Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models9 Apr
7 more
  1. 2026Entropy-Aware On-Policy Distillation of Language Models7 Mar
  2. 2026On-Policy Context Distillation for Language Models12 Feb
  3. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation12 Feb
  4. 2026Privileged Information Distillation for Language Models4 Feb
  5. 2026Reinforcement Learning via Self-Distillation28 Jan
  6. 2026Self-Distillation Enables Continual Learning27 Jan
  7. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models26 Jan

Other references · 20

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023Let's Verify Step by Stepcited by 3,789
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  6. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  7. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  8. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  9. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  10. 2015Policy Distillationcited by 440
  11. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 304
  12. 2015Unifying distillation and privileged informationcited by 233
8 more
  1. 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoningcited by 201
  2. 2022Learning by Distilling Contextcited by 138
  3. 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillationcited by 77
  4. 2026CRISP: Compressed Reasoning via Iterative Self-Policy Distillationcited by 58
  5. 2025Linguistic Generalizability of Test-Time Scaling in Mathematical Reasoningcited by 37
  6. 2025AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Modelscited by 31
  7. 2026Learn Hard Problems During RL with Reference Guided Fine-tuningcited by 12
  8. 2026ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillationcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Is Better Teacher Supervision Enough? Unlocking Student-side Learning in Multimodal On-Policy Distillationcited by 0
  2. 2026Better Supervision Is Nearby: Neighborhood On-Policy Self-Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026What Does Privileged Information Add to On-Policy Self-Distillation?
  2. 2026Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation
  3. 2026A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning
  4. 2026Learning from a Thoughtful Teacher: Adaptive On-Policy Self-Distillation for Mathematical Reasoning
  5. 2026PAST: Privileged Adaptation from Complete Student Trajectories for On-Policy Self-Distillation
  6. 2026Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning
  7. 2026A Good Self-Teacher Meets the Student Where They Are: Joint On-Policy Learning and Teaching
  8. 2026Recursive Self-Improvement via On-Policy Distillation for Reasoning