# Is More Privileged Information Better? From Solution Traces to Problem-Solving Structure in Self-Distilled Reasoning

arXiv: https://arxiv.org/abs/2608.01589 · HTML: https://mlsanity.com/paper/2608.01589/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 3 Aug 2026.

## Placed in

- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 22 of 22)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models](https://mlsanity.com/paper/2601.18734/) (2026)
- [OpenThoughts: Data Recipes for Reasoning Models](https://mlsanity.com/paper/2506.04178/) (2025)
- [Unifying distillation and privileged information](https://mlsanity.com/paper/1511.03643/) (2015)
- [Self-Distilled RLVR](https://mlsanity.com/paper/2604.03128/) (2026)
- [Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?](https://mlsanity.com/paper/2603.24472/) (2026)
- [Privileged Information Distillation for Language Models](https://mlsanity.com/paper/2602.04942/) (2026)
- [Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation](https://mlsanity.com/paper/2605.26844/) (2026)
- [The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes](https://mlsanity.com/paper/2605.11182/) (2026)
- [Deconstructing Long Chain-of-Thought: A Structured Reasoning Optimization Framework for Long CoT Distillation](https://mlsanity.com/paper/2503.16385/) (2025)
- [SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correction](https://mlsanity.com/paper/2410.09008/) (2024)
- [Rethinking On-Policy Self-Distillation for Thinking Models](https://mlsanity.com/paper/2607.05184/) (2026)
- [Purified OPSD: On-Policy Self-Distillation Without Losing How to Think](https://mlsanity.com/paper/2607.02234/) (2026)
- [Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning](https://mlsanity.com/paper/2605.11458/) (2026)
- [Learning to Hint for Reinforcement Learning](https://mlsanity.com/paper/2604.00698/) (2026)
- [Training with Harnesses: On-Policy Harness Self-Distillation for Complex Reasoning](https://mlsanity.com/paper/2605.08741/) (2026)
- [PAINT: Partial-Solution Adaptive Interpolated Training for Self-Distilled Reasoners](https://mlsanity.com/paper/2604.26573/) (2026)
- [AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals](https://mlsanity.com/paper/2605.20643/) (2026)
- [Reasoning Scaffolding: Distilling the Flow of Thought from LLMs](https://mlsanity.com/paper/2509.23619/) (2025)
- [Selective Off-Policy Reference Tuning with Plan Guidance](https://mlsanity.com/paper/2605.11505/) (2026)

## Built on by (most cited first, 9 of 9)

- [PAST: Privileged Adaptation from Complete Student Trajectories for On-Policy Self-Distillation](https://mlsanity.com/paper/2608.08726/) (2026)
- [What Does Privileged Information Add to On-Policy Self-Distillation?](https://mlsanity.com/paper/2609.20612/) (2026)
- [What Should a Self-Teacher See? Privileged Context Design for On-Policy Self-Distillation](https://mlsanity.com/paper/2609.25623/) (2026)
- [SCOPE-OPSD: Fisher-Conditioned Privileged Subspaces for On-Policy Self-Distillation](https://mlsanity.com/paper/2609.12579/) (2026)
- [When Clipping Reverses Correction: Failure Dynamics of Pointwise Forward-KL On-Policy Self-Distillation](https://mlsanity.com/paper/2609.38995/) (2026)
- [PG-SFT: Balancing Capability Acquisition and Retention in Offline Agent Fine-Tuning](https://mlsanity.com/paper/2610.00949/) (2026)
- [SpatialOPSD: Self-Distilling Spatial Intelligence from Verified Coding Agent Traces](https://mlsanity.com/paper/2610.11366/) (2026)
- [A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning](https://mlsanity.com/paper/2610.13744/) (2026)
- [Self-Distillation with Verbal Regularization](https://mlsanity.com/paper/2610.16354/) (2026)

## Abstract (arXiv)

On-policy self-distillation (OPSD) improves reasoning by using a privileged view of a model conditioned on reference solutions to supervise a student view that observes only the question. However, the teacher-provided token-level targets may depend on reference-specific information unavailable at inference time. We propose Problem-Space-Guided OPSD (PS-OPSD), which replaces the complete solution with trajectory-grounded guidance describing the initial state, goal conditions, constraints, and a selected state-transition path. The student rollout and OPSD objective remain unchanged. Across three mathematical reasoning benchmarks and model scales ranging from 1.7B to 8B, PS-OPSD achieves the highest aggregate question-only accuracy among the compared methods. Controlled experiments further indicate that guidance relevance and path coherence contribute to these gains, highlighting the representation of privileged information as an important design choice in OPSD.