# When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning

arXiv: https://arxiv.org/abs/2605.21606 · HTML: https://mlsanity.com/paper/2605.21606/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 20 May 2026.

## Placed in

- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 29 of 29)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models](https://mlsanity.com/paper/2601.18734/) (2026)
- [OpenThoughts: Data Recipes for Reasoning Models](https://mlsanity.com/paper/2506.04178/) (2025)
- [The False Promise of Imitating Proprietary LLMs](https://mlsanity.com/paper/2305.15717/) (2023)
- [Entropy-Aware On-Policy Distillation of Language Models](https://mlsanity.com/paper/2603.07079/) (2026)
- [DistiLLM: Towards Streamlined Distillation for Large Language Models](https://mlsanity.com/paper/2402.03898/) (2024)
- [A Survey of On-Policy Distillation for Large Language Models](https://mlsanity.com/paper/2604.00626/) (2026)
- [MiniLLM: On-Policy Distillation of Large Language Models](https://mlsanity.com/paper/2306.08543/) (2023)
- [Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs](https://mlsanity.com/paper/2605.00674/) (2026)
- [f-Divergence Minimization for Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/2307.15190/) (2023)
- [Scaling Reasoning Efficiently via Relaxed On-Policy Distillation](https://mlsanity.com/paper/2603.11137/) (2026)
- [Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2404.02657/) (2024)
- [GATES: Self-Distillation under Privileged Context with Consensus Gating](https://mlsanity.com/paper/2602.20574/) (2026)
- [ToDi: Token-wise Distillation via Fine-Grained Divergence Control](https://mlsanity.com/paper/2505.16297/) (2025)

## Built on by (most cited first, 30 of 30)

- [On-Policy Self-Distillation without Any Supervision](https://mlsanity.com/paper/2608.06296/) (2026)
- [Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation](https://mlsanity.com/paper/2608.09228/) (2026)
- [Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts](https://mlsanity.com/paper/2606.21994/) (2026)
- [Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation](https://mlsanity.com/paper/2608.04788/) (2026)
- [DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models](https://mlsanity.com/paper/2608.06243/) (2026)
- [PAST: Privileged Adaptation from Complete Student Trajectories for On-Policy Self-Distillation](https://mlsanity.com/paper/2608.08726/) (2026)
- [CROP: Task Relevance via Counterfactuals for Selective On-Policy Distillation](https://mlsanity.com/paper/2608.13387/) (2026)
- [Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation](https://mlsanity.com/paper/2609.02998/) (2026)
- [Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation](https://mlsanity.com/paper/2608.09263/) (2026)
- [Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation](https://mlsanity.com/paper/2608.08176/) (2026)
- [ReOrder-OPD:Reliability-Aware Prompt Ordering for On-Policy Distillation](https://mlsanity.com/paper/2608.10905/) (2026)
- [Trust Is Not Enough: Influence Calibration for On-Policy Self-Distillation in Agentic RL](https://mlsanity.com/paper/2608.14945/) (2026)
- [Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2609.23697/) (2026)
- [1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation](https://mlsanity.com/paper/2609.24432/) (2026)
- [When Clipping Reverses Correction: Failure Dynamics of Pointwise Forward-KL On-Policy Self-Distillation](https://mlsanity.com/paper/2609.38995/) (2026)
- [SR-OPSD: Self-Referenced On-Policy Self-Distillation](https://mlsanity.com/paper/2608.09745/) (2026)
- [Trajectory Learnability for Offline On-Policy Distillation with Imperfect Teachers](https://mlsanity.com/paper/2609.18321/) (2026)
- [Teaching a Moving Student: Rethinking the Curriculum of On-Policy Distillation](https://mlsanity.com/paper/2609.25048/) (2026)
- [CataOPD: Catalytic On-Policy Distillation for Large Language Model Reasoning](https://mlsanity.com/paper/2609.29518/) (2026)
- [Learning from Think-Mode Advantage via On-Policy Distillation](https://mlsanity.com/paper/2609.37044/) (2026)
- [Training LLM Judges from Language Feedback via Position-Selective Self-Distillation](https://mlsanity.com/paper/2609.38792/) (2026)
- [Diagnosing On-Policy Self-Distillation for Reasoning Language Models](https://mlsanity.com/paper/2609.39118/) (2026)
- [Better Supervision Is Nearby: Neighborhood On-Policy Self-Distillation](https://mlsanity.com/paper/2609.39687/) (2026)
- [Learning from Repaired Reasoning: Root-Cause-Guided On-Policy Distillation](https://mlsanity.com/paper/2610.03515/) (2026)
- [Outcome-Guided On-Policy Self-Distillation](https://mlsanity.com/paper/2610.05070/) (2026)
- [MetaOPD: Meta-Learned Token Weighting for On-Policy Distillation](https://mlsanity.com/paper/2610.11989/) (2026)
- [A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning](https://mlsanity.com/paper/2610.13744/) (2026)
- [PreSO: Privileged Representation Isolation for On-Policy Self-Distillation](https://mlsanity.com/paper/2610.14001/) (2026)
- [Learning from Self-Generated Responses: Reliability--Uncertainty Self-Distillation for Reasoning](https://mlsanity.com/paper/2610.19176/) (2026)
- [VASPO: Self-Distillation with Privileged Value Estimation](https://mlsanity.com/paper/2610.20516/) (2026)

## Abstract (arXiv)

On-policy self-distillation (OPSD) trains a student on its own rollouts using a privileged teacher, but its standard objective weights all generated tokens equally, implicitly treating the privileged teacher target as equally reliable at every student-visited prefix. Existing entropy-based OPD methods relax this uniformity by modulating token-level supervision with teacher entropy, but high teacher entropy in reasoning has an ambiguous reliability meaning: it can reflect either non-viable uncertainty or benign solution diversity. To identify this phenomenon, we introduce a branch-viability diagnostic. Specifically, we record next-token alternatives from the privileged-answer teacher prompt, force each alternative after the student prompt plus its on-policy spine prefix, and test whether the resulting student-template continuation recovers the correct answer. On Qwen3-4B, we find that an oriented within-sequence position score is the strongest tested predictor of teacher-token reliability, reaching an area-under-ROC-curve (AUROC) of 0.83; local uncertainty scores are at most 0.57. Motivated by this trajectory-level structure, we propose Position-Weighted On-Policy Self-Distillation (PW-OPSD), which applies an increasing position weight while keeping the same student rollout, privileged teacher pass, and clipped forward-KL target as OPSD. In our comprehensive evaluations with different random seeds, the diagnostic-derived PW-OPSD improves AIME 2024 and AIME 2025 Avg@12 by +1.0 and +1.1 points, and a generalization evaluation on two larger-scale models from different families, DeepSeek-R1-Distill-Llama-8B and Olmo-3-7B-Think, also demonstrates consistent aggregate Avg@12 improvements. These results show that teacher-token reliability in reasoning distillation is trajectory-structured and can be utilized without additional teacher computation.