mlsanityData to 13 Oct 2026

Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation

arXiv 2605.30833 ↗First posted 29 May 2026cs.CL cs.AIcited by 11

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

also in 1 more
Abstract (arXiv)

On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher. However, we identify a critical bottleneck, Supervision Fidelity Decay (SFD): as student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative. Consequently, the teacher-dependent corrective signal in reverse-KL distillation weakens, causing student drift to compound across long reasoning chains. To mitigate SFD, we introduce textbfLookahead Group Reward (ours). Building on the insight that next-step teacher confidence reflects the discriminative strength of future reverse-KL supervision, ours evaluates the student's top-K candidate tokens by the teacher confidence they induce at the subsequent step and assigns a group-normalized reward. To maintain computational efficiency, we further design an entropy-triggered tree-attention mechanism. Across six math and code benchmarks, ours improves mean@8 by 2.57 points over OPD for a 7B student, with gains increasing in longer-generation and reaching +4.92 points on AIME-26 at 39k tokens.

Key facts. Cited by 11 papers in the map (data through 13 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 21 of its 29 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

21 of 29 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe14 Apr
  2. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillation11 Mar
  3. 2026Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillation25 Feb

Other references · 18

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,247
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,951
  5. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 961
  6. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 873
  7. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 829
  8. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 786
  9. 2016Sequence-Level Knowledge Distillationcited by 770
  10. 2015Policy Distillationcited by 423
  11. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 167
  12. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 135
6 more
  1. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 134
  2. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 125
  3. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 76
  4. 2020Autoregressive Knowledge Distillation through Imitation Learningcited by 70
  5. 2019Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation?cited by 34
  6. 2025Distillation of Large Language Models via Concrete Score Matchingcited by 4

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 11.

11 papers.

  1. 2026Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillationcited by 9
  2. 2026DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Modelscited by 7
  3. 2026Mismatch Matters: On-Policy Distillation Beyond Token Agreementcited by 7
  4. 2026When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillationcited by 2
  5. 2026From Parameters to Behaviors: A Survey of Model Fusion for Large Language Modelscited by 1
  6. 2026Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillationcited by 0
  7. 2026What Shared Prefixes Hide: Trajectory Dropout for On-Policy Distillationcited by 0
  8. 2026UOPD: Uncertainty-Aware Intervention for On-Policy Distillation of Multi-Turn Agentscited by 0
  9. 2026Understanding On-Policy Distillation: A Mechanistic Interpretability Perspective via Sparse Crosscoderscited by 0
  10. 2026On the Off-Policy Teacher in On-Policy Distillationcited by 0
  11. 2026Better Supervision Is Nearby: Neighborhood On-Policy Self-Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward-Gated On-Policy Distillation
  2. 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why
  3. 2026Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning
  4. 2026E2-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation
  5. 2026Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress
  6. 2026Reward-Aligned Reweighting for On-Policy Distillation
  7. 2026Learning from Teacher Continuations at Student States
  8. 2026Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation