Recovering Reliable Supervision via Critique-Guided On-Policy Distillation
On the map: Knowledge distillation → On-policy and self-distillation for LLMs
Abstract (arXiv)
On-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student writes. We call this property the recoverability of the teacher and show it fails on incorrect rollouts: the teacher follows student errors, although it recognizes them. We propose CriticOPD, using error recognition to restore recoverability. The teacher critiques an incorrect rollout, allowing the student to rewrite its solution following the error. The resulting output, including the critique as privileged information, is scored by the teacher while the student uses the revision without the critique for training. On two Qwen3 students and six mathematical reasoning benchmarks, CriticOPD improves OPD, with the largest gains on competition problems.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 22 of its 28 references that are in the map. First posted on arXiv 14 Oct 2026.
Builds on
22 of 28 references.
Closest ancestors · 12
Same direction, newest first
- 2026When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation17 Sep
- 2026Pass the Baton: Trajectory-Relayed On-Policy Distillation28 Jul
- 2026SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision26 Jun
- 2026Escaping the KL Agreement Trap in On-Policy Distillation8 Jun
- 2026Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation29 May
- 2026Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation26 May
- 2026Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation13 May
- 2026OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning12 May
- 2026Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning8 May
- 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe14 Apr
- 2026SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting12 Apr
- 2026Fast and Effective On-policy Distillation from Reasoning Prefixes16 Feb
Other references · 10
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
- 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
- 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
- 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 297
- 2015Unifying distillation and privileged informationcited by 233
- 2023Training Language Models with Language Feedback at Scalecited by 128
- 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 90
- 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 81
- 2026Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillationcited by 8
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Correcting or Continuing? What On-Policy Distillation Learns from Student Errors
- 2026Reward-Gated On-Policy Distillation
- 2025Critique-Guided Distillation for Robust Reasoning via Refinement
- 2026Learning to Revise Reasoning with Segment-wise On-Policy Distillation
- 2026Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives
- 2026Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation
- 2026Outcome-Confounded Local Supervision in On-Policy Distillation
- 2026A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning