Not Every Divergence Should Be Suppressed: Counterfactual Recoverability in On-Policy Distillation
On the map: Knowledge distillation → On-policy and self-distillation for LLMs
Abstract (arXiv)
On-policy distillation (OPD) supervises student-visited trajectories, yet divergence-based rules cannot determine whether an erroneous prefix remains correctable. We formulate this decision as counterfactual recoverability and replay each error state through budget-matched teacher-continuation and rollback branches. Based on their relative success, states are categorized as recoverable, irreversible-but-avoidable, or ambiguous, and these labels guide whether training retains, rolls back, or conventionally supervises the corresponding trajectory. On AIME branch diagnostics, the mean continuation-minus-rollback effect is 0.185 for recoverable states and -1.000 for irreversible-but-avoidable states, demonstrating opposite intervention preferences. A branch-derived recoverability proxy achieves an AUC of 1.000, substantially outperforming divergence alone at 0.392. Across frozen evaluations, recoverability-aware control achieves the strongest recorded performance, reaching 0.578 success on held-out AIME2025 compared with 0.517 for the best baseline. It also improves AIME2024-2025 average@32 from 0.2656 to 0.3125 and GPQA-Diamond average@32 from 0.2702 to 0.3070. Component ablations further show that retaining teacher-correctable prefixes provides the largest individual contribution. These findings establish recoverability as an outcome-grounded decision variable for selective supervision in OPD.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 12 of its 18 references that are in the map. First posted on arXiv 5 Aug 2026.
Builds on
12 of 18 references.
Closest ancestors · 2
Same direction, newest first
- 2026On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents14 Jun
- 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe14 Apr
Other references · 10
- 2015Distilling the Knowledge in a Neural Networkcited by 10,180
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,502
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,367
- 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 786
- 2016Sequence-Level Knowledge Distillationcited by 770
- 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 764
- 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 719
- 2015Policy Distillationcited by 423
- 2020Autoregressive Knowledge Distillation through Imitation Learningcited by 70
- 2023Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomescited by 13
Built on by
Not cited yet.