mlsanityData to 7 Oct 2026

Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models

arXiv 2609.37132 ↗First posted 29 Sep 2026cs.AIcited by 0

On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training

Abstract (arXiv)

On-policy self-distillation (OPSD) improves large language models by letting a self-teacher with privileged information provide dense token-level supervision on the model's own trajectories. Yet existing methods typically construct the self-teacher from the current, initial, or slowly averaged policy state, leaving the quality of supervision constrained by the teacher's ability to exploit privileged information. We ask whether the model's own optimization progress can instead be recycled into a stronger self-teacher. In this paper, we introduce Bootstrapped On-Policy Self-Distillation (B-OPSD), which temporarily trains the policy ahead to obtain a future teacher, restores the student to the original policy state, and then uses the future teacher to supervise the restarted student. The future teacher improves supervision in two complementary ways, it can generate more reliable privileged trajectories and, conditioned on them, provide more informative token-level targets along the restarted student's on-policy trajectories. Experiments on mathematical reasoning with Qwen3-4B and Qwen3-8B show consistent improvements over standard OPSD in both settings, including gains from 27.50 to 41.30 and from 48.80 to 64.44 in the rollout-privileged setting. Our findings point to a broader principle for self-improving models that future learning progress can be distilled backward, preserving acquired knowledge while bootstrapping beyond the optimization state that produced it.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 23 of its 40 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

23 of 40 references.

References · 23

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2026Self-Distilled RLVRcited by 109
  3. 2026Privileged Information Distillation for Language Modelscited by 79
  4. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 65
  5. 2026GATES: Self-Distillation under Privileged Context with Consensus Gatingcited by 30
  6. 2026Purified OPSD: On-Policy Self-Distillation Without Losing How to Thinkcited by 23
  7. 2026OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoningcited by 14
  8. 2026On-Policy Self-Distillation without Any Supervisioncited by 11
  9. 2026Rethinking Reward Supervision: Rubric-Conditioned Self-Distillationcited by 9
  10. 2026UniSD: Towards a Unified Self-Distillation Framework for Large Language Modelscited by 6
  11. 2026Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillationcited by 6
  12. 2026Reducing the Safety Tax in LLM Safety Alignment with On-Policy Self-Distillationcited by 6
11 more
  1. 2026PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignmentcited by 6
  2. 2026Latent On-Policy Self-Distillationcited by 6
  3. 2026CriPO: Enhancing Rubric-based RL via Self-Distillationcited by 4
  4. 2026Learning from the Near Future: Temporal Self-Distillation for RLVRcited by 3
  5. 2026Consensus as Privileged Context for Label-Free Self-Distillationcited by 3
  6. 2026RISE: Recursive Improvement via Self-Extrapolating Policy Distillationcited by 3
  7. 2026Why Does Feedback-Augmented Self-Distillation Fail to Improve Retrieval-Interleaved Search Agents?cited by 2
  8. 2026I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimizationcited by 2
  9. 2025Future Policy Approximation for Offline Reinforcement Learning in LLM Reasoningcited by 1
  10. 2026Distilling LLM Feedback for Lean Theorem Provingcited by 1
  11. 2026Adaptive Supervised Anchoring for On-Policy Self-Distillationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
  2. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
  3. 2026A Brief Overview: On-Policy Self-Distillation In Large Language Models
  4. 2026ROSD: Reflective On-Policy Self-Distillation for Language Model Reasoning across Domains
  5. 2026dOPSD: On-Policy Self-Distillation for Diffusion Language Models
  6. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
  7. 2026β-OPSD: Deriving with Policy Optimization, Training with Self-Distillation
  8. 2026Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning