mlsanityData to 7 Oct 2026

SR-OPSD: Self-Referenced On-Policy Self-Distillation

arXiv 2608.09745 ↗First posted 10 Aug 2026cs.LG cs.AI stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

On-policy self-distillation (OPSD) converts feedback into dense token-level supervision on student-generated trajectories, complementing reinforcement learning with sparse outcome rewards. Its self-teacher, derived from the student's current or exponentially averaged parameters and conditioned on additional context, evolves alongside the student and its rollout context distribution. The benefit of modifying this moving target depends on how target--student probability mismatches translate into updates. We propose Self-Referenced On-Policy Self-Distillation (SR-OPSD), which constructs a normalized geometric target from the self-teacher and a frozen initial policy, then minimizes the forward R'enyi divergence from this target to the student. The interpolation coefficient controls the self-teacher's contribution, while the R'enyi order controls the power weighting of target-to-student probability ratios in the gradient. For fixed contexts and target components, we establish a conditional variational characterization and derive the exact token-logit gradient, revealing how anchoring and projection jointly shape the effective update target. Experiments across scientific reasoning, tool use, mathematical reasoning, and code generation demonstrate strong performance across multiple model families and scales. Ablations further show that reference anchoring can improve or degrade performance depending on the projection objective, supporting the joint design of target construction and projection geometry.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 39 references that are in the map. First posted on arXiv 10 Aug 2026.

Builds on

36 of 39 references.

References · 36

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  9. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  10. 2015Policy Distillationcited by 414
  11. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  12. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
24 more
  1. 2026Reinforcement Learning via Self-Distillationcited by 285
  2. 2026On-Policy Context Distillation for Language Modelscited by 161
  3. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  4. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  5. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 148
  6. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  7. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 112
  8. 2026Privileged Information Distillation for Language Modelscited by 79
  9. 2026CRISP: Compressed Reasoning via Iterative Self-Policy Distillationcited by 49
  10. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixescited by 37
  11. 2026GATES: Self-Distillation under Privileged Context with Consensus Gatingcited by 30
  12. 2026Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Modelscited by 24
  13. 2026KL for a KL: On-Policy Distillation with Control Variate Baselinecited by 24
  14. 2026When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoningcited by 24
  15. 2026Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoningcited by 20
  16. 2026STEPS: Selective On-Policy Self-Distillation for Reasoningcited by 15
  17. 2026Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularizationcited by 10
  18. 2026Adaptive Teacher Exposure for Self-Distillation in LLM Reasoningcited by 9
  19. 2026Reinforcement Learning from Rich Feedback with Distributional DAggercited by 9
  20. 2026When Should the Teacher Move? Temporal Coupling and Stability in Self On-Policy Distillationcited by 6
  21. 2026Training with Harnesses: On-Policy Harness Self-Distillation for Complex Reasoningcited by 5
  22. 2025Distillation of Large Language Models via Concrete Score Matchingcited by 4
  23. 2026Unifying Stable Optimization and Reference Regularization in RLHFcited by 4
  24. 2026f-Divergence Regularized RLHF: Two Tales of Sampling and Unified Analysescited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Outcome-Guided On-Policy Self-Distillation
  2. 2026A Predictive Law for On-Policy Self-Distillation From World Feedback
  3. 2026β-OPSD: Deriving with Policy Optimization, Training with Self-Distillation
  4. 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation
  5. 2026Latent On-Policy Self-Distillation
  6. 2026OPD+: Rethinking the Advantage Design for On-Policy Distillation
  7. 2026Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning
  8. 2026Self-Distilled Agentic Reinforcement Learning