mlsanityData to 7 Oct 2026

TISD: On-Policy Self-Distillation with Trajectory Intervention

arXiv 2609.30878 ↗First posted 25 Sep 2026cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

On-policy self-distillation (OPSD) provides dense teacher targets, but evaluates them only along student-sampled rollouts. When the privileged teacher favors an alternative action at a visited prefix, OPSD can provide a target for the branch decision but cannot supervise the successor contexts induced by that action unless the student samples it. This creates a training-time data-collection bottleneck and suggests a different role for teacher-student disagreement: proposing a trajectory branch rather than identifying a sufficient local repair. Our diagnostic framework using controlled token interventions reveals that a teacher-preferred token at peak disagreement can improve student continuation success, while its local corrective value is limited. Motivated by this finding, we introduce a simple branch-regenerate-distill algorithm, Trajectory-Intervention Self-Distillation (TISD). TISD forces a teacher-selected branch action, returns suffix generation to the student, and distills the full trajectory under the privileged-context-conditioned teacher. Across the coding models, TISD improves average Avg@4 over SDPO by 1.2 percentage points. Across the science domains, it improves average Avg@128 by 0.8 points under an equal-step budget and by 0.3 points under an equal-time budget. These results support teacher-guided branching as a way to expose useful successor contexts for self-distillation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 28 references that are in the map. First posted on arXiv 25 Sep 2026.

Builds on

24 of 28 references.

References · 24

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2023Let's Verify Step by Stepcited by 3,598
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  5. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  6. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  7. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  8. 2025Process Reinforcement through Implicit Rewardscited by 478
  9. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  10. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  11. 2026Reinforcement Learning via Self-Distillationcited by 285
  12. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
12 more
  1. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 146
  2. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 110
  3. 2026TIP: Token Importance in On-Policy Distillationcited by 90
  4. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?cited by 87
  5. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  6. 2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMscited by 43
  7. 2025GRPO-λ: Credit Assignment improves LLM Reasoningcited by 21
  8. 2025SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMscited by 17
  9. 2026Trajectory-Refined Distillationcited by 14
  10. 2026Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidancecited by 12
  11. 2026Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergencecited by 9
  12. 2026Rethinking Selective Knowledge Distillationcited by 6

Built on by

Not cited yet.