Teaching a Moving Student: Rethinking the Curriculum of On-Policy Distillation

arXiv 2609.25048 ↗First posted 6 Sep 2026cs.CL cs.AIcited by 0

On the map: Knowledge distillation → On-policy and self-distillation for LLMs

Abstract (arXiv)

In on-policy distillation (OPD), the student determines which states receive teacher supervision. As its policy evolves, earlier response prefixes become less likely even though teacher-student disagreement on them persists. Under matched trajectory and optimization budgets, neither more queries nor more frequent rollout resampling is uniformly beneficial. Current-policy rollouts outperform initial-policy rollouts at shorter response budgets, but this ranking reverses at longer budgets. Replaying initial-policy rollouts after current-policy training improves accuracy, whereas replaying fixed recent rollouts does not reproduce the gain. We propose R-OPD, a gradient-triggered curriculum that adaptively selects when to revisit initial-student trajectories. When changes in mean gradients fall within minibatch-level variation for two consecutive comparisons, training switches from the next iteration onward to initial-policy replay. Across eight mathematics benchmarks and three training-data orders, R-OPD improves average accuracy over continued current-policy sampling by 2.25/4.44 percentage points at 16K/32K for a 0.6B student and 2.81/6.15 points for a 1.7B student. With a 30B-A3B teacher, it improves 8B accuracy by 4.05 points at 32K. At 32K, R-OPD exceeds a fixed schedule of 40 current-policy updates followed by 20 replay updates by 1.39/1.66/1.85 points for 0.6B/1.7B/8B. At the same generation cap, R-OPD also produces longer responses, suggesting that well-timed revisits help students use more of their reasoning capacity.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 23 of its 24 references that are in the map. First posted on arXiv 6 Sep 2026.

Builds on

23 of 24 references.

Closest ancestors · 12

Same direction, newest first

Other references · 11

2025Qwen3 Technical Reportcited by 8,820
2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
2025s1: Simple test-time scalingcited by 1,341
2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
2025LIMO: Less is More for Reasoningcited by 498
2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 295
2025LIMR: Less is More for RL Scalingcited by 139
2025JustRL: Scaling a 1.5B LLM with a Simple RL Recipecited by 64
2026Efficient RL Training for LLMs with Experience Replaycited by 9
2025Learning More with Less: A Dynamic Dual-Level Down-Sampling Framework for Efficient Policy Optimizationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way