CLOOPD: Closing the Learner Loop in On-Policy Distillation
On the map: Knowledge distillation → On-policy and self-distillation for LLMs
Abstract (arXiv)
On-policy distillation (OPD) pays twice for each fresh batch: the student generates trajectories and a stronger teacher scores them. Existing methods improve which trajectories are scored and how the teacher signal is constructed, but usually consume it with one actor update. We introduce CLOOPD, a closed-loop framework separating teacher-signal acquisition from student-side realization. CLOOPD selects an adaptive α waypoint inside a KL envelope, freezes the scored batch and its advantages, re-forwards the student after each actor pass, measures realization, and allocates actor work under a separate token budget. The framework includes deterministic two- and three-pass policies, token-priced CLOOPD-TPMR, and a budget-matched control. Across six 300-step runs on an 8-H20 node, every CLOOPD policy improves the one-pass TOP-D anchor at comparable teacher-token scale: macro accuracy rises from 15.41 to 17.78 with CLOOPD-Fixed2 and 19.36 with CLOOPD-Fixed3. At step 100, CLOOPD-Fixed3 reaches 15.35, nearly matching TOP-D at step 300 while using 67.2% fewer teacher-scored tokens and 28.0% fewer GPU-hours. Earlier 8-A100 ablations show adaptive α eliminates observed trust-envelope violations; a third pass adds headroom. These results position CLOOPD as a framework for budgeting how fully students learn from teacher-scored tokens.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 14 of its 17 references that are in the map. First posted on arXiv 21 Sep 2026.
Builds on
14 of 17 references.
Closest ancestors · 4
Same direction, newest first
- 2026When Should the Teacher Move? Temporal Coupling and Stability in Self On-Policy Distillation2 Jun
- 2026OPD+: Rethinking the Advantage Design for On-Policy Distillation31 May
- 2026Trust-Region Behavior Blending for On-Policy Distillation29 May
- 2026boldsymbolf-OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Control18 May
Other references · 10
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
- 2015Trust Region Policy Optimizationcited by 2,580
- 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
- 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
- 2016Sequence-Level Knowledge Distillationcited by 801
- 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 186
- 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 162
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillation
- 2026Trajectory Learnability for Offline On-Policy Distillation with Imperfect Teachers
- 2026ReOrder-OPD:Reliability-Aware Prompt Ordering for On-Policy Distillation
- 2026Teaching a Moving Student: Rethinking the Curriculum of On-Policy Distillation
- 2026TISD: On-Policy Self-Distillation with Trajectory Intervention
- 2026On the Off-Policy Teacher in On-Policy Distillation
- 2026Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuning
- 2026Flash-OPD: Fast On-Policy Distillation