mlsanityData to 7 Oct 2026

Escaping the KL Agreement Trap in On-Policy Distillation

arXiv 2606.09471 ↗First posted 8 Jun 2026cs.LG cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

On-policy distillation (OPD) provides dense token-level supervision by asking a teacher to score student-generated rollouts. However, when the student drifts into an unrecoverable prefix, the teacher may locally agree with the degraded state, producing low reverse KL but little corrective training signal. We identify this persistent regime as a low-KL agreement trap. Further analyses show that tokens during and after such traps produce less useful supervision signals. We propose KAT (KL Agreement Trap Termination), an online OPD termination rule that detects persistent low-KL agreement with a dynamic training-adaptive threshold. By filtering weak supervision from degenerate agreement, KAT improves avg@k accuracy by 2.66% and pass@k by 3.43% across four mathematical benchmarks, while reducing average rollout length by 59.73%.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 11 of its 18 references that are in the map. First posted on arXiv 8 Jun 2026.

Builds on

11 of 18 references.

References · 11

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  3. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  4. 2016Sequence-Level Knowledge Distillationcited by 761
  5. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
  6. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  7. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  8. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rolloutscited by 108
  9. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 71
  10. 2026Stable On-Policy Distillation through Adaptive Target Reformulationcited by 47
  11. 2026Fast and Effective On-policy Distillation from Reasoning Prefixescited by 43

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Mismatch Matters: On-Policy Distillation Beyond Token Agreementcited by 7
  2. 2026Outcome-Confounded Local Supervision in On-Policy Distillationcited by 3
  3. 20261% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillationcited by 1
  4. 2026Look Before You Select: Rethinking Vocabulary Sparsification in On-Policy Distillationcited by 0
  5. 2026Understanding On-Policy Distillation: A Mechanistic Interpretability Perspective via Sparse Crosscoderscited by 0
  6. 2026Interpolated Policy Distillation: A Controllable Continuum Between Off-Policy and On-Policy Distillationcited by 0
  7. 2026Solving Without Stopping: On-Policy Distillation at Small Scalecited by 0
  8. 2026GFD-OPD: Guidance-Folded On-Policy Distillation of Diffusion Models Across Scalescited by 0