# Escaping the KL Agreement Trap in On-Policy Distillation

arXiv: https://arxiv.org/abs/2606.09471 · HTML: https://mlsanity.com/paper/2606.09471/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 11 of its 18 references that are in the map.
- First posted on arXiv 8 Jun 2026.

## Placed in

- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 11 of 11)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation](https://mlsanity.com/paper/2602.12125/) (2026)
- [Entropy-Aware On-Policy Distillation of Language Models](https://mlsanity.com/paper/2603.07079/) (2026)
- [Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts](https://mlsanity.com/paper/2506.02177/) (2025)
- [Scaling Reasoning Efficiently via Relaxed On-Policy Distillation](https://mlsanity.com/paper/2603.11137/) (2026)
- [Stable On-Policy Distillation through Adaptive Target Reformulation](https://mlsanity.com/paper/2601.07155/) (2026)
- [Fast and Effective On-policy Distillation from Reasoning Prefixes](https://mlsanity.com/paper/2602.15260/) (2026)

## Built on by (most cited first, 11 of 11)

- [Mismatch Matters: On-Policy Distillation Beyond Token Agreement](https://mlsanity.com/paper/2608.09836/) (2026)
- [Outcome-Confounded Local Supervision in On-Policy Distillation](https://mlsanity.com/paper/2607.23731/) (2026)
- [1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation](https://mlsanity.com/paper/2609.24432/) (2026)
- [Understanding On-Policy Distillation: A Mechanistic Interpretability Perspective via Sparse Crosscoders](https://mlsanity.com/paper/2609.35210/) (2026)
- [Solving Without Stopping: On-Policy Distillation at Small Scale](https://mlsanity.com/paper/2609.37326/) (2026)
- [Look Before You Select: Rethinking Vocabulary Sparsification in On-Policy Distillation](https://mlsanity.com/paper/2609.34386/) (2026)
- [Interpolated Policy Distillation: A Controllable Continuum Between Off-Policy and On-Policy Distillation](https://mlsanity.com/paper/2609.37170/) (2026)
- [Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives](https://mlsanity.com/paper/2609.38666/) (2026)
- [GFD-OPD: Guidance-Folded On-Policy Distillation of Diffusion Models Across Scales](https://mlsanity.com/paper/2609.39692/) (2026)
- [Branching On-Policy Distillation](https://mlsanity.com/paper/2610.13481/) (2026)
- [Recovering Reliable Supervision via Critique-Guided On-Policy Distillation](https://mlsanity.com/paper/2610.20572/) (2026)

## Abstract (arXiv)

On-policy distillation (OPD) provides dense token-level supervision by asking a teacher to score student-generated rollouts. However, when the student drifts into an unrecoverable prefix, the teacher may locally agree with the degraded state, producing low reverse KL but little corrective training signal. We identify this persistent regime as a low-KL agreement trap. Further analyses show that tokens during and after such traps produce less useful supervision signals. We propose KAT (KL Agreement Trap Termination), an online OPD termination rule that detects persistent low-KL agreement with a dynamic training-adaptive threshold. By filtering weak supervision from degenerate agreement, KAT improves avg@k accuracy by 2.66% and pass@k by 3.43% across four mathematical benchmarks, while reducing average rollout length by 59.73%.