# When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation

arXiv: https://arxiv.org/abs/2609.20511 · HTML: https://mlsanity.com/paper/2609.20511/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 21 of its 24 references that are in the map.
- First posted on arXiv 17 Sep 2026.

## Placed in

- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 21 of 21)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation](https://mlsanity.com/paper/2602.12125/) (2026)
- [Entropy-Aware On-Policy Distillation of Language Models](https://mlsanity.com/paper/2603.07079/) (2026)
- [Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes](https://mlsanity.com/paper/2603.25562/) (2026)
- [Fast and Effective On-policy Distillation from Reasoning Prefixes](https://mlsanity.com/paper/2602.15260/) (2026)
- [Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models](https://mlsanity.com/paper/2604.08527/) (2026)
- [Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning](https://mlsanity.com/paper/2605.07804/) (2026)
- [On the Position Bias of On-Policy Distillation](https://mlsanity.com/paper/2606.22600/) (2026)
- [Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillation](https://mlsanity.com/paper/2605.16826/) (2026)
- [Understanding the Effect of Stochasticity in Policy Optimization](https://mlsanity.com/paper/2110.15572/) (2021)
- [Pass the Baton: Trajectory-Relayed On-Policy Distillation](https://mlsanity.com/paper/2607.26057/) (2026)
- [Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation](https://mlsanity.com/paper/2605.13643/) (2026)
- [SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2410.19503/) (2024)
- [Trust-Region Behavior Blending for On-Policy Distillation](https://mlsanity.com/paper/2605.31159/) (2026)
- [SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning](https://mlsanity.com/paper/2608.14277/) (2026)
- [ADWIN: Adaptive Windows for Horizon-Aware On-Policy Distillation](https://mlsanity.com/paper/2605.28396/) (2026)

## Built on by (most cited first, 5 of 5)

- [An RL View of OPD: Least Square Policy Distillation for Sample-Efficient LLM Reasoning](https://mlsanity.com/paper/2609.35505/) (2026)
- [REVO: Rollout-Efficient Off-Policy Distillation via Variance-Guided Reuse](https://mlsanity.com/paper/2609.37500/) (2026)
- [When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better](https://mlsanity.com/paper/2610.11291/) (2026)
- [Branching On-Policy Distillation](https://mlsanity.com/paper/2610.13481/) (2026)
- [Recovering Reliable Supervision via Critique-Guided On-Policy Distillation](https://mlsanity.com/paper/2610.20572/) (2026)

## Abstract (arXiv)

We study length inflation in on-policy distillation (OPD), where student responses can become excessively long and even exhaust the generation budget. We identify termination-token mismatch between base students and post-trained teachers as an important source of this behavior. Across Qwen3, Llama, and Gemma, the two models can place their stopping probability on different EOS tokens, even when their declared stopping sets are identical. This mismatch can suppress the student's preferred termination action without reliably transferring the teacher-preferred alternative. We show that aligning the decoding stopping set alone is insufficient, while treating functionally equivalent EOS tokens as a shared semantic stopping action substantially mitigates mismatch-induced length inflation across all three model families. To further understand how termination behavior evolves over training, we study OPD across different K2-Horizon training stages. This stage-wise analysis shows that termination preferences can shift substantially during training, while also revealing a distinct length inflation late in the OPD run that persists beyond termination alignment. Together, these results identify termination mismatch as an important, but not exhaustive, source of OPD length dynamics. We release an implementation incorporating the proposed termination-handling corrections.