mlsanityData to 7 Oct 2026

The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs

arXiv 2605.08737 ↗First posted 9 May 2026cs.LG cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

On-policy distillation (OPD) is widely used for LLM post-training. When pushed with a reward-extrapolation coefficient lambda > 1, the student can lift past the teacher in domain, but past a threshold lambda* the same step violates the output contract on structured-output tasks. In a single-position Bernoulli reduction, we derive a closed-form base-relative clip-safety threshold lambda*(p,b,c) determined by three measurable quantities: the teacher modal probability, the warm-start mass, and the importance-sampling clip strength. Above lambda*, the extrapolated fixed point exits the clip-safe region, changing training from format-preserving to format-collapsing. We extend the rule to calibrated K-ary listwise JSON tasks where a single binding equivalence class dominates the output contract and SFT retains parse headroom. On Amazon Fashion, three pre-registered tests--a fine-grid cliff interval, a budget-extension test, and a small-clip cross-prediction--fall within their locked prediction windows, with the small-clip value matching the closed-form prediction below grid resolution. Operating just below lambda*, ListOPD brings a 1.7B Qwen3 student to in-domain parity with an 8B-SFT baseline at one-fifth the parameters. The gain is driven primarily by format adherence: NDCG@1 on parsed outputs remains flat across lambda, while parse validity sharply changes at the predicted boundary. The cliff diagnostic is rubric-independent, whereas the parity claim uses a Gemini-graded rubric and inherits that evaluator's exposure.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 36 of its 41 references that are in the map. First posted on arXiv 9 May 2026.

Builds on

36 of 41 references.

References · 36

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  5. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  6. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  7. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
  8. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  9. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
  10. 2021Overview of the TREC 2020 deep learning trackcited by 193
  11. 2023Efficient Guided Generation for Large Language Modelscited by 192
  12. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
24 more
  1. 2023RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!cited by 151
  2. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  3. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 148
  4. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 145
  5. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 122
  6. 2022RankT5: Fine-Tuning T5 for Text Ranking with Ranking Lossescited by 114
  7. 2024Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generationcited by 99
  8. 2026TIP: Token Importance in On-Policy Distillationcited by 90
  9. 2023Zero-Shot Listwise Document Reranking with a Large Language Modelcited by 81
  10. 2024XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Modelscited by 73
  11. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 73
  12. 2025JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Modelscited by 66
  13. 2024FIRST: Faster Improved Listwise Reranking with Single Token Decodingcited by 54
  14. 2026Stable On-Policy Distillation through Adaptive Target Reformulationcited by 47
  15. 2025The Price of Format: Diversity Collapse in LLMscited by 43
  16. 2025Learning to Generate Structured Output with Schema Reinforcement Learningcited by 27
  17. 2025When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RLcited by 21
  18. 2025Think Inside the JSON: Reinforcement Strategy for Strict LLM Schema Adherencecited by 16
  19. 2026VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Trainingcited by 9
  20. 2025Decoupling Task-Solving and Output Formatting in LLM Generationcited by 8
  21. 2026The Hidden Cost of Structured Generation in LLMs: Draft-Conditioned Constrained Decodingcited by 7
  22. 2026Coverage Improvement and Fast Convergence of On-policy Preference Learningcited by 5
  23. 2026LLMStructBench: Benchmarking Large Language Model Structured Data Extractioncited by 4
  24. 2026RLPO: Residual Listwise Preference Optimization for Long-Context Review Rankingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Look Before You Select: Rethinking Vocabulary Sparsification in On-Policy Distillationcited by 0
  2. 2026The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillationcited by 0