mlsanityData to 7 Oct 2026

Scaling Properties of Same-Family On-Policy Distillation

arXiv 2609.32722 ↗First posted 26 Sep 2026cs.LGcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

*Reinforcement learning (RL)* can induce substantial reasoning capabilities in large language models (LLMs), but how much of this capability transfers across model scales, and how quickly, remains unclear. We study the scaling properties of *on-policy distillation (OPD)* across *weak-to-strong*, *same-base*, and *strong-to-weak* teacher--student setups. We find that early OPD training dynamics uniformly exhibit a regular *useful-transfer* regime, in which held-out accuracy (the *gold score*, G) rises approximately linearly in d=√KL(πθ Vert πref), the square root of token-level reverse KL divergence from the student initialization. In every observed weak-to-strong pair, the student's peak gold score exceeds its teacher's own, so a compact RL expert can transfer capability to a much larger student via OPD. To estimate OPD outcomes, we fit *power laws* for how Gpeak and the slope of the useful-transfer regime scale with student and teacher parameter counts and with teacher gold score. These laws show that peak gold score improves with teacher scale only up to roughly the student's scale, and that at a matched gold score smaller teachers transfer better, so a teacher's score alone does not define its supervision value. We also study the scaling effects of two OPD variants, bootstrapping weak-to-strong OPD, and the degree of on-policy supervision.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 51 of its 60 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

51 of 60 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligence31 Aug
  2. 2026H2SD: Hybrid Hindsight Self-Distillation21 Jul
  3. 2026Understanding Reasoning from Pretraining to Post-Training17 Jul
  4. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing2 Apr

Other references · 47

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2015Trust Region Policy Optimizationcited by 2,503
  9. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  10. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  11. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  12. 2016Sequence-Level Knowledge Distillationcited by 761
35 more
  1. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  2. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  3. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  4. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
  5. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
  6. 2026MiMo-V2-Flash Technical Reportcited by 163
  7. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  8. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 148
  9. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  10. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 122
  11. 2026Self-Distilled RLVRcited by 109
  12. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 91
  13. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 71
  14. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 70
  15. 2025Distillation Scaling Lawscited by 59
  16. 2017Learning the Structure of Generative Models without Labeled Datacited by 50
  17. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  18. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  19. 2025On a few pitfalls in KL divergence gradient estimation for RLcited by 27
  20. 2025Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedbackcited by 26
  21. 2024A Hitchhiker's Guide to Scaling Law Estimationcited by 23
  22. 2024High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Lawscited by 21
  23. 2026Weak-to-Strong Generalization via Direct On-Policy Distillationcited by 14
  24. 2026On-policy Distillation with Verifiable Rewardcited by 10
  25. 2025Incentivizing Strong Reasoning from Weak Supervisioncited by 8
  26. 2026Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentioncited by 8
  27. 2026Distilled Reinforcement Learning for LLM Post-trainingcited by 7
  28. 2025On Teacher Hacking in Language Model Distillationcited by 5
  29. 2026Weak Critics Make Strong Learners: On-Policy Critique Distillation for Scalable Oversightcited by 5
  30. 2026Behavior Cloning is Not All You Need: The Optimality of On-Policy Distillation for Noisy Expert Feedbackcited by 3
  31. 2026Strong Teacher Not Needed? On Distillation in LLM Pretrainingcited by 2
  32. 2026Unified Neural Scaling Lawscited by 2
  33. 2026Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillationcited by 2
  34. 2026The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledgecited by 1
  35. 2026T1: Terminal Agent Reinforcement Learning for Long-Horizon Taskscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
  2. 2026On the Geometry of On-Policy Distillation
  3. 2026Towards Understanding On-Policy Distillation through the Lens of Test-Time Scaling
  4. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agents
  5. 2026PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation
  6. 2025Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning
  7. 2026A Brief Overview: On-Policy Self-Distillation In Large Language Models
  8. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe