# Weak-to-Strong On-Policy Distillation arXiv: https://arxiv.org/abs/2607.26246 ยท HTML: https://mlsanity.com/paper/2607.26246/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. - Builds on 53 of its 58 references that are in the map. - First posted on arXiv 28 Jul 2026. ## Placed in - Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/) ## Builds on (most cited first, 30 of 53) - [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024) - [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025) - [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023) - [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024) - [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023) - [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026) - [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023) - [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019) - [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026) - [Process Reinforcement through Implicit Rewards](https://mlsanity.com/paper/2502.01456/) (2025) - [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023) - [Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models](https://mlsanity.com/paper/2601.18734/) (2026) - [DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts](https://mlsanity.com/paper/2105.03023/) (2021) - [GeDi: Generative Discriminator Guided Sequence Generation](https://mlsanity.com/paper/2009.06367/) (2020) - [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026) - [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026) - [MathArena: Evaluating LLMs on Uncontaminated Math Competitions](https://mlsanity.com/paper/2505.23281/) (2025) - [DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning](https://mlsanity.com/paper/2504.11456/) (2025) - [Self-Distillation Enables Continual Learning](https://mlsanity.com/paper/2601.19897/) (2026) - [Generalizing Verifiable Instruction Following](https://mlsanity.com/paper/2507.02833/) (2025) - [MiMo-V2-Flash Technical Report](https://mlsanity.com/paper/2601.02780/) (2026) - [On-Policy Context Distillation for Language Models](https://mlsanity.com/paper/2602.12275/) (2026) - [Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation](https://mlsanity.com/paper/2602.12125/) (2026) - [Entropy-Aware On-Policy Distillation of Language Models](https://mlsanity.com/paper/2603.07079/) (2026) - [A Survey of On-Policy Distillation for Large Language Models](https://mlsanity.com/paper/2604.00626/) (2026) - [COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics](https://mlsanity.com/paper/2202.11705/) (2022) - [Supervising strong learners by amplifying weak experts](https://mlsanity.com/paper/1810.08575/) (2018) - [Weak-to-Strong Jailbreaking on Large Language Models](https://mlsanity.com/paper/2401.17256/) (2024) - [Tuning Language Models by Proxy](https://mlsanity.com/paper/2401.08565/) (2024) - [Scaling Reasoning Efficiently via Relaxed On-Policy Distillation](https://mlsanity.com/paper/2603.11137/) (2026) ## Built on by (most cited first, 8 of 8) - [Where to Look Matters: On-Policy Self-Distillation for Long-Video Understanding](https://mlsanity.com/paper/2608.25356/) (2026) - [Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoning](https://mlsanity.com/paper/2608.02831/) (2026) - [Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation](https://mlsanity.com/paper/2609.08798/) (2026) - [WDL-OPD: Weak-Driven On-Policy Distillation via Mixture-Constrained Co-Training](https://mlsanity.com/paper/2608.09447/) (2026) - [CompassOPD: Cross-Family On-Policy Distillation via Within-Family Likelihood Shifts](https://mlsanity.com/paper/2609.10154/) (2026) - [Lightning Weave: Improving the Accuracy-Efficiency Frontier of Reasoning Models through Capability Composition](https://mlsanity.com/paper/2609.14708/) (2026) - [Allspark: Weak to Strong Transfer via Alternating Chain of Thought](https://mlsanity.com/paper/2609.32913/) (2026) - [CURIO: Curiosity-Driven Test-Time Learning for Open-Ended Discovery](https://mlsanity.com/paper/2610.04851/) (2026) ## Abstract (arXiv) On-policy distillation (OPD), which aligns a student with the teacher's token-level distribution on the student's own rollouts, is an effective paradigm for transferring capabilities across LLMs. Prevailing approaches assume a teacher at least as capable as the student: they either distill a larger model into a smaller one, which fails at the frontier where no larger teacher exists, or consolidate multiple domain experts trained from a shared base, which requires costly training at the student's scale. We introduce Weak-to-Strong On-Policy Distillation (W2S-OPD), a simple yet effective OPD framework that improves the strong student by distilling from multiple weak models. W2S-OPD constructs a proxy teacher in logit space from a contrast pair of a positive and a negative model, both smaller than the student and cheap to obtain. Their logit difference isolates the capability direction, which is added to the student's own base model, yielding a proxy teacher that couples this direction while staying distributionally adjacent to the student. The student then distills it by minimizing the per-token reverse KL on its own rollouts. We instantiate the contrast pair as i) a post-RL expert against its pre-RL initialization, isolating the skill RL instills, ii) a larger against a smaller base model, isolating the capability from scale, and iii) a small base model with correct versus wrong hints, isolating the instance-level direction toward the solution. Across four math and three code benchmarks, W2S-OPD outperforms OPD, enables the student to surpass the domain teacher, and keeps improving the student even when every supervision source is weaker. Analysis shows different contrasts yield distinct signals: the post-RL and hint contrasts emphasize reasoning frameworks, while the scale contrast emphasizes the solving procedure. Our code will be available at https://github.com/Yu-Fangxu/W2S-OPD.