# On-policy Distillation with Verifiable Reward

arXiv: https://arxiv.org/abs/2608.24696 · HTML: https://mlsanity.com/paper/2608.24696/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 34 of its 38 references that are in the map.
- First posted on arXiv 25 Aug 2026.

## Placed in

- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 30 of 34)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026)
- [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.16419/) (2025)
- [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026)
- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning](https://mlsanity.com/paper/2504.11456/) (2025)
- [ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learning](https://mlsanity.com/paper/2504.01296/) (2025)
- [MiMo-V2-Flash Technical Report](https://mlsanity.com/paper/2601.02780/) (2026)
- [Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation](https://mlsanity.com/paper/2602.12125/) (2026)
- [Entropy-Aware On-Policy Distillation of Language Models](https://mlsanity.com/paper/2603.07079/) (2026)
- [Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes](https://mlsanity.com/paper/2603.25562/) (2026)
- [Self-Distilled RLVR](https://mlsanity.com/paper/2604.03128/) (2026)
- [Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing](https://mlsanity.com/paper/2604.02288/) (2026)
- [Learn to Reason Efficiently with Adaptive Length-based Reward Shaping](https://mlsanity.com/paper/2505.15612/) (2025)
- [Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning](https://mlsanity.com/paper/2506.05256/) (2025)
- [Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe](https://mlsanity.com/paper/2605.03677/) (2026)
- [Trust Region On-Policy Distillation](https://mlsanity.com/paper/2606.01249/) (2026)
- [Rethinking Entropy Regularization in Large Reasoning Models](https://mlsanity.com/paper/2509.25133/) (2025)
- [KL for a KL: On-Policy Distillation with Control Variate Baseline](https://mlsanity.com/paper/2605.07865/) (2026)
- [CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning](https://mlsanity.com/paper/2509.20712/) (2025)
- [Train Long, Think Short: Curriculum Learning for Efficient Reasoning](https://mlsanity.com/paper/2508.08940/) (2025)
- [HAPO: Training Language Models to Reason Concisely via History-Aware Policy Optimization](https://mlsanity.com/paper/2505.11225/) (2025)
- [EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Control](https://mlsanity.com/paper/2511.15248/) (2025)
- [SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling](https://mlsanity.com/paper/2606.09304/) (2026)

## Built on by (most cited first, 15 of 15)

- [Train4Merge: A Controlled Single-Teacher Study of RL vs. SFT Teachers for OPD-Based Model Merging](https://mlsanity.com/paper/2609.32303/) (2026)
- [Expected Reasoning-Step Return Unifies On-Policy Learning from Rewards and Teachers](https://mlsanity.com/paper/2609.32674/) (2026)
- [Scaling Properties of Same-Family On-Policy Distillation](https://mlsanity.com/paper/2609.32722/) (2026)
- [DuoOPD: Learning from Joint Teacher-Student Outcomes for Multi-Task On-Policy Distillation](https://mlsanity.com/paper/2609.33711/) (2026)
- [Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?](https://mlsanity.com/paper/2609.33791/) (2026)
- [TIDE: Teacher-Student Transition via Informative Distillation and Exploration for Agentic RL](https://mlsanity.com/paper/2609.35058/) (2026)
- [Reward-Aligned Reweighting for On-Policy Distillation](https://mlsanity.com/paper/2609.35517/) (2026)
- [Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models](https://mlsanity.com/paper/2609.38025/) (2026)
- [On the Off-Policy Teacher in On-Policy Distillation](https://mlsanity.com/paper/2609.38360/) (2026)
- [From Given to Gathered Evidence: Agentic Learning for Longitudinal Medical Reasoning](https://mlsanity.com/paper/2609.39566/) (2026)
- [SCAD: Structured Credit Assignment and Distillation for Long-Horizon Agents](https://mlsanity.com/paper/2610.03372/) (2026)
- [Transfer-Stratified On-Policy Distillation for RL-Improved Reasoning Teachers](https://mlsanity.com/paper/2610.05974/) (2026)
- [Residual Advantage: Student-Relative Teacher Guidance for RL with Verifiable Rewards](https://mlsanity.com/paper/2610.11519/) (2026)
- [Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2610.13521/) (2026)
- [LLM Value Functions for Credit Assignment in Agentic RL](https://mlsanity.com/paper/2610.15840/) (2026)

## Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) and on-policy distillation (OPD) have become two widely adopted paradigms for post-training large language models. However, RLVR suffers from sparse task-level feedback, while OPD provides dense token-level guidance but ignores trajectory correctness, limiting its performance to that of the teacher. Combining them is a promising direction: OPD supplies dense supervisory signals, while RLVR provides task-level correctness. Nevertheless, existing integrations often rely on weighted combination or heuristic switching, introducing extra hyperparameters and trade-offs. We propose On-policy Distillation with Verifiable Reward (OPDVR), a simple yet effective method that seamlessly combines OPD and RLVR without adding any hyperparameters. We first reformulate the implicit reward of sampled-token OPD based on trajectory correctness, then apply a ReLU gating mechanism to ensure that correct trajectories receive non-negative rewards and incorrect ones receive non-positive rewards---thereby aligning the distillation signal with task success while preserving the teacher's distributional guidance. Furthermore, our modification transforms sampled-token OPD into a proper RLVR method, making it readily combinable with any policy gradient algorithm, such as GRPO. Experiments on six reasoning benchmarks show that OPDVR consistently outperforms standard OPD. Our code is available at https://github.com/LeapLabTHU/OPDVR.