# SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling

arXiv: https://arxiv.org/abs/2606.09304 · HTML: https://mlsanity.com/paper/2606.09304/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 15 of its 19 references that are in the map.
- First posted on arXiv 8 Jun 2026.

## Placed in

- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)
- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 15 of 15)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning](https://mlsanity.com/paper/2504.11456/) (2025)
- [Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation](https://mlsanity.com/paper/2602.12125/) (2026)
- [On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting](https://mlsanity.com/paper/2508.11408/) (2025)
- [Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillation](https://mlsanity.com/paper/2602.02994/) (2026)
- [Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level](https://mlsanity.com/paper/2605.06387/) (2026)

## Built on by (most cited first, 13 of 13)

- [On-policy Distillation with Verifiable Reward](https://mlsanity.com/paper/2608.24696/) (2026)
- [Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models](https://mlsanity.com/paper/2607.28449/) (2026)
- [Simple-OPD: Demystifying Warm-up for On-policy Distillation](https://mlsanity.com/paper/2608.06802/) (2026)
- [Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning](https://mlsanity.com/paper/2608.19181/) (2026)
- [REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation](https://mlsanity.com/paper/2608.11698/) (2026)
- [Trust Is Not Enough: Influence Calibration for On-Policy Self-Distillation in Agentic RL](https://mlsanity.com/paper/2608.14945/) (2026)
- [Who Teaches Which Token? Verifier-Gated Multi-Expert On-Policy Distillation for Scientific Reasoning](https://mlsanity.com/paper/2609.15404/) (2026)
- [When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillation](https://mlsanity.com/paper/2608.27960/) (2026)
- [Blockwise Policy-Drift Gating for On-Policy Distillation](https://mlsanity.com/paper/2606.24084/) (2026)
- [When and Where to Trust the Teacher: Unifying On-Policy Distillation and GRPO through Entropy-Calibrated Credit Assignment](https://mlsanity.com/paper/2609.28385/) (2026)
- [Look Before You Select: Rethinking Vocabulary Sparsification in On-Policy Distillation](https://mlsanity.com/paper/2609.34386/) (2026)
- [Reward-Aligned Reweighting for On-Policy Distillation](https://mlsanity.com/paper/2609.35517/) (2026)
- [LLM Value Functions for Credit Assignment in Agentic RL](https://mlsanity.com/paper/2610.15840/) (2026)

## Abstract (arXiv)

On-policy distillation (OPD) trains a student on its own trajectories with dense per-token supervision from a stronger teacher, and often outperforms off-policy distillation and standard reinforcement learning. However, we find that its effectiveness implicitly relies on two assumptions that frequently break in practice: trajectory-level alignment between the student and the teacher, and uniform token-level reliability of the teacher's preferences. We therefore propose Sign-Gated On-Policy Distillation (SG-OPD), which uses a binary verifier as a trust signal for the teacher at two complementary granularities: phased teacher sampling mixes in verifier-endorsed teacher rollouts at cold-start, and a sign-consistency gate extrapolates the distillation update on tokens where the teacher agrees with the verifier-correct direction and interpolates it where it disagrees. Experiments on competition-level mathematical reasoning benchmarks show that SG-OPD consistently outperforms standard OPD, with average gains of 1.98 and 7.50 at the per-sample and per-question levels, respectively.