mlsanityData to 14 Oct 2026

On-Policy Delta Distillation

arXiv 2607.15161 ↗First posted 16 Jul 2026cs.LG cs.CLcited by 15

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

also in 1 more
Abstract (arXiv)

On-policy distillation is an alternative post-training method in reinforcement learning that alleviates the constraints imposed by reward models by providing token-level supervision from a teacher model. Although on-policy distillation has been studied and applied across various settings, its fundamental design remains underexplored. In this paper, we introduce a new distillation reward, termed the delta signal, instead of directly imitating the teacher's output distribution. The delta signal is defined as the difference between the teacher model and its base model prior to instruction tuning for reasoning capability. It therefore captures the changes induced by reasoning tuning and provides a more direct signal for transferring reasoning capabilities. Using extensive empirical evidence, we show that the delta signal substantially improves on-policy distillation and refer to the new distillation method as On-Policy Delta Distillation (OPD2). Experiments across mathematics, science, and code-reasoning benchmarks demonstrate that OPD2 consistently outperforms conventional on-policy distillation, enabling reasoning LLMs to achieve strong performance with only a short post-training period. Code will be available at https://github.com/naver-ai/opd2

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 42 references that are in the map. First posted on arXiv 16 Jul 2026.

Builds on

35 of 42 references.

References · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  9. 2025s1: Simple test-time scalingcited by 1,341
  10. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  11. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  12. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
23 more
  1. 2016Sequence-Level Knowledge Distillationcited by 801
  2. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  3. 2015Policy Distillationcited by 440
  4. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 304
  5. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 299
  6. 2026Gemma 4 Technical Reportcited by 259
  7. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 256
  8. 2026MiMo-V2-Flash Technical Reportcited by 191
  9. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 184
  10. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 175
  11. 2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning datasetcited by 174
  12. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 163
  13. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 163
  14. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 137
  15. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 130
  16. 2025OpenCodeReasoning: Advancing Data Distillation for Competitive Codingcited by 113
  17. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?cited by 103
  18. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 82
  19. 2026MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Trainingcited by 72
  20. 2023An Emulator for Fine-Tuning Large Language Models using Small Language Modelscited by 61
  21. 2026CRISP: Compressed Reasoning via Iterative Self-Policy Distillationcited by 58
  22. 2024Pre-training Distillation for Large Language Models: A Design Space Explorationcited by 27
  23. 2024BiLD: Bi-directional Logits Difference Loss for Large Language Model Distillationcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 15.

15 papers.

Sort:

By area:

  1. 2026Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progresscited by 3
  2. 2026Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillationcited by 2
  3. 2026Negative Self-Distillation: Learning to Reason by Avoiding Flawscited by 2
  4. 2026LastOPD: Taming Collapse in Latent On-Policy Distillationcited by 2
  5. 2026On-Policy Delta Distillation for Multilingual Math Reasoningcited by 1
  6. 2026Data-Free On-Policy Distillation: How Far Can We Go Without External Data?cited by 1
  7. 2026WDL-OPD: Weak-Driven On-Policy Distillation via Mixture-Constrained Co-Trainingcited by 0
  8. 2026Not Every Token Is Worth Distilling: Selective Supervision for Direct-OPDcited by 0
  9. 2026Expected Reasoning-Step Return Unifies On-Policy Learning from Rewards and Teacherscited by 0
  10. 2026Sharpening Tax in Post-Trainingcited by 0
  11. 2026ResOPD: Tail Residualization for Sparse On-Policy Distillationcited by 0
  12. 2026On-Policy Distillation with Negative-Policy Rolloutscited by 0
3 more
  1. 2026Composing What Each Teacher Learned: Multi-Teacher On-Policy Distillation through Teacher-Relative Shiftscited by 0
  2. 2026Rethinking Relative Policy Differences Transfer of On-Policy Distillationcited by 0
  3. 2026Reward-Aligned Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Fast and Effective On-policy Distillation from Reasoning Prefixes
  2. 2026Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations
  3. 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why
  4. 2026Does self-distillation improve reasoning?
  5. 2026Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR
  6. 2026Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization
  7. 2026Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement
  8. 2026On the Position Bias of On-Policy Distillation