mlsanityData to 7 Oct 2026

Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models

arXiv 2607.28449 ↗First posted 30 Jul 2026cs.CLcited by 3

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

On-policy distillation (OPD) provides dense token-level supervision from a teacher, but its effectiveness can depend on teacher consistency, meaning that the model providing OPD supervision should also have generated the demonstrations used to train the supervised fine-tuning (SFT) reference. However, this condition is frequently violated in practice when SFT data have mixed or unknown provenance or when different models are preferred for SFT data generation and subsequent distillation. In such cross-teacher settings, even a stronger OPD teacher can yield little improvement over the SFT reference. We find that raw teacher--reference disagreement contains potentially useful context-specific teacher evidence as well as a recurring component associated with differences in wording, formatting, and reasoning cadence. We introduce Lightning OPD 2.0 with cross-fitted style residualization, which uses rollout-level cross-fitting to estimate this recurring component as an operational proxy for style-token bias and subtracts it before constructing the token-level OPD update. Across mathematical reasoning and code generation benchmarks, Lightning OPD 2.0 consistently outperforms Lightning OPD in cross-teacher settings. Starting from Klear-Reasoner-8B-SFT, Lightning OPD 2.0 reaches 82.4% on AIME 2024 and 63.0% on LiveCodeBench v5. Together, these results establish Lightning OPD 2.0 as a practical approach to cross-teacher OPD, relaxing teacher consistency as a prerequisite and allowing the SFT data generator and distillation teacher to be selected independently. Code will be released soon.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 57 of its 75 references that are in the map. First posted on arXiv 30 Jul 2026.

Builds on

57 of 75 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Self-Supervised On-Policy Distillation for Reasoning Language Models17 May
  2. 2026Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning13 May
  3. 2026OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning12 May
  4. 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why11 May
  5. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe14 Apr
  6. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation14 Apr
  7. 2026Reinforcement-aware Knowledge Distillation for LLM Reasoning26 Feb
  8. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models26 Jan

Other references · 49

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  8. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  9. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  10. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  11. 2025OpenAI GPT-5 System Cardcited by 910
  12. 2025Group Sequence Policy Optimizationcited by 788
37 more
  1. 2024DeepSeek-V3 Technical Reportcited by 724
  2. 2025Process Reinforcement through Implicit Rewardscited by 478
  3. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 450
  4. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  5. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
  6. 2026Reinforcement Learning via Self-Distillationcited by 285
  7. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Taskscited by 281
  8. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 275
  9. 2025Learning to Reason under Off-Policy Guidancecited by 266
  10. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  11. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalizationcited by 233
  12. 2026Self-Distillation Enables Continual Learningcited by 212
  13. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  14. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  15. 2026MiMo-V2-Flash Technical Reportcited by 163
  16. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 162
  17. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  18. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  19. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 148
  20. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 122
  21. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 103
  22. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secretcited by 95
  23. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questionscited by 95
  24. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 88
  25. 2026Privileged Information Distillation for Language Modelscited by 79
  26. 2025UFT: Unifying Supervised and Reinforcement Fine-Tuningcited by 58
  27. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 57
  28. 2025NVIDIA Nemotron 3: Efficient and Open Intelligencecited by 54
  29. 2025Black-Box On-Policy Distillation of Large Language Modelscited by 53
  30. 2026Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillationcited by 53
  31. 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 45
  32. 2026Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillationcited by 42
  33. 2025Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoningcited by 28
  34. 2026On the Position Bias of On-Policy Distillationcited by 23
  35. 2026SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Samplingcited by 12
  36. 2025Revealing the Power of Post-Training for Small Language Models via Knowledge Distillationcited by 6
  37. 2026ORBIT: On-policy Exploration-Exploitation for Controllable Multi-Budget Reasoningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Lightning Weave: Improving the Accuracy-Efficiency Frontier of Reasoning Models through Capability Compositioncited by 0
  2. 2026Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillationcited by 0
  3. 2026When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Bettercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability
  2. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe
  3. 2026Learning to Revise Reasoning with Segment-wise On-Policy Distillation
  4. 2026Contrastive On-Policy Distillation
  5. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
  6. 2026Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation
  7. 2026Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning
  8. 2026Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts