mlsanityData to 7 Oct 2026

Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation

arXiv 2604.13010 ↗First posted 14 Apr 2026cs.LG cs.AIcited by 28

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

On-policy distillation (OPD) is an effective post-training paradigm for large language models but requires a live teacher server throughout training, resulting in substantial infrastructure overhead. We investigate whether OPD can be performed offline by precomputing teacher log-probabilities once over SFT rollouts and reusing them during training. We find that naively doing so fails to reliably match standard OPD, and trace the root cause to a previously overlooked condition we term teacher consistency, requiring that the same teacher be used for both supervised fine-tuning and OPD. Violating this condition introduces a gradient bias that degrades performance for both offline and online OPD. Building on this insight, we propose Lightning OPD, an offline on-policy distillation framework that enforces teacher consistency and eliminates the need for a live teacher server entirely. We prove that, under teacher consistency, Lightning OPD shares the same optimum as standard OPD, with bounded gradient discrepancy and an implicit regularization effect that helps prevent policy drift. Experiments on math reasoning and code generation show that Lightning OPD achieves comparable performance to standard OPD while delivering 4.0x higher training efficiency. Starting from an SFT-initialized Qwen3-8B-Base model, Lightning OPD reaches 69.9% on AIME 2024 in just 30 GPU hours. Lightning OPD further scales to MoE architectures, training Qwen3-30B-A3B to 71.0% on AIME 2024 on a single 8xH100 node, substantially lowering the barrier for academic research on LLM post-training. Our code is released at https://github.com/jet-ai-projects/Lightning-OPD.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 52 of its 73 references that are in the map. First posted on arXiv 14 Apr 2026.

Builds on

52 of 73 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Reinforcement-aware Knowledge Distillation for LLM Reasoning26 Feb
  2. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models26 Jan

Other references · 50

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  8. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  9. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  10. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  11. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  12. 2025OpenAI GPT-5 System Cardcited by 910
38 more
  1. 2025Group Sequence Policy Optimizationcited by 788
  2. 2024DeepSeek-V3 Technical Reportcited by 724
  3. 2025Process Reinforcement through Implicit Rewardscited by 478
  4. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 450
  5. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  6. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  7. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
  8. 2026Reinforcement Learning via Self-Distillationcited by 285
  9. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Taskscited by 281
  10. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 275
  11. 2025Learning to Reason under Off-Policy Guidancecited by 266
  12. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  13. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalizationcited by 233
  14. 2026Self-Distillation Enables Continual Learningcited by 212
  15. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  16. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  17. 2026MiMo-V2-Flash Technical Reportcited by 163
  18. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 162
  19. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  20. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  21. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 148
  22. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 122
  23. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 103
  24. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secretcited by 95
  25. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questionscited by 95
  26. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 88
  27. 2026Privileged Information Distillation for Language Modelscited by 79
  28. 2025UFT: Unifying Supervised and Reinforcement Fine-Tuningcited by 58
  29. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 57
  30. 2025NVIDIA Nemotron 3: Efficient and Open Intelligencecited by 54
  31. 2025Black-Box On-Policy Distillation of Large Language Modelscited by 53
  32. 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 45
  33. 2025Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoningcited by 28
  34. 2025EpiCoder: Encompassing Diversity and Complexity in Code Generationcited by 21
  35. 2026LLMs Can Learn to Reason Via Off-Policy RLcited by 10
  36. 2025Revealing the Power of Post-Training for Small Language Models via Knowledge Distillationcited by 6
  37. 2026ORBIT: On-policy Exploration-Exploitation for Controllable Multi-Budget Reasoningcited by 6
  38. 2026PCL-Reasoner-V1.5: Advancing Math Reasoning with Offline Reinforcement Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 28.

28 papers.

Sort:

By area:

  1. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 43
  2. 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillationcited by 33
  3. 2026Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillationcited by 31
  4. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 17
  5. 2026Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillationcited by 13
  6. 2026Rubric-based On-policy Distillationcited by 11
  7. 2026SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillationcited by 11
  8. 2026On-Policy Self-Distillation without Any Supervisioncited by 11
  9. 2026Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Trainingcited by 9
  10. 2026DOPD: Dual On-policy Distillationcited by 9
  11. 2026Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rolloutscited by 7
  12. 2026Distilled Reinforcement Learning for LLM Post-trainingcited by 7
16 more
  1. 2026Latent On-Policy Self-Distillationcited by 6
  2. 2026A Formula-Driven Survey and Research Agenda for On-Policy Distillationcited by 5
  3. 2026Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Modelscited by 3
  4. 2026Simple-OPD: Demystifying Warm-up for On-policy Distillationcited by 3
  5. 2026SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervisioncited by 2
  6. 2026OPD-Evolver: Cultivating Holistic Agent Evolver via On-Policy Distillationcited by 1
  7. 2026Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillationcited by 1
  8. 2026From Parameters to Behaviors: A Survey of Model Fusion for Large Language Modelscited by 1
  9. 2026SpikeOPD: Stable On-Policy Distillation for Autoregressive Spiking Language Modelscited by 0
  10. 2026Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillationcited by 0
  11. 2026Lightning Weave: Improving the Accuracy-Efficiency Frontier of Reasoning Models through Capability Compositioncited by 0
  12. 2026Trajectory Learnability for Offline On-Policy Distillation with Imperfect Teacherscited by 0
  13. 2026WAM-OPD: Sharpening World Action Models via On-Policy Distillationcited by 0
  14. 2026Unbiased Top-k Estimation for On-Policy Distillationcited by 0
  15. 2026Act First, Reason Later: Accelerating On-Policy Distillation for Multi-Turn Agents via Reference-Conditioned Inverse Dynamicscited by 0
  16. 2026When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Bettercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
  2. 2026Distribution Corrected Offline Data Distillation for Large Language Models
  3. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
  4. 2026Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning
  5. 2026A Brief Overview: On-Policy Self-Distillation In Large Language Models
  6. 2026Diagnosing On-Policy Self-Distillation for Reasoning Language Models
  7. 2026What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection
  8. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe