mlsanityData to 14 Oct 2026

On-policy and self-distillation for LLMs

446 papers22 this weekHigh confidence in this grouping: the papers form one clear topicHigh confidence in this grouping: the papers form one clear topic · About

Keywords: on-policy, opd, teacher, on-policy distillation, reasoning

Key facts. On-policy and self-distillation for LLMs: 446 papers on arXiv since 2021 (data through 14 Oct 2026). 167 of its 446 mapped papers (37%) cite “Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models” (2026, arXiv 2601.18734). 184 new papers 25 Sep–14 Oct.

What’s new

25 Sep – 14 Oct 2026

184 new papers, 25 Sep – 14 Oct; 22 this week.

  1. This week · 22
  2. Do As I Do, Not As I Say: On-Policy Distillation Inherits the Teacher in Vision-Language Models
  3. Recovering Reliable Supervision via Critique-Guided On-Policy Distillation
  4. VASPO: Self-Distillation with Privileged Value Estimation
  5. RM-OPD: On-Policy Distillation for Generative Reward Models
  6. ActionOPD: Distilling Decisions, Not Tokens
  7. More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning
  8. Function-Space Temporal Self-Distillation for Policy Optimization
  9. Divergence-Routed On-Policy Distillation for Language Models
  10. DTDD: Divergence-Triggered Dynamic Distillation for Reliable On-Policy Supervision
  11. Outcome-Routed Distillation and Reinforcement Learning for Diffusion Language Models
All 184
  1. Demystifying On-Policy Distillation: Where Does the Teacher's Advantage Live?
  2. Does self-distillation improve reasoning?
  3. Imprinting: Knowledge Distillation Can Shift Student Behavior
  4. Teacher-Guided Tool Learning for Small Language Models via Bidirectional KL Constraints
  5. Teach Me After You Learn It: Lookahead Self-Distillation for RLVR
  6. LoOPSD: On-Policy Self-Distillation with Looped Teachers
  7. Failure Recovery via Belief Updates and World Modeling for Long-Horizon Agents
  8. Correcting or Continuing? What On-Policy Distillation Learns from Student Errors
  9. Explore with Memory, Learn without It: In-Context Exploration and Self-Distillation for LLM Reasoning
  10. RELAY: Selective Self-Distillation with Adaptive Hand-off for Rubric-Based Reinforcement Learning
  11. Training Generalist Navigation Policies via Action Self-Distillation
  12. Self-Distillation with Verbal Regularization
  13. Earlier · 162
  14. Radioactive Feedback: Tracing Models Trained by Label-Only LLM Supervision
  15. A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning
  16. Defence against the dark arts of distillation: Unsupervised discovery of subliminal learning also fits: Knowledge distillation for LLMs
  17. Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation
  18. Branching On-Policy Distillation
  19. CAST: Credit Alignment for Cross-Tokenizer On-Policy Distillation
  20. State-OPD: State-Grounded On-Policy Distillation for Multi-Turn Agents
  21. Rethinking Relative Policy Differences Transfer of On-Policy Distillation
  22. Redirecting Teacher Supervision: Temporal-Response-Guided Latent Distillation for Video-Language Models also fits: Multimodal distillation
  23. JoyInside-VLM-Agent: Product-Conditioned Multimodal Policy Learning for Embodied Agents
  24. MetaOPD: Meta-Learned Token Weighting for On-Policy Distillation
  25. DIAL-OPD: Learning More from Fewer Tokens in On-Policy Distillation
  26. ReTeach: Building a Self-Teacher through Multi-Round Reflection and Retry
  27. Residual Advantage: Student-Relative Teacher Guidance for RL with Verifiable Rewards
  28. Policy Alignment: New Signals for Membership Auditing in On-Policy Distillation
  29. Environmental Feedback Modeling Matters: Rethinking Feedback Treatment in Agentic Hindsight Self-Distillation
  30. ReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery
  31. When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better
  32. Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals
  33. Stochastic Teacher Intervention for Agentic On-Policy Distillation
  34. Beyond Owls: Subliminal Learning Can Transfer Learned Capabilities and Backdoors also fits: Knowledge distillation for LLMs
  35. Recurrent Self-Improvement: Dynamic Cross-Loop On-Policy Distillation for Looped Language Models
  36. Composing What Each Teacher Learned: Multi-Teacher On-Policy Distillation through Teacher-Relative Shifts
  37. A Good Self-Teacher Meets the Student Where They Are: Joint On-Policy Learning and Teaching
  38. Learning to Act with Task Progress: Distilling Small Agents from Compact Teacher Supervision
  39. Multi-Agent Coordination via Support-Preserving Distillation
  40. SAPD: Step-Aligned Privileged Distillation
  41. On-Policy Distillation Teaches New Skills but Not New Knowledge
  42. TAP: Efficient Long-Horizon Agent Pruning via Trajectory-Anchored Recovery
  43. GraphOPD: Graph-Augmented On-Policy Distillation for LLM Agents
  44. How Much Evidence Should a Coding Agent's Self-Correction Carry? Adaptive Dirichlet Evidence for Self-Distillation
  45. Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability
  46. UP-MOPD: Update Projection in Multi-Teacher On-Policy Distillation
  47. On-Policy Distillation with Negative-Policy Rollouts
  48. Privileged Context as Drift in On-Policy Self-Distillation
  49. Does On-Policy Distillation for Safety Pose Backdoor Risks?
  50. RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents
  51. MeSD: Multi-Evidence Self-Distillation for VideoLLM
  52. Flash-OPD: Fast On-Policy Distillation
  53. Transfer-Stratified On-Policy Distillation for RL-Improved Reasoning Teachers
  54. Off-Policy Merging Beats On-Policy Self-Distillation for Continual Learning
  55. Beyond In-Distribution Preservation: Recovering Generalization in Quantized VLAs via Vulnerability-Oriented Tuning
  56. ASCENT: Online Test-Time Training of Long-Horizon Agents via Self-Distillation of Verified Experience
  57. Outcome-Guided On-Policy Self-Distillation
  58. E2-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation
  59. How Should Teachers Be Prepared? RL on Student-Induced States for On-Policy Distillation
  60. ResOPD: Tail Residualization for Sparse On-Policy Distillation
  61. DiffGate: Difficulty-Gated Teacher Guidance for On-Policy Distillation
  62. Rethinking Self-Distillation for Multi-Teacher Capability Merging
  63. Divergence controls entropy in distillation
  64. Learning from Repaired Reasoning: Root-Cause-Guided On-Policy Distillation
  65. Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective
  66. Harness-Aware Distillation for Small Language Model Agents
  67. FastOPD: On-Policy Distillation for Lightweight VLA Deployment
  68. OPD Before RL: Warm-Starting Rubric-Based RL with On-Policy Distillation
  69. Learning to Revise Reasoning with Segment-wise On-Policy Distillation
  70. Learning from Evolving Errors: Adaptive Iterative Repair for On-Policy Distillation
  71. Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation
  72. Token Utility for On-Policy Distillation via Local Trust-Region Optimization
  73. Inverse Reasoning Models for Efficient Post-Training and Verification from Answer-Only Data
  74. Latent-MOPD: Latent Multi-Teacher On-Policy Distillation
  75. Lexicographic Multi-Objective On-Policy Distillation
  76. Slow-Fast Multi-Teacher On-Policy Distillation for Capability Preservation
  77. From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation
  78. RISED: RubrIcs for agentic multi-environment Selection and sElf-Distillation
  79. SHARPO: Segment-Level Credit Assignment for Agentic Reinforcement Learning
  80. PivotOPD: Learning to Recover from Pivotal Mistakes in Multi-Turn Agents
  81. ComputerSD: Online Self-Distillation from Real-Time Feedback for Computer-Use Agents
  82. Revisiting On-policy Adversarial Black-Box Distillation: Calibrating Groupwise Reward Geometry for Effective Advantage Construction
  83. Better Supervision Is Nearby: Neighborhood On-Policy Self-Distillation
  84. Disentangling Self-Distillation: Measuring and Modeling Acquisition and Retention also fits: Knowledge distillation for LLMs
  85. From Imitation to Reward Discovery: On-Policy Warmup for Agentic RL
  86. No Task Vector Is an Island: A Comprehensive Study on the Composability of Task Vectors from On-Policy Distillation
  87. Learning Beyond Full Imitation: Task-Preserving Knowledge Distillation also fits: General knowledge distillation methods
  88. ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillation
  89. ReTaCo: Residual-Target Control for On-Policy Distillation
  90. Is Better Teacher Supervision Enough? Unlocking Student-side Learning in Multimodal On-Policy Distillation
  91. Diagnosing On-Policy Self-Distillation for Reasoning Language Models
  92. Frame Differential On-Policy Self-Distillation for Video Reasoning
  93. When Clipping Reverses Correction: Failure Dynamics of Pointwise Forward-KL On-Policy Self-Distillation
  94. Smaller Models, Better Rejects: Preference Distillation Scaling also fits: Knowledge distillation for LLMs
  95. Mitigating the Length-Scaling Tax with Online Distillation
  96. Training LLM Judges from Language Feedback via Position-Selective Self-Distillation also fits: Knowledge distillation for LLMs
  97. OPMD: On-Policy Mechanism Distillation Beyond Output Supervision
  98. The Weakest Link: Distilling LLM Reasoning with Worst-Case Constrained Reinforcement Learning
  99. Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives
  100. On the Off-Policy Teacher in On-Policy Distillation
  101. Activation-Conditioned Self-Distillation
  102. Improving OCR Faithfulness via Gated and Attenuated On-Policy Distillation also fits: Multimodal distillation
  103. AdviSD: Learning to Advise Frontier LLMs via Targeted Multi-Turn Self-Distillation
  104. Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models
  105. Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning
  106. Guide, Then Let Go: Gap-Adaptive Teacher Scheduling for Sparse-Reward Agentic RL
  107. FlowMap-OPD: Rollout--Kernel Separation for On-Policy Distillation of Few-Step Flow-Map Generators
  108. Graph-Conditioned On-Policy Agent Distillation from Off-the-Shelf Teachers
  109. From Dissonance to Orchestration: Teacher Intervention in On-Policy Distillation
  110. Beyond Prompt Count: How Data Shapes Transfer in On-Policy Distillation
  111. Solving Without Stopping: On-Policy Distillation at Small Scale
  112. Interpolated Policy Distillation: A Controllable Continuum Between Off-Policy and On-Policy Distillation
  113. Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models
  114. Learning from Think-Mode Advantage via On-Policy Distillation
  115. Beam Search as Test-Time Self-Distillation via Counterfactual Contexts
  116. On-Policy Visual Evidence Distillation also fits: Multimodal distillation
  117. SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation
  118. Know Thyself, Teach Thyself: Internal Information Flow for Selective Self-Distillation
  119. PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning
  120. PE-OPSD: Internalizing Prompt Enhancement into Flow-matching Models via On-Policy Self-Distillation also fits: Multimodal distillation
  121. Act First, Reason Later: Accelerating On-Policy Distillation for Multi-Turn Agents via Reference-Conditioned Inverse Dynamics
  122. SAKI: Maximal-Coupling-Routed Teacher Supervision for On-Policy Distillation
  123. Interactive-Policy Distillation with Bidirectional Propose-and-Verify
  124. The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation
  125. REVO: Rollout-Efficient Off-Policy Distillation via Variance-Guided Reuse
  126. ROSS: Relearning from Self-Generated Rollouts through Selective Supervision
  127. On-Policy Self-Distillation for Multi-Turn Image Editing also fits: Multimodal distillation
  128. Reward-Aligned Reweighting for On-Policy Distillation
  129. An RL View of OPD: Least Square Policy Distillation for Sample-Efficient LLM Reasoning
  130. Inductive Feedback for Mixed-Policy Distillation
  131. Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation
  132. Teacher-Student Gaps Are Not Enough: Outcome-Guided On-Policy Distillation for Multi-Turn Autonomous Agents
  133. PIVOT: Pivot-Aware On Policy Self Distillation for Multi-Turn VLM Agents
  134. Rubric-Aware On-Policy Self-Distillation for LLM Personalization
  135. Understanding On-Policy Distillation: A Mechanistic Interpretability Perspective via Sparse Crosscoders
  136. TIDE: Teacher-Student Transition via Informative Distillation and Exploration for Agentic RL
  137. When Sparse Reward Meets Dense Distillation: Training Dynamics of On-Policy Distillation
  138. Can We Trust the Teacher? Decoupled Credit Direction-Magnitude for Self-Distillation
  139. DivOPD: Spread Wide, Look Close for Asynchronous On-Policy Distillation of Multi-turn Agents
  140. UniOPSD: Unifying Outcome and Hindsight Feedback for Agentic Reinforcement Learning
  141. No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation
  142. Beyond Token Alignment: Event Completion for Cross-Tokenizer On-Policy Distillation
  143. CapField-OPD: Learning Continuous Capability Fields via Joint-Anchored Multi-Teacher On-Policy Distillation for Flow Models
  144. PMOPD: Task Ordering, Cycling, and Parameter-Update Subspace Protection in Multi-Teacher On-Policy Distillation
  145. EOPSA: Efficient On-Policy Self-Distilled Safety Alignment
  146. Unbiased Top-k Estimation for On-Policy Distillation
  147. OSPD: On-Policy Self-Distillation for Persona-Consistent Dialogue
  148. Look Before You Select: Rethinking Vocabulary Sparsification in On-Policy Distillation
  149. WAM-OPD: Sharpening World Action Models via On-Policy Distillation
  150. MAS-OPD: On-Policy Distillation for Multi-agent Systems
  151. USA: Update-aware SAM for Cross-domain On-Policy Disitllation of Language Agents
  152. PreSO: Privileged Representation Isolation for On-Policy Self-Distillation
  153. UOPD: Uncertainty-Aware Intervention for On-Policy Distillation of Multi-Turn Agents
  154. Fisher-Informed Recalibration for Feedback-Based On-Policy Self-Distillation of LLMs
  155. ChemOPD: Multi-Teacher On-Policy Distillation for Multi-Task Chemical Reasoning
  156. Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models? also fits: Knowledge distillation for LLMs
  157. DuoOPD: Learning from Joint Teacher-Student Outcomes for Multi-Task On-Policy Distillation
  158. What Shared Prefixes Hide: Trajectory Dropout for On-Policy Distillation
  159. TeacherGRPO: Closing the Capacity Gap in Reasoning Distillation via Teacher Alignment
  160. Dense Is Not Enough: Hierarchical Supervision Allocation for Long-Horizon On-Policy Distillation
  161. Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents
  162. Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoning
  163. SeOPD: Self-Evolving LLMs via Online Policy Distillation from Self-Generated Chain-of-Thought
  164. Leaky Students: Membership Inference against On-Policy Distillation
  165. Scaling Properties of Same-Family On-Policy Distillation
  166. IGSD: Environment-Verified Hindsight Self-Distillation for Search Agents
  167. Learning from a Thoughtful Teacher: Adaptive On-Policy Self-Distillation for Mathematical Reasoning
  168. Trapped by Their Own Rollouts: Understanding Aggregation--Rollout Feedback in Federated On-Policy Distillation
  169. RE-0: Verified Recursive Improvement of Embodied Code-as-Policy Agents through Local On-Policy Distillation
  170. Train4Merge: A Controlled Single-Teacher Study of RL vs. SFT Teachers for OPD-Based Model Merging
  171. Understanding the Synergy between SFT, RLVR, and OPD in LLM Post-Training
  172. TISD: On-Policy Self-Distillation with Trajectory Intervention
  173. Persistent Negatives for Adversarial Black-Box On-Policy Distillation
  174. MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillation
  175. Recursive Self-Improvement via On-Policy Distillation for Reasoning

Foundations

Most cited by papers here

  1. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models2026cited by 167 here
  2. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe2026cited by 156 here
  3. Reinforcement Learning via Self-Distillation2026cited by 128 here
  4. Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation2026cited by 103 here
  5. Entropy-Aware On-Policy Distillation of Language Models2026cited by 103 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 284 here
  2. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 38 here
  3. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 36 here
  4. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 167 here
  5. 2026Self-Distillation Enables Continual Learningcited by 89 here
  6. 2026Reinforcement Learning via Self-Distillationcited by 128 here
  7. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 103 here
  8. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 90 here
  9. 2026Self-Distilled RLVRcited by 61 here
  10. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 156 here
  11. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 27 here
13 more it builds on, not in this chain
  1. 2025Black-Box On-Policy Distillation of Large Language Modelscited by 29 here
  2. 2026Stable On-Policy Distillation through Adaptive Target Reformulationcited by 35 here
  3. 2026Privileged Information Distillation for Language Modelscited by 45 here
  4. 2026On-Policy Context Distillation for Language Modelscited by 88 here
  5. 2026Fast and Effective On-policy Distillation from Reasoning Prefixescited by 31 here
  6. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 103 here
  7. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 55 here
  8. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?cited by 64 here
  9. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 90 here
  10. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 40 here
  11. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agentscited by 30 here
  12. 2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 37 here
  13. 2026TIP: Token Importance in On-Policy Distillationcited by 56 here

The 24 most cited of 423.

Browse

30 of 446, ranked 6–35

  1. 6Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes202690
  2. 7A Survey of On-Policy Distillation for Large Language Models202690
  3. 8Self-Distillation Enables Continual Learning202689
  4. 9On-Policy Context Distillation for Language Models202688
  5. 10Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?202664
  6. 11Self-Distilled RLVR202661
  7. 12TIP: Token Importance in On-Policy Distillation202656
  8. 13Scaling Reasoning Efficiently via Relaxed On-Policy Distillation202655
  9. 14Privileged Information Distillation for Language Models202645
  10. 15Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing202640
All 30
  1. 16Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision202637
  2. 17Stable On-Policy Distillation through Adaptive Target Reformulation202635
  3. 18Fast and Effective On-policy Distillation from Reasoning Prefixes202631
  4. 19Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents202630
  5. 20Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe202627
  6. 21SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting202626
  7. 22HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation202624
  8. 23Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation202624
  9. 24Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation202622
  10. 25Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation202622
  11. 26GATES: Self-Distillation under Privileged Context with Consensus Gating202621
  12. 27TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents202621
  13. 28MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training202621
  14. 29The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes202619
  15. 30RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation202619
  16. 31Reinforcement-aware Knowledge Distillation for LLM Reasoning202618
  17. 32OPSDL: On-Policy Self-Distillation for Long-Context Language Models202618
  18. 33Self-Distilled Agentic Reinforcement Learning202618
  19. 34When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning202618
  20. 35Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillation202618