mlsanityData to 14 Oct 2026

Self-Distilled RLVR

arXiv 2604.03128 ↗First posted 3 Apr 2026cs.LG cs.CLcited by 131

On the map: Knowledge distillation → On-policy and self-distillation for LLMs

Abstract (arXiv)

On-policy distillation (OPD) has become a popular training paradigm in the LLM community. This paradigm selects a larger model as the teacher to provide dense, fine-grained signals for each sampled trajectory, in contrast to reinforcement learning with verifiable rewards (RLVR), which only obtains sparse signals from verifiable outcomes in the environment. Recently, the community has explored on-policy self-distillation (OPSD), where the same model serves as both teacher and student, with the teacher receiving additional privileged information such as reference answers to enable self-evolution. This paper demonstrates that learning signals solely derived from the privileged teacher result in severe information leakage and unstable long-term training. Accordingly, we identify the optimal niche for self-distillation and propose RLSD (RLVR with Self-Distillation). Specifically, we leverage self-distillation to obtain token-level policy differences for determining fine-grained update magnitudes, while continuing to use RLVR to derive reliable update directions from environmental feedback (e.g., response correctness). This enables RLSD to simultaneously harness the strengths of both RLVR and OPSD, achieving a higher convergence ceiling and superior training stability.

Key facts. Cited by 131 papers in the map (data through 14 Oct 2026). Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 30 of its 35 references that are in the map. First posted on arXiv 3 Apr 2026.

Builds on

30 of 35 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes26 Mar
  2. 2026On-Policy Context Distillation for Language Models12 Feb
  3. 2026Privileged Information Distillation for Language Models4 Feb
  4. 2026Self-Distillation Enables Continual Learning27 Jan
  5. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models26 Jan

Other references · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Let's Verify Step by Stepcited by 3,789
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  6. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  7. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  8. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
  9. 2025Process Reinforcement through Implicit Rewardscited by 504
  10. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  11. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  12. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 286
13 more
  1. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
  2. 2025Dynamic Early Exit in Reasoning Modelscited by 219
  3. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimizationcited by 89
  4. 2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Modelscited by 87
  5. 2024Step-level Value Preference Optimization for Mathematical Reasoningcited by 72
  6. 2025ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Modelscited by 58
  7. 2025Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoningcited by 34
  8. 2026MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methodscited by 30
  9. 2025Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimizationcited by 25
  10. 2025KTAE: A Model-Free Algorithm to Key-Tokens Advantage Estimation in Mathematical Reasoningcited by 19
  11. 2026Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoningcited by 19
  12. 2025Test-time Prompt Interventioncited by 12
  13. 2025Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 131.

131 papers.

Sort:

By area:

  1. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 299
  2. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 49
  3. 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillationcited by 45
  4. 2026Rethinking On-Policy Self-Distillation for Thinking Modelscited by 39
  5. 2026Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillationcited by 32
  6. 2026OPID: On-Policy Skill Distillation for Agentic Reinforcement Learningcited by 32
  7. 2026Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoningcited by 23
  8. 2026D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Modelscited by 22
  9. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 21
  10. 2026Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillationcited by 18
  11. 2026MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debatecited by 17
  12. 2026On-Policy Self-Distillation without Any Supervisioncited by 17
38 more
  1. 2026STEPS: Selective On-Policy Self-Distillation for Reasoningcited by 16
  2. 2026OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoningcited by 16
  3. 2026Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVRcited by 15
  4. 2026On-policy Distillation with Verifiable Rewardcited by 15
  5. 2026Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Informationcited by 14
  6. 2026Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularizationcited by 12
  7. 2026GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillationcited by 12
  8. 2026ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillationcited by 12
  9. 2026UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillationcited by 12
  10. 2026DOPD: Dual On-policy Distillationcited by 12
  11. 2026Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Groundingcited by 11
  12. 2026Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Trainingcited by 11
  13. 2026OPRD: On-Policy Representation Distillationcited by 11
  14. 2026V-Zero: Answer-Label-Free On-Policy Distillation with Contrastive Evidence Gating for Fine-Grained Visual Reasoningcited by 11
  15. 2026Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvementcited by 11
  16. 2026VISD: Enhancing Video Reasoning via Structured Self-Distillationcited by 10
  17. 2026Training with Harnesses: On-Policy Harness Self-Distillation for Complex Reasoningcited by 10
  18. 2026Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoningcited by 10
  19. 2026Reinforcement Learning from Rich Feedback with Distributional DAggercited by 10
  20. 2026On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversitycited by 10
  21. 2026Visual Contrastive Self-Distillationcited by 10
  22. 2026Is More Privileged Information Better? From Solution Traces to Problem-Solving Structure in Self-Distilled Reasoningcited by 9
  23. 2026Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillationcited by 8
  24. 2026CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimizationcited by 8
  25. 2026Self-Distilled Policy Gradientcited by 8
  26. 2026On-Policy Distillation with Best-of-N Teacher Rollout Selectioncited by 7
  27. 2026Learning from Language Feedback via Variational Policy Distillationcited by 7
  28. 2026H2SD: Hybrid Hindsight Self-Distillationcited by 7
  29. 2026OPSD Compresses What RLVR Teaches: A Post-RL Compaction Stage for Reasoning Modelscited by 6
  30. 2026EDGE-OPD: Internalizing Privileged Context with Evidence Guided On-Policy Distillationcited by 6
  31. 2026PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignmentcited by 6
  32. 2026Procedural Memory Distillation: Online Reflection for Self-Improving Language Modelscited by 6
  33. 2026DemoPSD: Disagreement-Modulated Policy Self-Distillationcited by 6
  34. 2026Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillationcited by 5
  35. 2026Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillationcited by 5
  36. 2026RISE: Recursive Improvement via Self-Extrapolating Policy Distillationcited by 5
  37. 2026Healthcare AI GYM for Medical Agentscited by 4
  38. 2026OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoningcited by 4

50 most cited of 131 All 131 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Outcome-Guided On-Policy Self-Distillation
  2. 2026SR-OPSD: Self-Referenced On-Policy Self-Distillation
  3. 2026Reinforcement Learning via Self-Distillation
  4. 2026A Predictive Law for On-Policy Self-Distillation From World Feedback
  5. 2026Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals
  6. 2026Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR
  7. 2026Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance
  8. 2026On the Geometry of On-Policy Distillation