mlsanityData to 7 Oct 2026

Self-Distilled Policy Gradient

arXiv 2606.04036 ↗First posted 2 Jun 2026cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

On-policy self-distillation, where a language model conditions on privileged context to supervise its own generations, is a promising source of dense supervision for sparse-reward reinforcement learning. Actually, it can be instantiated as an auxiliary full-vocabulary student-to-teacher reverse Kullback-Leibler divergence loss. We therefore propose SDPG, a self-distilled policy-gradient framework that combines group-relative verifier advantages with normalized standard deviation, exact full-vocabulary on-policy self-distillation, as well as reference-policy KL regularization. Empirically, SDPG improves stability and performance over RLVR and self-distillation baselines. The code is available at https://github.com/lauyikfung/SDPG.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 39 of its 46 references that are in the map. First posted on arXiv 2 Jun 2026.

Builds on

39 of 46 references.

References · 39

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  8. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  9. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  10. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  11. 2025Group Sequence Policy Optimizationcited by 788
  12. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
27 more
  1. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  2. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  3. 2015Policy Distillationcited by 414
  4. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  5. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
  6. 2026Reinforcement Learning via Self-Distillationcited by 285
  7. 2025Reasoning with Exploration: An Entropy Perspectivecited by 277
  8. 2026Self-Distillation Enables Continual Learningcited by 212
  9. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  10. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  11. 2026MiMo-V2-Flash Technical Reportcited by 163
  12. 2026On-Policy Context Distillation for Language Modelscited by 161
  13. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 122
  14. 2026Self-Distilled RLVRcited by 109
  15. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learningcited by 88
  16. 2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Modelscited by 87
  17. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimizationcited by 87
  18. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 85
  19. 2026Privileged Information Distillation for Language Modelscited by 79
  20. 2024Step-level Value Preference Optimization for Mathematical Reasoningcited by 71
  21. 2025Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoningcited by 34
  22. 2025On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoningcited by 30
  23. 2026Reinforcement-aware Knowledge Distillation for LLM Reasoningcited by 29
  24. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Modelscited by 19
  25. 2026Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoningcited by 18
  26. 2023Instruction Tuning with Human Curriculumcited by 15
  27. 2025Test-time Prompt Interventioncited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026CriPO: Enhancing Rubric-based RL via Self-Distillationcited by 4
  2. 2026Geometric Self-Distillation for Reasoning Generalizationcited by 2
  3. 2026β-OPSD: Deriving with Policy Optimization, Training with Self-Distillationcited by 2
  4. 2026TTPO: Test-Time Policy Optimizationcited by 1
  5. 2026UniOPSD: Unifying Outcome and Hindsight Feedback for Agentic Reinforcement Learningcited by 0
  6. 2026MIRROR: From Imitation to Internalization in LLM Personalizationcited by 0