mlsanityData to 14 Oct 2026

Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing

arXiv 2604.02288 ↗First posted 2 Apr 2026cs.LG cs.AIcited by 76

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for post-training large language models. While Group Relative Policy Optimization (GRPO) is widely adopted, its coarse credit assignment uniformly penalizes failed rollouts, lacking the token-level focus needed to efficiently address specific deviations. Self-Distillation Policy Optimization (SDPO) addresses this by providing denser, more targeted logit-level supervision that facilitates rapid early improvement, yet it frequently collapses during prolonged training. We trace this late-stage instability to two intrinsic flaws: self-distillation on already-correct samples introduces optimization ambiguity, and the self-teacher's signal reliability progressively degrades. To resolve these issues, we propose Sample-Routed Policy Optimization (SRPO), a unified on-policy framework that routes correct samples to GRPO's reward-aligned reinforcement and failed samples to SDPO's targeted logit-level correction. SRPO further incorporates an entropy-aware dynamic weighting mechanism to suppress high-entropy, unreliable distillation targets while emphasizing confident ones. Evaluated across five benchmarks and two model scales, SRPO achieves both the rapid early improvement of SDPO and the long-horizon stability of GRPO. It consistently surpasses the peak performance of both baselines, raising the five-benchmark average on Qwen3-8B by 3.4% over GRPO and 6.3% over SDPO, while simultaneously yielding moderate response lengths and lowering per-step compute cost by up to 17.2%.

Key facts. Cited by 76 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 37 of its 44 references that are in the map. First posted on arXiv 2 Apr 2026.

Builds on

37 of 44 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026Execution-Grounded Credit Assignment for GRPO in Code Generation17 Mar
  2. 2026Beyond Uniform Credit: Causal Credit Assignment for Policy Optimization10 Feb
  3. 2025Semantic Soft Bootstrapping: Long Context Reasoning in LLMs without Reinforcement Learning4 Dec
  4. 2025GRPO-λ: Credit Assignment improves LLM Reasoning30 Sep
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  7. 2025Process Reinforcement through Implicit Rewards3 Feb
  8. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  9. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
  10. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search6 Jun
  11. 2023Let's Verify Step by Step31 May
  12. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 25

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  8. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  9. 2025Group Sequence Policy Optimizationcited by 835
  10. 2016Sequence-Level Knowledge Distillationcited by 801
  11. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  12. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
13 more
  1. 2026Reinforcement Learning via Self-Distillationcited by 332
  2. 2026Self-Distillation Enables Continual Learningcited by 247
  3. 2023Inverse Scaling: When Bigger Isn't Bettercited by 207
  4. 2026On-Policy Context Distillation for Language Modelscited by 186
  5. 2022Learning by Distilling Contextcited by 138
  6. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?cited by 103
  7. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 101
  8. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 90
  9. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 53
  10. 2025Scaling Laws Are Unreliable for Downstream Tasks: A Reality Checkcited by 29
  11. 2026Aligning Language Models from User Interactionscited by 28
  12. 2025Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learningcited by 9
  13. 2025TDRM: Smooth Reward Models with Temporal Difference for LLM RL and Inferencecited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 76.

76 papers.

Sort:

By area:

  1. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 299
  2. 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillationcited by 45
  3. 2026Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoningcited by 27
  4. 2026On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agentscited by 27
  5. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 21
  6. 2026ROSD: Reflective On-Policy Self-Distillation for Language Model Reasoning across Domainscited by 17
  7. 2026STEPS: Selective On-Policy Self-Distillation for Reasoningcited by 16
  8. 2026OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoningcited by 16
  9. 2026Trajectory-Refined Distillationcited by 16
  10. 2026Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVRcited by 15
  11. 2026On-policy Distillation with Verifiable Rewardcited by 15
  12. 2026Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Informationcited by 14
38 more
  1. 2026Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularizationcited by 12
  2. 2026SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillationcited by 12
  3. 2026OPRD: On-Policy Representation Distillationcited by 11
  4. 2026Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidancecited by 11
  5. 2026Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVRcited by 11
  6. 2026VISD: Enhancing Video Reasoning via Structured Self-Distillationcited by 10
  7. 2026Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoningcited by 10
  8. 2026Reinforcement Learning from Rich Feedback with Distributional DAggercited by 10
  9. 2026Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillationcited by 8
  10. 2026AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signalscited by 7
  11. 2026H2SD: Hybrid Hindsight Self-Distillationcited by 7
  12. 2026A Formula-Driven Survey and Research Agenda for On-Policy Distillationcited by 6
  13. 2026DemoPSD: Disagreement-Modulated Policy Self-Distillationcited by 6
  14. 2026CriPO: Enhancing Rubric-based RL via Self-Distillationcited by 5
  15. 2026Healthcare AI GYM for Medical Agentscited by 4
  16. 2026SocraticPO: Policy Optimization via Interactive Guidancecited by 4
  17. 2026LLM-as-a-Coach: Experiential Learning for Non-Verifiable Taskscited by 4
  18. 2026When Context Returns: Toward Robust Internalization in On-Policy Distillationcited by 3
  19. 2026The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillationcited by 3
  20. 2026Scaling Scientific Discovery Environments for Turn-Level Agentic RLcited by 3
  21. 2026PaperGym: Rubric-Centered Evolution for Research-Plan Generationcited by 3
  22. 2026AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignmentcited by 2
  23. 2026Restoring the Sweet Spot: Pass-Rate Weighted Self-Distillation for LLM Reasoningcited by 2
  24. 2026EST-PRM: Stress-Testing Process Reward Models Before They Become Load-Bearingcited by 2
  25. 2026DRIFT: Difficulty Routing Self-DIstillation with Rhythm-Gated Exploration and Success BuFfer Trainingcited by 2
  26. 2026Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agentscited by 2
  27. 2026TTPO: Test-Time Policy Optimizationcited by 2
  28. 2026When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillationcited by 2
  29. 2026Physics-Guided Policy Optimization with Self-Distillationcited by 1
  30. 2026Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimizationcited by 1
  31. 2026TurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated Reasoningcited by 1
  32. 2026On Repulsive and Attractive Teachers: Separating Correctness from Behavior in Self-Distillationcited by 1
  33. 2026PathRouter: Aligning Rewards with Retrieval Quality in Agentic Graph Retrieval-Augmented Generationcited by 0
  34. 2026OptiAgent: End-to-End Optimization Modeling via Multi-Agent Iterative Refinementcited by 0
  35. 2026AdaStop: Cost-Aware Early Stopping for DNN Test Selectioncited by 0
  36. 2026EvoSQL: Memory-Augmented Critic-Generator Co-Evolution for Text-to-SQLcited by 0
  37. 2026RoCo-ACE: Rollout-Conditioned Online Distillation for Retention-Aware Knowledge Injectioncited by 0
  38. 2026Not All Tokens Deserve Equal Credit: Counterfactual Sensitivity Credit Reallocation for Long-CoT Reasoningcited by 0

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimization
  2. 2026Self-Distilled Policy Gradient
  3. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
  4. 2026SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation
  5. 2025SSPO: Subsentence-level Policy Optimization
  6. 2026EP-GRPO: Entropy-Progress Aligned Group Relative Policy Optimization with Implicit Process Guidance
  7. 2026iGRPO: Self-Feedback-Driven LLM Reasoning
  8. 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization