mlsanityData to 7 Oct 2026

HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation

arXiv 2603.23871 ↗First posted 25 Mar 2026cs.LG cs.AIcited by 39

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Large language models trained with reinforcement learning (RL) for mathematical reasoning face a fundamental challenge: on problems the model cannot solve at all - "cliff" prompts - the RL gradient vanishes entirely, preventing any learning signal from reaching these failure modes. We introduce Hybrid Distillation Policy Optimization (HDPO), which augments standard RL with privileged self-distillation targeting cliff prompts. On each training step, HDPO identifies prompts where all rollouts fail, generates privileged rollouts by providing the model with ground-truth information, filters for correct solutions, and distills the teacher's token-level distribution into the student. Because teacher and student share the same weights - differing only in their input - the realizability gap is provably bounded, unlike cross-model distillation. We prove that R=1 filtered privileged generation recovers the optimal KL-regularized RL policy in the hard-threshold limit. Experiments on OpenMathInstruct-2 with Qwen2.5-Math-1.5B-Instruct show that HDPO consistently improves coverage metrics (pass@4 by +0.8-1.1%, pass@8 by +0.4-1.7%) while maintaining greedy accuracy, with the distillation weight lambda providing direct control over the exploration-exploitation tradeoff.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 33 references that are in the map. First posted on arXiv 25 Mar 2026.

Builds on

28 of 33 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning22 Oct
  2. 2025Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts10 Oct
  3. 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping26 Sep
  4. 2025VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models24 Sep
  5. 2025EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learning11 Aug
  6. 2025RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning10 Jul
  7. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions9 Jun
  8. 2025KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning2 Jun
  9. 2025Improving RL Exploration for LLM Reasoning through Retrospective Replay19 Apr
  10. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  11. 2025Process Reinforcement through Implicit Rewards3 Feb
  12. 2023Let's Verify Step by Step31 May
2 more
  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 14

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  6. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  7. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  8. 2026Reinforcement Learning via Self-Distillationcited by 285
  9. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 194
  10. 2020The Lipschitz Constant of Self-Attentioncited by 171
  11. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  12. 2026Privileged Information Distillation for Language Modelscited by 79
2 more
  1. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 36
  2. 2026Reinforcement-aware Knowledge Distillation for LLM Reasoningcited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 39.

39 papers.

Sort:

By area:

  1. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
  2. 2026Self-Distilled Agentic Reinforcement Learningcited by 49
  3. 2026Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillationcited by 31
  4. 2026Rethinking On-Policy Self-Distillation for Thinking Modelscited by 28
  5. 2026Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoningcited by 23
  6. 2026GenEvolve: Self-Evolving Image Generation Agents via Tool-Orchestrated Visual Experience Distillationcited by 15
  7. 2026OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoningcited by 14
  8. 2026Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillationcited by 13
  9. 2026Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularizationcited by 10
  10. 2026Adaptive Teacher Exposure for Self-Distillation in LLM Reasoningcited by 9
  11. 2026OPRD: On-Policy Representation Distillationcited by 9
  12. 2026Multi-Rollout On-Policy Distillation via Peer Successes and Failurescited by 8
27 more
  1. 2026On-Policy Distillation with Best-of-N Teacher Rollout Selectioncited by 7
  2. 2026CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimizationcited by 7
  3. 2026H-OPD: Confidence Aware Heterogeneous Multi-Teacher Multimodal On-policy Distillationcited by 7
  4. 2026Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVRcited by 7
  5. 2026OPSD Compresses What RLVR Teaches: A Post-RL Compaction Stage for Reasoning Modelscited by 6
  6. 2026DemoPSD: Disagreement-Modulated Policy Self-Distillationcited by 6
  7. 2026Skill0.5: Joint Skill Internalization and Utilization for Out-of-Distribution Generalization in Agentic Reinforcement Learningcited by 5
  8. 2026A Formula-Driven Survey and Research Agenda for On-Policy Distillationcited by 5
  9. 2026Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewardscited by 4
  10. 2026AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signalscited by 3
  11. 2026CAST: Non-Privileged Clipped Asymmetric Self-Teaching with Advantage Flipping for GRPOcited by 3
  12. 2026SocraticPO: Policy Optimization via Interactive Guidancecited by 3
  13. 2026Restoring the Sweet Spot: Pass-Rate Weighted Self-Distillation for LLM Reasoningcited by 2
  14. 2026Self-Distillation Policy Optimization via Visual Feedback: Bridging Code and Visual Artifactscited by 1
  15. 2026A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensionscited by 1
  16. 2026RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learningcited by 1
  17. 2026Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimizationcited by 0
  18. 2026ClassicLogic: A Knowledge-Driven Benchmark of Classic Puzzle Games for Evaluating Compositional Generalizationcited by 0
  19. 2026LoRA Scaffolded Policy Optimization (LSPO): A Sampling-Time Low-Rank Scaffold for Recovering Reinforcement-Learning Gradient on Zero-Reward Cliff Promptscited by 0
  20. 2026Hybrid-Policy Self-Editing for Composable Unstructured Knowledge Editingcited by 0
  21. 2026CataOPD: Catalytic On-Policy Distillation for Large Language Model Reasoningcited by 0
  22. 2026SeOPD: Self-Evolving LLMs via Online Policy Distillation from Self-Generated Chain-of-Thoughtcited by 0
  23. 2026Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoningcited by 0
  24. 2026When Sparse Reward Meets Dense Distillation: Training Dynamics of On-Policy Distillationcited by 0
  25. 2026Guide, Then Let Go: Gap-Adaptive Teacher Scheduling for Sparse-Reward Agentic RLcited by 0
  26. 2026A Good Self-Teacher Meets the Student Where They Are: Joint On-Policy Learning and Teachingcited by 0
  27. 2026PIVOT: Perplexity-Informed KD-to-RL Transition Scheduling for Vertical-Domain Few-Shot Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026β-OPSD: Deriving with Policy Optimization, Training with Self-Distillation
  2. 2026SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation
  3. 2026Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergence
  4. 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation
  5. 2026SAPD: Step-Aligned Privileged Distillation
  6. 2026Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillation
  7. 2026Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models
  8. 2026Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation