mlsanityData to 7 Oct 2026

RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation

arXiv 2606.11709 ↗First posted 10 Jun 2026cs.LG cs.CLcited by 33

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

On-policy self-distillation (OPSD) provides dense, token-level supervision for reasoning models by aligning a model's own distribution with that under privileged context, typically a verified solution. However, we show that the resulting distributional gap concentrates on style tokens rather than task-bearing ones, as the hinted model tends to produce shorter, more direct outputs. We term this pathology privilege-induced style drift, which can destabilize training and shorten responses. To address this, we propose RLCSD (Reinforcement Learning with Contrastive on-policy Self-Distillation), which mitigates this drift by contrasting the teacher-student gap under a correct hint against that under a wrong hint, suppressing style shifts induced by hints regardless of correctness and yielding a signal more concentrated on task-bearing tokens. Experiments on Qwen3 (1.7B/4B/8B) and Olmo-3-7B-Think across mathematical and logical reasoning show that RLCSD consistently outperforms GRPO and prior OPSD methods, with additional results on agentic tasks supporting broader applicability. We further show that the contrastive principle is general: it plugs into existing OPSD methods to improve them, and its underlying insight extends to broader cross-model on-policy distillation.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 41 of its 49 references that are in the map. First posted on arXiv 10 Jun 2026.

Builds on

41 of 49 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning13 May
  2. 2026Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information12 May
  3. 2026Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning12 May
  4. 2026Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR11 May
  5. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe14 Apr
  6. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation14 Apr
  7. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models26 Jan

Other references · 34

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  8. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  9. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  10. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  11. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  12. 2025Group Sequence Policy Optimizationcited by 788
22 more
  1. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  2. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  3. 2026Reinforcement Learning via Self-Distillationcited by 285
  4. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  5. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 260
  6. 2026MiMo-V2-Flash Technical Reportcited by 163
  7. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 148
  8. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 145
  9. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 122
  10. 2026Self-Distilled RLVRcited by 109
  11. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 65
  12. 2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 58
  13. 2025Black-Box On-Policy Distillation of Large Language Modelscited by 53
  14. 2026The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligencecited by 50
  15. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixescited by 37
  16. 2026STEPS: Selective On-Policy Self-Distillation for Reasoningcited by 15
  17. 2025d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Modelscited by 14
  18. 2026Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularizationcited by 10
  19. 2026From Generic Correlation to Input-Specific Credit in On-Policy Self Distillationcited by 9
  20. 2026CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimizationcited by 7
  21. 2026UniSD: Towards a Unified Self-Distillation Framework for Large Language Modelscited by 6
  22. 2026Self-Policy Distillation via Capability-Selective Subspace Projectioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 33.

33 papers.

Sort:

By area:

  1. 2026Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loopscited by 25
  2. 2026Weak-to-Strong On-Policy Distillationcited by 7
  3. 2026Agentic Reinforcement Learning with Observation-Calibrated Self-Distillationcited by 6
  4. 2026Latent On-Policy Self-Distillationcited by 6
  5. 2026CriPO: Enhancing Rubric-based RL via Self-Distillationcited by 4
  6. 2026Group-Reflective Self-Distillation for Agentic Reinforcement Learningcited by 4
  7. 2026SAF-OPD: Stable Advantage Fusion for On-Policy Distillationcited by 4
  8. 2026Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillationcited by 3
  9. 2026Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillationcited by 3
  10. 2026DRIFT: Difficulty Routing Self-DIstillation with Rhythm-Gated Exploration and Success BuFfer Trainingcited by 2
  11. 2026Trust Is Not Enough: Influence Calibration for On-Policy Self-Distillation in Agentic RLcited by 2
  12. 2026Contrastive Reinforced Policy Optimization via Privileged Self-Distillationcited by 1
21 more
  1. 2026Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agentscited by 1
  2. 2026Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillationcited by 1
  3. 2026Negative Self-Distillation: Learning to Reason by Avoiding Flawscited by 1
  4. 2026RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learningcited by 1
  5. 2026On Repulsive and Attractive Teachers: Separating Correctness from Behavior in Self-Distillationcited by 1
  6. 2026The Hitchhiker's Guide to Agentic AI: From Foundations to Systemscited by 0
  7. 2026Not All Tokens Deserve Equal Credit: Counterfactual Sensitivity Credit Reallocation for Long-CoT Reasoningcited by 0
  8. 2026VERPO: Verified Evidence Regularized Policy Optimizationcited by 0
  9. 2026Beyond Verified Answers: Solver-Informed Self-Distillation for Bootstrapping Operations Research Language Modelscited by 0
  10. 2026When and Where to Trust the Teacher: Unifying On-Policy Distillation and GRPO through Entropy-Calibrated Credit Assignmentcited by 0
  11. 2026CataOPD: Catalytic On-Policy Distillation for Large Language Model Reasoningcited by 0
  12. 2026Recursive Self-Improvement via On-Policy Distillation for Reasoningcited by 0
  13. 2026Instruct, Not Answer: Using Instruction Privileges in On-Policy Context Distillationcited by 0
  14. 2026Inductive Feedback for Mixed-Policy Distillationcited by 0
  15. 2026SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillationcited by 0
  16. 2026Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVRcited by 0
  17. 2026AdviSD: Learning to Advise Frontier LLMs via Targeted Multi-Turn Self-Distillationcited by 0
  18. 2026Activation-Conditioned Self-Distillationcited by 0
  19. 2026When Clipping Reverses Correction: Failure Dynamics of Pointwise Forward-KL On-Policy Self-Distillationcited by 0
  20. 2026Diagnosing On-Policy Self-Distillation for Reasoning Language Modelscited by 0
  21. 2026ReTeach: Building a Self-Teacher through Multi-Round Reflection and Retrycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Outcome-Guided On-Policy Self-Distillation
  2. 2026β-OPSD: Deriving with Policy Optimization, Training with Self-Distillation
  3. 2026Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR
  4. 2026PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning
  5. 2026Contrastive On-Policy Distillation
  6. 2026Reward-Gated On-Policy Distillation
  7. 2026HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation
  8. 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why