mlsanityData to 7 Oct 2026

Reinforcement Learning via Self-Distillation

arXiv 2601.20802 ↗First posted 28 Jan 2026cs.LG cs.AIcited by 285

Not on the map. Why?

Cited by 88 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models are increasingly post-trained with reinforcement learning in verifiable domains such as code and math. Yet, current methods for reinforcement learning with verifiable rewards (RLVR) learn only from a scalar outcome reward per attempt, creating a severe credit-assignment bottleneck. Many verifiable environments actually provide rich textual feedback, such as runtime errors or judge evaluations, that explain why an attempt failed. We formalize this setting as reinforcement learning with rich feedback and introduce Self-Distillation Policy Optimization (SDPO), which converts tokenized feedback into a dense learning signal without any external teacher or explicit reward model. SDPO treats the current model conditioned on feedback as a self-teacher and distills its feedback-informed next-token predictions back into the policy. In this way, SDPO leverages the model's ability to retrospectively identify its own mistakes in-context. Across scientific reasoning, tool use, and competitive programming on LiveCodeBench v6, SDPO improves sample efficiency and final accuracy over strong RLVR baselines. Notably, SDPO also outperforms baselines in standard RLVR environments that only return scalar feedback by using successful rollouts as implicit feedback for failed attempts. Finally, applying SDPO to individual questions at test time accelerates discovery on difficult binary-reward tasks, achieving the same discovery probability as best-of-k sampling or multi-turn conversations with 3x fewer attempts.

Key facts. Cited by 285 papers in the map (data through 7 Oct 2026). Builds on 110 of its 137 references that are in the map. First posted on arXiv 28 Jan 2026.

Builds on

110 of 137 references.

References · 110

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  7. 2025Qwen3 Technical Reportcited by 8,195
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  11. 2024Qwen2.5 Technical Reportcited by 4,713
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
38 more
  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  4. 2023Let's Verify Step by Stepcited by 3,598
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  8. 2015Trust Region Policy Optimizationcited by 2,503
  9. 2021Convex optimizationcited by 2,479
  10. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  11. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  12. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  13. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  14. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  15. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  16. 2025s1: Simple test-time scalingcited by 1,311
  17. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  18. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  19. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  20. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  21. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  22. 2017Hindsight Experience Replaycited by 1,096
  23. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  24. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  25. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  26. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  27. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  28. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  29. 2025Group Sequence Policy Optimizationcited by 788
  30. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 763
  31. 2016Sequence-Level Knowledge Distillationcited by 761
  32. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  33. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  34. 2014Convex Optimization: Algorithms and Complexitycited by 569
  35. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  36. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  37. 2025Process Reinforcement through Implicit Rewardscited by 478
  38. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466

50 most cited of 110 All 110 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 285.

285 papers.

Sort:

By area:

  1. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
  2. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  3. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 122
  4. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?cited by 87
  5. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 83
  6. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agentscited by 75
  7. 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillationcited by 71
  8. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 65
  9. 2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 58
  10. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  11. 2026CRISP: Compressed Reasoning via Iterative Self-Policy Distillationcited by 49
  12. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 43
38 more
  1. 2026HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillationcited by 39
  2. 2026SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weightingcited by 38
  3. 2026Embarrassingly Simple Self-Distillation Improves Code Generationcited by 36
  4. 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillationcited by 33
  5. 2026Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillationcited by 31
  6. 2026OPSDL: On-Policy Self-Distillation for Long-Context Language Modelscited by 29
  7. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationcited by 28
  8. 2026Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learningcited by 28
  9. 2026Rethinking On-Policy Self-Distillation for Thinking Modelscited by 28
  10. 2026Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Modelscited by 24
  11. 2026KL for a KL: On-Policy Distillation with Control Variate Baselinecited by 24
  12. 2026SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learningcited by 23
  13. 2026Aligning Language Models from User Interactionscited by 21
  14. 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Whycited by 21
  15. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 17
  16. 2026ROSD: Reflective On-Policy Self-Distillation for Language Model Reasoning across Domainscited by 17
  17. 2026STEPS: Selective On-Policy Self-Distillation for Reasoningcited by 15
  18. 2026GenEvolve: Self-Evolving Image Generation Agents via Tool-Orchestrated Visual Experience Distillationcited by 15
  19. 2026OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoningcited by 14
  20. 2026Are Full Rollouts Necessary for On-Policy Distillation?cited by 14
  21. 2026Trajectory-Refined Distillationcited by 14
  22. 2026Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVRcited by 13
  23. 2026MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debatecited by 12
  24. 2026Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Informationcited by 12
  25. 2026ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillationcited by 12
  26. 2026OPSD-V: On-Policy Self-Distillation for Post-Training Few-Step Autoregressive Video Generatorscited by 12
  27. 2026Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillationcited by 12
  28. 2026An Imperfect Verifier is Good Enough: Learning with Noisy Rewardscited by 11
  29. 2026Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillationcited by 11
  30. 2026Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillationcited by 11
  31. 2026SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillationcited by 11
  32. 2026Reward-Gated On-Policy Distillationcited by 11
  33. 2026On-Policy Self-Distillation without Any Supervisioncited by 11
  34. 2026Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularizationcited by 10
  35. 2026Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillationcited by 10
  36. 2026V-Zero: Answer-Label-Free On-Policy Distillation with Contrastive Evidence Gating for Fine-Grained Visual Reasoningcited by 10
  37. 2026Pass the Baton: Trajectory-Relayed On-Policy Distillationcited by 10
  38. 2026On-policy Distillation with Verifiable Rewardcited by 10

50 most cited of 285 All 285 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SVR: Self-Verifying Refinement via Joint Verdict-Confidence Reinforcement Learning for Adaptive Test-Time Compute
  2. 2026Verifier-Induced Support Reshaping in On-Policy Optimization
  3. 2025Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards
  4. 2026I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimization
  5. 2026From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement
  6. 2025LaSeR: Reinforcement Learning with Last-Token Self-Rewarding
  7. 2025Reinforcement Learning vs. Distillation: Understanding Accuracy and Capability in LLM Reasoning
  8. 2026Self-Review Reinforcement Learning (SRRL) with Cross-Episode Memory and Policy Distillation