mlsanityData to 14 Oct 2026

Reinforcement Learning vs. Distillation: Understanding Accuracy and Capability in LLM Reasoning

arXiv 2505.14216 ↗First posted 20 May 2025cs.AI cs.CLcited by 14

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Recent studies have shown that reinforcement learning with verifiable rewards (RLVR) enhances overall accuracy (pass@1) but often fails to improve capability (pass@k) of LLMs in reasoning tasks, while distillation can improve both. In this paper, we investigate the mechanisms behind these phenomena. First, we demonstrate that RLVR struggles to improve capability as it focuses on improving the accuracy of the easier questions to the detriment of the accuracy of the most difficult questions. Second, we show that RLVR does not merely increase the success probability for the easier questions, but in our small model settings, produces quality responses that were absent in its original output distribution. In addition, we show these responses are neither noticeably longer nor feature more reflection-related keywords, underscoring the need for more reliable indicators of response quality. Third, from the experiment distilling teacher responses to in-distribution problems, we find that capability does not always improve with distillation. We conjecture that capability improves only when new knowledge is introduced, whereas distilling reasoning patterns only improves accuracy but not capability, sacrificing performance on the most difficult questions, similar to RLVR. Together, these findings offer a clearer understanding of how RLVR and distillation shape reasoning behavior in LLMs

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 46 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

33 of 46 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  2. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  3. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  4. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  5. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  6. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  7. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  9. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  10. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
  11. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  12. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
1 more
  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov

Other references · 20

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2025s1: Simple test-time scalingcited by 1,341
  6. 2024OpenAI o1 System Cardcited by 1,244
  7. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  8. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  9. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  10. 2025LIMO: Less is More for Reasoningcited by 498
  11. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 326
  12. 2024Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systemscited by 164
8 more
  1. 2024Distilling System 2 into System 1cited by 159
  2. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 143
  3. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 109
  4. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 93
  5. 2024O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?cited by 92
  6. 2025Concise Reasoning via Reinforcement Learningcited by 79
  7. 2025Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Mathcited by 44
  8. 2025Med-RLVR: Emerging Medical Reasoning from a 3B base model via reinforcement Learningcited by 40

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 8.

14 papers.

Sort:

By area:

  1. 2025RL Fine-Tuning Heals OOD Forgetting in SFTcited by 31
  2. 2025CEC-Zero: Zero-Supervision Character Error Correction with Self-Generated Rewardscited by 10
  3. 2026Large Reasoning Models Are (Not Yet) Multilingual Latent Reasonerscited by 9
  4. 2025DiSRouter: Distributed Self-Routing for LLM Selectionscited by 5
  5. 2025Recall-Extend Dynamics: Enhancing Small Language Models through Controlled Exploration and Refined Offline Integrationcited by 3
  6. 2026Training Reasoning Models on Saturated Problems via Failure-Prefix Conditioningcited by 3
  7. 2026HEAL: Hindsight Entropy-Assisted Learning for Reasoning Distillationcited by 3
  8. 2026Offline Exploration-Aware Fine-Tuning for Long-Chain Mathematical Reasoningcited by 2
  9. 2025EvoSyn: Generalizable Evolutionary Data Synthesis for Verifiable Learningcited by 1
  10. 2026RLVR is a Kernel, Not a Function: Statistical Inference for pass@k Crossoverscited by 1
  11. 2025Optimizing for Persuasion Improves LLM Generalization: Evidence from Quality-Diversity Evolution of Debate Strategiescited by 0
  12. 2026Distilling Long-CoT Reasoning through Collaborative Step-wise Multi-Teacher Decodingcited by 0
2 more
  1. 2026Learning What to Learn: Stage-Specific Data Sets for SFT-then-RL in Small Language Model Reasoningcited by 0
  2. 2026Selecting Diverse SFT Traces Improves Post-RL Generalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  2. 2026On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation
  3. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
  4. 2025The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View
  5. 2025RLPR: Extrapolating RLVR to General Domains without Verifiers
  6. 2026SUPERNOVA: Eliciting General Reasoning in LLMs with Reinforcement Learning on Natural Instructions
  7. 2026Beyond Reasoning: Reinforcement Learning Unlocks Parametric Knowledge in LLMs
  8. 2025Harnessing Negative Signals: Reinforcement Distillation from Teacher Data for LLM Reasoning