mlsanityData to 14 Oct 2026

Why Distillation can Outperform Zero-RL: The Role of Flexible Reasoning

arXiv 2505.21067 ↗First posted 27 May 2025cs.AIcited by 15

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning (RL) has played an important role in improving the reasoning ability of large language models (LLMs). Some studies apply RL directly to textitsmaller base models (known as zero-RL) and also achieve notable progress. However, in this paper, we show that using only 920 examples, a simple distillation method based on the base model can clearly outperform zero-RL, which typically requires much more data and computational cost. By analyzing the token frequency in model outputs, we find that the distilled model shows more flexible reasoning. It uses anthropomorphic tokens and logical connectors much more often than the zero-RL model. Further analysis reveals that distillation enhances the presence of two advanced cognitive behaviors: Multi-Perspective Thinking or Attempting and Metacognitive Awareness. Frequent occurrences of these two advanced cognitive behaviors give rise to flexible reasoning, which is essential for solving complex reasoning problems, while zero-RL fails to significantly boost the frequency of these behaviors.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 33 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

28 of 33 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  2. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  3. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  5. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  7. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  8. 2025LIMR: Less is More for RL Scaling17 Feb
  9. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  10. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  11. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2024GPT-4o System Cardcited by 4,588
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  11. 2025s1: Simple test-time scalingcited by 1,341
  12. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
5 more
  1. 2025LIMO: Less is More for Reasoningcited by 498
  2. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  3. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 93
  4. 2025RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?cited by 60
  5. 2025Understanding Aha Moments: from External Observations to Internal Mechanismscited by 50

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 7.

15 papers.

Sort:

By area:

  1. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
  2. 2025rStar2-Agent: Agentic Reasoning Technical Reportcited by 54
  3. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Rewardcited by 25
  4. 2026On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitationcited by 13
  5. 2025Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoningcited by 11
  6. 2025Debunk the Myth of SFT Generalizationcited by 8
  7. 2025Training Large Language Models To Reason In Parallel With Global Forking Tokenscited by 6
  8. 2025CoT-Evo: Evolutionary Distillation of Chain-of-Thought for Scientific Reasoningcited by 5
  9. 2025AIR: Post-training Data Selection for Reasoning via Attention Head Influencecited by 3
  10. 2026HEAL: Hindsight Entropy-Assisted Learning for Reasoning Distillationcited by 3
  11. 2026AGPO: Asymmetric Group Policy Optimization for Verifiable Reasoning and Search Ads Relevance at JDcited by 2
  12. 2026D-CORE: Incentivizing Task Decomposition in Large Reasoning Models for Complex Tool Usecited by 0
3 more
  1. 2026Distilling Long-CoT Reasoning through Collaborative Step-wise Multi-Teacher Decodingcited by 0
  2. 2026When Does Reasoning Help in Machine Translation? A Hierarchical Analysis of LRM Reasoning Tracescited by 0
  3. 2026Distillation Defenses Easily Break After Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Zero Reinforcement Learning Towards General Domains
  2. 2025Reinforcement Learning vs. Distillation: Understanding Accuracy and Capability in LLM Reasoning
  3. 2026RL Starts before RL: On Policy Distillation for Better Reinforcement Learning
  4. 2025Thinking with DistilQwen: A Tale of Four Distilled Reasoning and Reward Model Series
  5. 2025Harnessing Negative Signals: Reinforcement Distillation from Teacher Data for LLM Reasoning
  6. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
  7. 2025Small Models Struggle to Learn from Strong Reasoners
  8. 2026Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning