mlsanityData to 7 Oct 2026

Understanding R1-Zero-Like Training: A Critical Perspective

arXiv 2503.20783 ↗First posted 26 Mar 2025cs.LG cs.AI cs.CLcited by 1,276

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

DeepSeek-R1-Zero has shown that reinforcement learning (RL) at scale can directly enhance the reasoning capabilities of LLMs without supervised fine-tuning. In this work, we critically examine R1-Zero-like training by analyzing its two core components: base models and RL. We investigate a wide range of base models, including DeepSeek-V3-Base, to understand how pretraining characteristics influence RL performance. Our analysis reveals that DeepSeek-V3-Base already exhibit ''Aha moment'', while Qwen2.5 base models demonstrate strong reasoning capabilities even without prompt templates, suggesting potential pretraining biases. Additionally, we identify an optimization bias in Group Relative Policy Optimization (GRPO), which artificially increases response length (especially for incorrect outputs) during training. To address this, we introduce Dr. GRPO, an unbiased optimization method that improves token efficiency while maintaining reasoning performance. Leveraging these insights, we present a minimalist R1-Zero recipe that achieves 43.3% accuracy on AIME 2024 with a 7B base model, establishing a new state-of-the-art. Our code is available at https://github.com/sail-sg/understand-r1-zero.

Key facts. Cited by 1,276 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 20 of its 35 references that are in the map. First posted on arXiv 26 Mar 2025.

Builds on

20 of 35 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Process Reinforcement through Implicit Rewards3 Feb
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 17

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  8. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  11. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  12. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
5 more
  1. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
  2. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 323
  3. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 307
  4. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  5. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 551; 2026: 725.

1,276 papers.

Sort:

By area:

  1. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  2. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
  3. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  4. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 450
  5. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  6. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  7. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  8. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  9. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  10. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
  11. 2026Reinforcement Learning via Self-Distillationcited by 285
  12. 2025Reasoning with Exploration: An Entropy Perspectivecited by 277
38 more
  1. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 276
  2. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 267
  3. 2025Learning to Reason under Off-Policy Guidancecited by 266
  4. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  5. 2025TTRL: Test-Time Reinforcement Learningcited by 253
  6. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  7. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  8. 2025Dynamic Early Exit in Reasoning Modelscited by 217
  9. 2025Learning to Reason without External Rewardscited by 199
  10. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 199
  11. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  12. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 178
  13. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  14. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 175
  15. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 166
  16. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 162
  17. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  18. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  19. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoningcited by 147
  20. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 145
  21. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 141
  22. 2025Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correctioncited by 140
  23. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 134
  24. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  25. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 123
  26. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
  27. 2025Memento: Fine-tuning LLM Agents without Fine-tuning LLMscited by 119
  28. 2025Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Searchcited by 116
  29. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 110
  30. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rolloutscited by 108
  31. 2025Thinkless: LLM Learns When to Thinkcited by 107
  32. 2025Geometric-Mean Policy Optimizationcited by 107
  33. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questionscited by 95
  34. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Modelscited by 93
  35. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 91
  36. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
  37. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 89
  38. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimizationcited by 87

50 most cited of 1,276 All 1,276 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild
  2. 2025Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't
  3. 2025SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM
  4. 2025Adaptive Group Policy Optimization: Towards Stable Training and Token-Efficient Reasoning
  5. 2025Reinforcement Learning is all You Need
  6. 2025Predictive Scaling Laws for Efficient GRPO Training of Large Reasoning Models
  7. 2025R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Model
  8. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection