mlsanityData to 7 Oct 2026

SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild

arXiv 2503.18892 ↗First posted 24 Mar 2025cs.LG cs.AI cs.CLcited by 505

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

DeepSeek-R1 has shown that long chain-of-thought (CoT) reasoning can naturally emerge through a simple reinforcement learning (RL) framework with rule-based rewards, where the training may directly start from the base models-a paradigm referred to as zero RL training. Most recent efforts to reproduce zero RL training have primarily focused on the Qwen2.5 model series, which may not be representative as we find the base models already exhibit strong instruction-following and self-reflection abilities. In this work, we investigate zero RL training across 10 diverse base models, spanning different families and sizes including LLama3-8B, Mistral-7B/24B, DeepSeek-Math-7B, Qwen2.5-math-7B, and all Qwen2.5 models from 0.5B to 32B. Leveraging several key design strategies-such as adjusting format reward and controlling query difficulty-we achieve substantial improvements in both reasoning accuracy and response length across most settings. However, by carefully monitoring the training dynamics, we observe that different base models exhibit distinct patterns during training. For instance, the increased response length does not always correlate with the emergence of certain cognitive behaviors such as verification (i.e., the "aha moment"). Notably, we observe the "aha moment" for the first time in small models not from the Qwen family. We share the key designs that enable successful zero RL training, along with our findings and practices. To facilitate further research, we open-source the code, models, and analysis tools.

Key facts. Cited by 505 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 37 references that are in the map. First posted on arXiv 24 Mar 2025.

Builds on

29 of 37 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  2. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
  3. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  4. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  5. 2023Let's Verify Step by Step31 May

Other references · 24

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2024Qwen2.5 Technical Reportcited by 4,713
  8. 2024GPT-4o System Cardcited by 4,467
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  10. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  11. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  12. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
12 more
  1. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  2. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  3. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  4. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
  5. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  6. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 323
  7. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
  8. 2024Planning In Natural Language Improves LLM Search For Code Generationcited by 79
  9. 2025An Empirical Study on Eliciting and Improving R1-like Reasoning Modelscited by 78
  10. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 68
  11. 2024Diving into Self-Evolving Training for Multimodal Reasoningcited by 31
  12. 2024B-STaR: Monitoring and Balancing Exploration and Exploitation in Self-Taught Reasonerscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 317; 2026: 188.

505 papers.

Sort:

By area:

  1. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  2. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  3. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 450
  4. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  5. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  6. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  7. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
  8. 2025Reasoning with Exploration: An Entropy Perspectivecited by 277
  9. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 276
  10. 2025Learning to Reason under Off-Policy Guidancecited by 266
  11. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  12. 2025TTRL: Test-Time Reinforcement Learningcited by 253
38 more
  1. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  2. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  3. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  4. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 209
  5. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 199
  6. 2025ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learningcited by 187
  7. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 187
  8. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  9. 2025Agentic Reinforced Policy Optimizationcited by 166
  10. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162
  11. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  12. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domainscited by 150
  13. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 145
  14. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 141
  15. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 134
  16. 2025General-Reasoner: Advancing LLM Reasoning Across All Domainscited by 132
  17. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  18. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 110
  19. 2025Thinkless: LLM Learns When to Thinkcited by 107
  20. 2025Geometric-Mean Policy Optimizationcited by 107
  21. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 101
  22. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questionscited by 95
  23. 2025Efficient Reasoning Models: A Surveycited by 93
  24. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 92
  25. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
  26. 2025Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoningcited by 89
  27. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learningcited by 88
  28. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 88
  29. 2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Modelscited by 87
  30. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimizationcited by 87
  31. 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentationcited by 86
  32. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedbackcited by 86
  33. 2025WebExplorer: Explore and Evolve for Training Long-Horizon Web Agentscited by 86
  34. 2025VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Usecited by 84
  35. 2025Reasoning with Sampling: Your Base Model is Smarter Than You Thinkcited by 84
  36. 2025RLPR: Extrapolating RLVR to General Domains without Verifierscited by 83
  37. 2025Acting Less is Reasoning More! Teaching Model to Act Efficientlycited by 80
  38. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoningcited by 80

50 most cited of 505 All 505 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Understanding R1-Zero-Like Training: A Critical Perspective
  2. 2025You Need Reasoning to Learn Reasoning: The Limitations of Label-Free RL in Weak Base Models
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
  4. 2025Reinforcement Learning is all You Need
  5. 2025The Challenge of Teaching Reasoning to LLMs Without RL or Distillation
  6. 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning
  7. 2025Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL
  8. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models