mlsanityData to 7 Oct 2026

Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model

arXiv 2503.24290 ↗First posted 31 Mar 2025cs.LG cs.CLcited by 450

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

We introduce Open-Reasoner-Zero, the first open source implementation of large-scale reasoning-oriented RL training on the base model focusing on scalability, simplicity and accessibility. Through extensive experiments, we demonstrate that a minimalist approach, vanilla PPO with GAE (λ=1, γ=1) and straightforward rule-based rewards, without any KL regularization, is sufficient to scale up both benchmark performance and response length, replicating the scaling phenomenon observed in DeepSeek-R1-Zero. Using the same base model, Qwen2.5-32B base, as DeepSeek-R1-Zero-Qwen-32B, our implementation achieves superior performance across AIME2024, MATH500, and GPQA Diamond, while demonstrating remarkable efficiency, requiring only 1/10 of the training steps compared to the DeepSeek-R1-Zero pipeline. Moreover, our analysis not only covers training dynamics and ablation for critical design choices, but also quantitatively shows how the learned critic in Reasoner-Zero training effectively identifies and devalues repetitive response patterns, yielding more robust advantage estimations and enhancing training stability. Embracing the principles of open-source, we release our source code, training data, and various model weights, fostering reproducibility and encouraging further exploration of the properties of related models.

Key facts. Cited by 450 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 16 of its 34 references that are in the map. First posted on arXiv 31 Mar 2025.

Builds on

16 of 34 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  2. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  3. 2025Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning10 Feb
  4. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 11

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  8. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  9. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  10. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 323
  11. 2024Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systemscited by 162

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 269; 2026: 181.

450 papers.

Sort:

By area:

  1. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  2. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  3. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  4. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  5. 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 320
  6. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  7. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 276
  8. 2025Learning to Reason under Off-Policy Guidancecited by 266
  9. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  10. 2025TTRL: Test-Time Reinforcement Learningcited by 253
  11. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  12. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
38 more
  1. 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 218
  2. 2025Learning to Reason without External Rewardscited by 199
  3. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 199
  4. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  5. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 166
  6. 2025Agentic Reinforced Policy Optimizationcited by 166
  7. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 162
  8. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  9. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  10. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoningcited by 147
  11. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 145
  12. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 134
  13. 2025General-Reasoner: Advancing LLM Reasoning Across All Domainscited by 132
  14. 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 130
  15. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivizationcited by 124
  16. 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttrainingcited by 117
  17. 2025Perception-R1: Pioneering Perception Policy with Reinforcement Learningcited by 116
  18. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 110
  19. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertaintycited by 107
  20. 2025Geometric-Mean Policy Optimizationcited by 107
  21. 2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 102
  22. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questionscited by 95
  23. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 91
  24. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
  25. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learningcited by 88
  26. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 88
  27. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimizationcited by 87
  28. 2025CWM: An Open-Weights LLM for Research on Code Generation with World Modelscited by 87
  29. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learningcited by 84
  30. 2025Reasoning with Sampling: Your Base Model is Smarter Than You Thinkcited by 84
  31. 2025RLPR: Extrapolating RLVR to General Domains without Verifierscited by 83
  32. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 83
  33. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Modelscited by 81
  34. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoningcited by 80
  35. 2025Reinforcing General Reasoning without Verifierscited by 76
  36. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspectivecited by 75
  37. 2025Kwai Keye-VL 1.5 Technical Reportcited by 68
  38. 2025In-the-Flow Agentic System Optimization for Effective Planning and Tool Usecited by 67

50 most cited of 450 All 450 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025OpenSIR: Open-Ended Self-Improving Reasoner
  2. 2025Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM Reasoners With Verifiers
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale
  4. 2025Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners
  5. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  6. 2024OpenR: An Open Source Framework for Advanced Reasoning with Large Language Models
  7. 2025Reinforcement Learning is all You Need
  8. 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning