Data to 7 Oct 2026

ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models

arXiv 2505.24864 ↗First posted 30 May 2025cs.CL cs.AIcited by 187

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Recent advances in reasoning-centric language models have highlighted reinforcement learning (RL) as a promising method for aligning models with verifiable rewards. However, it remains contentious whether RL truly expands a model's reasoning capabilities or merely amplifies high-reward outputs already latent in the base model's distribution, and whether continually scaling up RL compute reliably leads to improved reasoning performance. In this work, we challenge prevailing assumptions by demonstrating that prolonged RL (ProRL) training can uncover novel reasoning strategies that are inaccessible to base models, even under extensive sampling. We introduce ProRL, a novel training methodology that incorporates KL divergence control, reference policy resetting, and a diverse suite of tasks. Our empirical analysis reveals that RL-trained models consistently outperform base models across a wide range of pass@k evaluations, including scenarios where base models fail entirely regardless of the number of attempts. We further show that reasoning boundary improvements correlates strongly with task competence of base model and training duration, suggesting that RL can explore and populate new regions of solution space over time. These findings offer new insights into the conditions under which RL meaningfully expands reasoning boundaries in language models and establish a foundation for future work on long-horizon RL for reasoning. We release model weights to support further research: https://huggingface.co/nvidia/Nemotron-Research-Reasoning-Qwen-1.5B

Key facts. Cited by 187 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 27 of its 53 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

27 of 53 references.

Closest ancestors · 6

Same direction, newest first

Other references · 21

2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
2013Playing Atari with Deep Reinforcement Learningcited by 3,211
2019The Curious Case of Neural Text Degenerationcited by 2,881
2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
2016Concrete Problems in AI Safetycited by 1,582
2022Competition-Level Code Generation with AlphaCodecited by 1,427
2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
2022STaR: Bootstrapping Reasoning With Reasoningcited by 895

9 more

2021Measuring Coding Challenge Competence With APPScited by 879
2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
2024Language Models Learn to Mislead Humans via RLHFcited by 118
2020Mirror Descent Policy Optimizationcited by 102
2024AI as Humanity's Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Textcited by 40
2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 36
2025SCP-116K: A High-Quality Problem-Solution Dataset and a Generalized Pipeline for Automated Extraction in the Higher Education Science Domaincited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 91; 2026: 96.

187 papers.

Sort:

By area:

2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 276
2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 184
2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 145
2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 141
2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 134
2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 92
2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoningcited by 80
2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 79
2025AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergycited by 75

38 more

2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Modelscited by 64
2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 57
2025JustRL: Scaling a 1.5B LLM with a Simple RL Recipecited by 57
2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 56
2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 56
2025rStar2-Agent: Agentic Reasoning Technical Reportcited by 54
2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoningcited by 53
2025Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVRcited by 53
2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replaycited by 52
2025The Path Not Taken: RLVR Provably Learns Off the Principalscited by 52
2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspectivecited by 50
2025OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalizationcited by 48
2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?cited by 48
2025From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Onescited by 40
2025QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentationcited by 39
2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 39
2025Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVRcited by 38
2025DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per Token via Reinforcement Learningcited by 34
2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMscited by 32
2025Scaling up Multi-Turn Off-Policy RL and Multi-Agent Tree Search for LLM Step-Proverscited by 26
2025Nudging the Boundaries of LLM Reasoningcited by 25
2025Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Insteadcited by 24
2025Language Models that Think, Chat Bettercited by 23
2025RL Grokking Recipe: How Does RL Unlock and Transfer New Algorithms in LLMs?cited by 23
2025Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoningcited by 23
2025BroRL: Scaling Reinforcement Learning via Broadened Explorationcited by 23
2025Knapsack RL: Unlocking Exploration of LLMs via Optimizing Budget Allocationcited by 21
2025When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RLcited by 21
2025Rethinking Thinking Tokens: LLMs as Improvement Operatorscited by 20
2025Differential Smoothing Mitigates Sharpening and Improves LLM Reasoningcited by 19
2025Rethinking Sample Polarity in Reinforcement Learning with Verifiable Rewardscited by 19
2025ReForm: Reflective Autoformalization with Prospective Bounded Sequence Optimizationcited by 18
2025Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoningcited by 18
2025Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learningcited by 17
2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Modelscited by 17
2025Don't Just Fine-tune the Agent, Tune the Environmentcited by 17
2025AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agentcited by 17
2026ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agentscited by 17

50 most cited of 187 All 187 (JSON)

Similar, not cited

Close in wording, no citation either way