mlsanityData to 7 Oct 2026

Process Reinforcement through Implicit Rewards

arXiv 2502.01456 ↗First posted 3 Feb 2025cs.LG cs.AI cs.CLcited by 478

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Dense process rewards have proven a more effective alternative to the sparse outcome-level rewards in the inference-time scaling of large language models (LLMs), particularly in tasks requiring complex multi-step reasoning. While dense rewards also offer an appealing choice for the reinforcement learning (RL) of LLMs since their fine-grained rewards have the potential to address some inherent issues of outcome rewards, such as training efficiency and credit assignment, this potential remains largely unrealized. This can be primarily attributed to the challenges of training process reward models (PRMs) online, where collecting high-quality process labels is prohibitively expensive, making them particularly vulnerable to reward hacking. To address these challenges, we propose PRIME (Process Reinforcement through IMplicit rEwards), which enables online PRM updates using only policy rollouts and outcome labels through implict process rewards. PRIME combines well with various advantage functions and forgoes the dedicated reward model training phrase that existing approaches require, substantially reducing the development overhead. We demonstrate PRIME's effectiveness on competitional math and coding. Starting from Qwen2.5-Math-7B-Base, PRIME achieves a 15.1% average improvement across several key reasoning benchmarks over the SFT model. Notably, our resulting model, Eurus-2-7B-PRIME, surpasses Qwen2.5-Math-7B-Instruct on seven reasoning benchmarks with 10% of its training data.

Key facts. Cited by 478 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 42 of its 60 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

42 of 60 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  2. 2024Free Process Rewards without Process Labels2 Dec
  3. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  4. 2024On Designing Effective RL Reward at Training Time for LLM Reasoning19 Oct
  5. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
  6. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  7. 2023Let's Verify Step by Step31 May
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2022Solving math word problems with process- and outcome-based feedback25 Nov
  10. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  9. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  11. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
20 more
  1. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  2. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  4. 2021Measuring Coding Challenge Competence With APPScited by 879
  5. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  6. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  7. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  8. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  9. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
  10. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
  11. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  12. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 194
  13. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  14. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  15. 2023TACO: Topics in Algorithmic COde generation datasetcited by 150
  16. 2024MAmmoTH2: Scaling Instructions from the Webcited by 143
  17. 2024Noise Contrastive Alignment of Language Models with Explicit Rewardscited by 85
  18. 2024AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modelingcited by 76
  19. 2024Bootstrapping Language Models with DPO Implicit Rewardscited by 53
  20. 2024Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Modelscited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 244; 2026: 234.

478 papers.

Sort:

By area:

  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  2. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  3. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  4. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  5. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  6. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
  7. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  8. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  9. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  10. 2025MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learningcited by 294
  11. 2026Reinforcement Learning via Self-Distillationcited by 285
  12. 2025Reasoning with Exploration: An Entropy Perspectivecited by 277
38 more
  1. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 276
  2. 2025Learning to Reason under Off-Policy Guidancecited by 266
  3. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  4. 2025TTRL: Test-Time Reinforcement Learningcited by 253
  5. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  6. 2025ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learningcited by 187
  7. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 187
  8. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  9. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  10. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 175
  11. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systemscited by 170
  12. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
  13. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  14. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  15. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoningcited by 147
  16. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 141
  17. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scalingcited by 135
  18. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivizationcited by 124
  19. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 123
  20. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuningcited by 121
  21. 2025Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoningcited by 118
  22. 2026Self-Distilled RLVRcited by 109
  23. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  24. 2025Geometric-Mean Policy Optimizationcited by 107
  25. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learningcited by 100
  26. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questionscited by 95
  27. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoningcited by 94
  28. 2025TempFlow-GRPO: When Timing Matters for GRPO in Flow Modelscited by 93
  29. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
  30. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learningcited by 88
  31. 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentationcited by 86
  32. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learningcited by 84
  33. 2025RLPR: Extrapolating RLVR to General Domains without Verifierscited by 83
  34. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 83
  35. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  36. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoningcited by 80
  37. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 78
  38. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignmentcited by 77

50 most cited of 478 All 478 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026LLM Reasoning with Process Rewards for Outcome-Guided Steps
  2. 2025From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling
  3. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Models
  4. 2026Reward Granularity in RLVR: Comparing Process and Outcome Reward Structures for Mathematical Reasoning in Small Language Models
  5. 2025Accelerating LLM Reasoning via Early Rejection with Partial Reward Modeling
  6. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning
  7. 2023Let's Reinforce Step by Step
  8. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models