mlsanityData to 14 Oct 2026

FreePRM: Training Process Reward Models Without Ground Truth Process Labels

arXiv 2506.03570 ↗First posted 4 Jun 2025cs.CLcited by 11

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) have demonstrated that Process Reward Models (PRMs) play a crucial role in enhancing model performance. However, training PRMs typically requires step-level labels, either manually annotated or automatically generated, which can be costly and difficult to obtain at scale. To address this challenge, we introduce FreePRM, a weakly supervised framework for training PRMs without access to ground-truth step-level labels. FreePRM first generates pseudo step-level labels based on the correctness of final outcome, and then employs Buffer Probability to eliminate impact of noise inherent in pseudo labeling. Experimental results show that FreePRM achieves an average F1 score of 53.0% on ProcessBench, outperforming fully supervised PRM trained on Math-Shepherd by +24.1%. Compared to other open-source PRMs, FreePRM outperforms upon RLHFlow-PRM-Mistral-8B (28.4%) by +24.6%, EurusPRM (31.3%) by +21.7%, and Skywork-PRM-7B (42.1%) by +10.9%. This work introduces a new paradigm in PRM training, significantly reducing reliance on costly step-level annotations while maintaining strong performance.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 29 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

25 of 29 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Process Reinforcement through Implicit Rewards3 Feb
  2. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  3. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoning9 Dec
  4. 2024Free Process Rewards without Process Labels2 Dec
  5. 2024Process Reward Model with Q-Value Rankings15 Oct
  6. 2024Generative Verifiers: Reward Modeling as Next-Token Prediction27 Aug
  7. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  8. 2024AlphaMath Almost Zero: Process Supervision without Process6 May
  9. 2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning8 Feb
  10. 2024Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervision5 Feb
  11. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  12. 2023Let's Verify Step by Step31 May

Other references · 13

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  7. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  8. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  9. 2024Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planningcited by 97
  10. 2023OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoningcited by 85
  11. 2024Evaluating Mathematical Reasoning Beyond Accuracycited by 69
  12. 2024LLM Critics Help Catch Bugs in Mathematics: Towards a Better Mathematical Verifier with Natural Language Feedbackcited by 41
1 more
  1. 2024AutoPSV: Automated Process-Supervised Verifiercited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 5.

11 papers.

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Modelscited by 38
  3. 2025When Agents go Astray: Course-Correcting SWE Agents with PRMscited by 28
  4. 2025GroundedPRM: Tree-Guided and Fidelity-Aware Process Reward Modeling for Step-Level Reasoningcited by 10
  5. 2026PaLMR: Towards Faithful Visual Reasoning via Multimodal Process Alignmentcited by 5
  6. 2025TrajSelector: Harnessing Latent Representations for Efficient and Effective Best-of-N in Large Reasoning Modelcited by 3
  7. 2025Adversarial Training for Process Reward Modelscited by 3
  8. 2026EGSS: Entropy-guided Stepwise Scaling for Reliable Software Engineeringcited by 2
  9. 2026Unsupervised Process Reward Modelscited by 1
  10. 2026Process Rewards with Learned Reliabilitycited by 0
  11. 2026Auditing Proxy-Based Validation Across Text Spanscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient Process Reward Model Training via Active Learning
  2. 2025DreamPRM-Code: Function-as-Step Process Reward Model with Label Correction for LLM Coding
  3. 2025VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data
  4. 2025R-PRM: Reasoning-Driven Process Reward Modeling
  5. 2025Dynamic and Generalizable Process Reward Modeling
  6. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning
  7. 2025VRPRM: Process Reward Modeling via Visual Reasoning
  8. 2026Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning