mlsanityData to 7 Oct 2026

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data

arXiv 2502.06737 ↗First posted 10 Feb 2025cs.LGcited by 39

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Process Reward Models (PRMs) have proven effective at enhancing mathematical reasoning for Large Language Models (LLMs) by leveraging increased inference-time computation. However, they are predominantly trained on mathematical data and their generalizability to non-mathematical domains has not been rigorously studied. In response, this work first shows that current PRMs have poor performance in other domains. To address this limitation, we introduce VersaPRM, a multi-domain PRM trained on synthetic reasoning data generated using our novel data generation and annotation method. VersaPRM achieves consistent performance gains across diverse domains. For instance, in the MMLU-Pro category of Law, VersaPRM via weighted majority voting, achieves a 7.9% performance gain over the majority voting baseline -- surpassing Qwen2.5-Math-PRM's gain of 1.3%. We further contribute to the community by open-sourcing all data, code and models for VersaPRM.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 40 of its 62 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

40 of 62 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  2. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoning9 Dec
  3. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  4. 2024Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervision5 Feb
  5. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  6. 2023Let's Verify Step by Step31 May
  7. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  8. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 32

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
20 more
  1. 2023Mistral 7Bcited by 3,065
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  4. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  5. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  6. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  7. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  8. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  9. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  10. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  11. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  12. 2023Self-Evaluation Guided Beam Search for Reasoningcited by 195
  13. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
  14. 2024OpenR: An Open Source Framework for Advanced Reasoning with Large Language Modelscited by 92
  15. 2022Self-Training: A Surveycited by 85
  16. 2024Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervisioncited by 60
  17. 2023LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedbackcited by 39
  18. 2023Can Large Language Models Follow Concept Annotation Guidelines? A Case Study on Scientific and Financial Domainscited by 7
  19. 2024Exploring Mathematical Extrapolation of Large Language Models with Synthetic Datacited by 6
  20. 2024Beyond Introspection: Reinforcing Thinking via Externalist Behavioral Feedbackcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 19; 2026: 20.

39 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoningcited by 94
  3. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
  4. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Modelscited by 37
  5. 2025Latent Thinking Optimization: Your Latent Reasoning Language Model Secretly Encodes Reward Signals in Its Latent Thoughtscited by 18
  6. 2025Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoningcited by 14
  7. 2025T1: Tool-integrated Verification for Test-time Compute Scaling in Small Language Modelscited by 13
  8. 2025Guiding Reasoning in Small Language Models with LLM Assistancecited by 13
  9. 2025GM-PRM: A Generative Multimodal Process Reward Model for Multimodal Mathematical Reasoningcited by 13
  10. 2025Temporal Consistency for LLM Reasoning Process Error Identificationcited by 10
  11. 2025Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoningcited by 9
  12. 2025Can Knowledge-Graph-based Retrieval Augmented Generation Really Retrieve What You Need?cited by 8
27 more
  1. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  2. 2025Efficient PRM Training Data Synthesis via Formal Verificationcited by 5
  3. 2026OpenRCA 2.0: From Outcome Labels to Causal Process Supervisioncited by 5
  4. 2025CoLD: Counterfactually-Guided Length Debiasing for Process Reward Models in Mathematical Reasoningcited by 4
  5. 2025Diagnosing Memorization in Chain-of-Thought Reasoning, One Token at a Timecited by 4
  6. 2026Grounding the Score: Explicit Visual Premise Verification for Reliable Vision-Language Process Reward Modelscited by 4
  7. 2026PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoningcited by 3
  8. 2026GRACE: Step-Level Benchmark for Faithful Reasoning over Contextcited by 2
  9. 2026Neglected Free Lunch from Post-training: Progress Advantage for LLM Agentscited by 2
  10. 2026Towards Generalizable Reasoning: Group Causal Counterfactual Policy Optimization for LLM Reasoningcited by 1
  11. 2026One-Token Verification for Reasoning Correctness Estimationcited by 1
  12. 2026Verifiable Counterfactual Supervision for Process Reward Modelscited by 1
  13. 2026STRIDE: Learnable Stepwise Language Feedback for LLM Reasoningcited by 1
  14. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  15. 2026Test-Time Scaling for Small VLMs on Multilingual Visual MCQcited by 1
  16. 2026Cliff: Learning Process Rewards from the First Mistakecited by 1
  17. 2025ContextPRM: Leveraging Contextual Coherence for multi-domain Test-Time Scalingcited by 0
  18. 2025Evaluating the Role of Verifiers in Test-Time Scaling for Legal Reasoning Taskscited by 0
  19. 2025MUSIC: MUlti-Step Instruction Contrast for Multi-Turn Reward Modelscited by 0
  20. 2026DC-W2S: Dual-Consensus Weak-to-Strong Training for Reliable Process Reward Modeling in Biological Reasoningcited by 0
  21. 2026PARM: Pipeline-Adapted Reward Modelcited by 0
  22. 2026From Long News to Accurate Forecast: Importance-Aware Fusion and PRM-Guided Reflection for Time Series Forecastingcited by 0
  23. 2026Operadic consistency: a label-free signal for compositional reasoning failures in LLMscited by 0
  24. 2026Test-Time Scaling in the Wild: Why Exploitation, Not Exploration, Is the Bottleneckcited by 0
  25. 2026SPEAR: Distilling Domain-Adaptive Reasoning Skeletons via Sequential Symbolic Alignment in Reinforcement Learningcited by 0
  26. 2026PRO-Step: Step-level Process Reward Optimization for Retrieval-Augmented Generationcited by 0
  27. 2026Characterizing a Configuration Where Inference-Time PRM-Pruned Fragment Grafting Is Inert: Evidence from Three Reasoning LMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025R-PRM: Reasoning-Driven Process Reward Modeling
  2. 2025From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling
  3. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning
  4. 2025Coarse-to-Fine Process Reward Modeling for Mathematical Reasoning
  5. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning
  6. 2025Accelerating LLM Reasoning via Early Rejection with Partial Reward Modeling
  7. 2025Retrieval-Augmented Process Reward Model for Generalizable Mathematical Reasoning
  8. 2026Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards