mlsanityData to 7 Oct 2026

Process Reward Model with Q-Value Rankings

arXiv 2410.11287 ↗First posted 15 Oct 2024cs.CL cs.AIcited by 78

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Process Reward Modeling (PRM) is critical for complex reasoning and decision-making tasks where the accuracy of intermediate steps significantly influences the overall outcome. Existing PRM approaches, primarily framed as classification problems, employ cross-entropy loss to independently evaluate each step's correctness. This method can lead to suboptimal reward distribution and does not adequately address the interdependencies among steps. To address these limitations, we introduce the Process Q-value Model (PQM), a novel framework that redefines PRM in the context of a Markov Decision Process. PQM optimizes Q-value rankings based on a novel comparative loss function, enhancing the model's ability to capture the intricate dynamics among sequential decisions. This approach provides a more granular and theoretically grounded methodology for process rewards. Our extensive empirical evaluations across various sampling policies, language model backbones, and multi-step reasoning benchmarks show that PQM outperforms classification-based PRMs. The effectiveness of the comparative loss function is highlighted in our comprehensive ablation studies, confirming PQM's practical efficacy and theoretical advantage.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 100 of its 156 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

100 of 156 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search6 Jun
  2. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  3. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  4. 2023Let's Verify Step by Step31 May
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 94

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  9. 2024Qwen2 Technical Reportcited by 2,076
  10. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  11. 2020Energy-based Out-of-distribution Detectioncited by 1,296
  12. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
38 more
  1. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  2. 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
  3. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  4. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  5. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 596
  6. 2021ReAct: Out-of-distribution Detection With Rectified Activationscited by 486
  7. 2017Snapshot Ensembles: Train 1, get M for freecited by 349
  8. 2021On the Importance of Gradients for Detecting Distributional Shifts in the Wildcited by 338
  9. 2022VOS: Learning What You Don't Know by Virtual Outlier Synthesiscited by 300
  10. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  11. 2022Mitigating Neural Network Overconfidence with Logit Normalizationcited by 251
  12. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  13. 2021MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Spacecited by 233
  14. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  15. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 215
  16. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 215
  17. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
  18. 2023OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detectioncited by 178
  19. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 155
  20. 2021A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challengescited by 154
  21. 2024Token-level Direct Preference Optimizationcited by 135
  22. 2023Non-Parametric Outlier Synthesiscited by 129
  23. 2022How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection?cited by 128
  24. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  25. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Modelscited by 111
  26. 2022Is Out-of-Distribution Detection Learnable?cited by 109
  27. 2022POEM: Out-of-Distribution Detection with Posterior Samplingcited by 108
  28. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  29. 2021Can multi-label classification networks know what they don't know?cited by 100
  30. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  31. 2024HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detectioncited by 96
  32. 2024Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planningcited by 95
  33. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 94
  34. 2022Training OOD Detectors in their Natural Habitatscited by 86
  35. 2020Model Patching: Closing the Subgroup Performance Gap with Data Augmentationcited by 82
  36. 2021Provable Guarantees for Understanding Out-of-distribution Detectioncited by 78
  37. 2023Dream the Impossible: Outlier Imagination with Diffusion Modelscited by 77
  38. 2024Step-level Value Preference Optimization for Mathematical Reasoningcited by 71

50 most cited of 94 All 94 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 51; 2026: 27.

78 papers.

Sort:

By area:

  1. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 464
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  3. 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Stepscited by 225
  4. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  5. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scalingcited by 135
  6. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoningcited by 94
  7. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 69
  8. 2025Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoningcited by 67
  9. 2025Unlocking Multimodal Mathematical Reasoning via Process Reward Modelcited by 64
  10. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  11. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMscited by 44
  12. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Modelscited by 37
38 more
  1. 2025VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoTcited by 36
  2. 2025Robust Reward Modeling via Causal Rubricscited by 18
  3. 2025Athena: Enhancing Multimodal Reasoning with Data-efficient Process Reward Modelscited by 16
  4. 2025Dynamic and Generalizable Process Reward Modelingcited by 16
  5. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 15
  6. 2025Efficient Process Reward Model Training via Active Learningcited by 14
  7. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  8. 2025Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoningcited by 14
  9. 2025E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testingcited by 14
  10. 2025GM-PRM: A Generative Multimodal Process Reward Model for Multimodal Mathematical Reasoningcited by 13
  11. 2025Is PRM Necessary? Problem-Solving RL Implicitly Induces PRM Capability in LLMscited by 12
  12. 2025A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematicscited by 11
  13. 2025BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Datasetcited by 11
  14. 2025Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learnercited by 11
  15. 2026TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agentscited by 11
  16. 2025FreePRM: Training Process Reward Models Without Ground Truth Process Labelscited by 10
  17. 2025Large Language Models Have Intrinsic Meta-Cognition, but Need a Good Lenscited by 10
  18. 2025GRPO is Secretly a Process Reward Modelcited by 10
  19. 2026Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoningcited by 10
  20. 2026A Subgoal-driven Framework for Improving Long-Horizon LLM Agentscited by 10
  21. 2026GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillationcited by 10
  22. 2025CMCTS: A Constrained Monte Carlo Tree Search Framework for Mathematical Reasoning in Large Language Modelcited by 9
  23. 2025Thinking Preference Optimizationcited by 8
  24. 2025MMaDA-Parallel: Multimodal Large Diffusion Language Models for Thinking-Aware Editing and Generationcited by 8
  25. 2025TIM-PRM: Verifying multimodal reasoning with Tool-Integrated PRMcited by 8
  26. 2025Coarse-to-Fine Process Reward Modeling for Mathematical Reasoningcited by 7
  27. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  28. 2026Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learningcited by 7
  29. 2025Rank-Then-Score: Enhancing Large Language Models for Automated Essay Scoringcited by 6
  30. 2025Video-R4: Reinforcing Text-Rich Video Reasoning with Visual Ruminationcited by 6
  31. 2025FANS -- Formal Answer Selection for Natural Language Math Reasoning Using Lean4cited by 5
  32. 2025Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data?cited by 5
  33. 2025TaTToo: Tool-Grounded Thinking PRM for Test-Time Scaling in Tabular Reasoningcited by 5
  34. 2026When Domains Interact: Asymmetric and Order-Sensitive Cross-Domain Effects in Reinforcement Learning for Reasoningcited by 5
  35. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
  36. 2025CoLD: Counterfactually-Guided Length Debiasing for Process Reward Models in Mathematical Reasoningcited by 4
  37. 2025The Bidirectional Process Reward Modelcited by 4
  38. 2026Grounding the Score: Explicit Visual Premise Verification for Reliable Vision-Language Process Reward Modelscited by 4

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025R-PRM: Reasoning-Driven Process Reward Modeling
  2. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning
  3. 2024Free Process Rewards without Process Labels
  4. 2025Discriminative Policy Optimization for Token-Level Reward Models
  5. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models
  6. 2025Socratic-PRMBench: Benchmarking Process Reward Models with Systematic Reasoning Patterns
  7. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning
  8. 2025More Bang for the Buck: Process Reward Modeling with Entropy-Driven Uncertainty