mlsanityData to 14 Oct 2026

Uncertainty-Aware Step-wise Verification with Generative Reward Models

arXiv 2502.11250 ↗First posted 16 Feb 2025cs.CLcited by 22

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Complex multi-step reasoning tasks, such as solving mathematical problems, remain challenging for large language models (LLMs). While outcome supervision is commonly used, process supervision via process reward models (PRMs) provides intermediate rewards to verify step-wise correctness in solution traces. However, as proxies for human judgement, PRMs suffer from reliability issues, including susceptibility to reward hacking. In this work, we propose leveraging uncertainty quantification (UQ) to enhance the reliability of step-wise verification with generative reward models for mathematical reasoning tasks. We introduce CoT Entropy, a novel UQ method that outperforms existing approaches in quantifying a PRM's uncertainty in step-wise verification. Our results demonstrate that incorporating uncertainty estimates improves the robustness of judge-LM PRMs, leading to more reliable verification.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 23 of its 28 references that are in the map. First posted on arXiv 16 Feb 2025.

Builds on

23 of 28 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  2. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoning9 Dec
  3. 2024Generative Verifiers: Reward Modeling as Next-Token Prediction27 Aug
  4. 2023Let's Verify Step by Step31 May
  5. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 18

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2017On Calibration of Modern Neural Networkscited by 4,960
  7. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  8. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  9. 2017Deep Bayesian Active Learning with Image Datacited by 819
  10. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  11. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  12. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 259
6 more
  1. 2024Iterative Reasoning Preference Optimizationcited by 246
  2. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 121
  3. 2024Improving Reward Models with Synthetic Critiquescited by 44
  4. 2024Improving Uncertainty Quantification in Large Language Models via Semantic Embeddingscited by 30
  5. 2024Deep Bayesian Active Learning for Preference Modeling in Large Language Modelscited by 19
  6. 2023How to Fix a Broken Confidence Estimator: Evaluating Post-hoc Methods for Selective Classification with Deep Neural Networkscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 10.

22 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimizationcited by 88
  3. 2025TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoningcited by 24
  4. 2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
  5. 2025Know What You Don't Know: Uncertainty Calibration of Process Reward Modelscited by 21
  6. 2025Semantic Energy: Detecting LLM Hallucination Beyond Entropycited by 18
  7. 2026InfoPO: Information-Driven Policy Optimization for User-Centric Agentscited by 12
  8. 2025Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learnercited by 11
  9. 2025More Bang for the Buck: Process Reward Modeling with Entropy-Driven Uncertaintycited by 10
  10. 2025Towards Agents That Know When They Don't Know: Uncertainty as a Control Signal for Structured Reasoningcited by 10
  11. 2025SSR: Socratic Self-Refine for Large Language Model Reasoningcited by 7
  12. 2025On the Limits of Test-Time Compute: Sequential Reward Filtering for Better Inferencecited by 6
10 more
  1. 2026Grounding the Score: Explicit Visual Premise Verification for Reliable Vision-Language Process Reward Modelscited by 4
  2. 2025Reasoning Path and Latent State Analysis for Multi-view Visual Spatial Reasoning: A Cognitive Science Perspectivecited by 3
  3. 2026Predicting and improving test-time scaling laws via reward tail-guided searchcited by 2
  4. 2026Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attributioncited by 2
  5. 2026The Origins of Stochasticity: Comprehensive Investigations on Uncertainty Quantification for Large Language Modelscited by 2
  6. 2026Verify, Repair, Repeat, or Stop? Robust Stopping for Noisy Verify-Repair Loops in LLM Agentscited by 2
  7. 2026One-Token Verification for Reasoning Correctness Estimationcited by 1
  8. 2026Granularity-Regulated Adaptive Computational Efficiency for Optimal Verification in Test-Time Scalingcited by 1
  9. 2026VeriBound: PAC-Bayesian Generalization Bounds for Process Reward Models Trained with Formal Verification Toolscited by 1
  10. 2026Rewarding Better Thinking for LLM Preference Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Uncertainty-Based Methods for Automated Process Reward Data Construction and Output Aggregation in Mathematical Reasoning
  2. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning
  3. 2025Error Typing for Smarter Rewards: Improving Process Reward Models with Error-Aware Hierarchical Supervision
  4. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Models
  5. 2026LLM Reasoning with Process Rewards for Outcome-Guided Steps
  6. 2025From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling
  7. 2025GroundedPRM: Tree-Guided and Fidelity-Aware Process Reward Modeling for Step-Level Reasoning
  8. 2025ScalePRM: Training Process Reward Models by Scaling Verification Compute Without Ground Truth