mlsanityData to 14 Oct 2026

EST-PRM: Stress-Testing Process Reward Models Before They Become Load-Bearing

arXiv 2606.00437 ↗First posted 30 May 2026cs.LGcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Process reward models (PRMs) are widely used in language-model training with dense step-level supervision. They assume PRM scores are stable proxies for step correctness under label-preserving transformations. These transformations change reasoning structure but preserve final answers. We argue this assumption is not well validated. Such transformations can change how PRM scores relate to correctness signals, leading to different failure modes across models.To address this gap, we introduce EST-PRM, a stress-testing framework for dense process rewards. It applies three transformations: (1) step inflation, (2) dependency-aware step reordering, and (3) confidence markers. A vulnerability decomposition is defined that separates reward inflation from loss of correctness sensitivity. Five PRM-style models are evaluated on 4,687 reasoning chains from MATH-500, GSM8K, and PRMBench.The results indicate clear differences in vulnerability patterns across models. Math-Shepherd shows the strongest sensitivity to position perturbations, with a Pearson correlation drop of 0.152 ± 0.038 and a 32.8 ± 4.9% score inflation rate. Qwen2.5-Math-PRM is most affected by step inflation, reaching a 47.6 ± 4.3% inflation rate. Confidence-based perturbations also distort reward calibration, revealing inconsistencies in correctness estimation. Three mitigation strategies are evaluated, highlighting trade-offs between robustness coverage and false-positive rates.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 22 of its 42 references that are in the map. First posted on arXiv 30 May 2026.

Builds on

22 of 42 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing2 Apr
  2. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan

Other references · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2024Mixtral of Expertscited by 1,823
  4. 2016Concrete Problems in AI Safetycited by 1,642
  5. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  7. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  8. 2026Reinforcement Learning via Self-Distillationcited by 332
  9. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 271
  10. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 184
  11. 2025Skywork Open Reasoner 1 Technical Reportcited by 165
  12. 2018Categorizing Variants of Goodhart's Lawcited by 149
8 more
  1. 2026Self-Distilled RLVRcited by 131
  2. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 90
  3. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 82
  4. 2026Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillationcited by 60
  5. 2026Aligning Language Models from User Interactionscited by 28
  6. 2025Detecting Proxy Gaming in RL and LLM Alignment via Evaluator Stress Testscited by 19
  7. 2026Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 14
  8. 2025Anytime-Valid Answer Sufficiency Certificates for LLM Generation via Sequential Information Liftcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Quality-Diversity Stress Tests for Process Reward Models:What Archive Coverage Can and Cannot Certifycited by 0
  2. 2026Auditing Proxy-Based Validation Across Text Spanscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models
  2. 2025From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling
  3. 2025Error Typing for Smarter Rewards: Improving Process Reward Models with Error-Aware Hierarchical Supervision
  4. 2025GroundedPRM: Tree-Guided and Fidelity-Aware Process Reward Modeling for Step-Level Reasoning
  5. 2024Free Process Rewards without Process Labels
  6. 2025R-PRM: Reasoning-Driven Process Reward Modeling
  7. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Models
  8. 2025Rethinking Reward Models for Multi-Domain Test-Time Scaling