# EST-PRM: Stress-Testing Process Reward Models Before They Become Load-Bearing

arXiv: https://arxiv.org/abs/2606.00437 · HTML: https://mlsanity.com/paper/2606.00437/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 22 of its 42 references that are in the map.
- First posted on arXiv 30 May 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)
- LLM alignment / RLHF / preference optimisation / [Process reward models for reasoning](https://mlsanity.com/alignment/process-reward-models-for-reasoning/)

## Builds on (most cited first, 22 of 22)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [RewardBench: Evaluating Reward Models for Language Modeling](https://mlsanity.com/paper/2403.13787/) (2024)
- [The Lessons of Developing Process Reward Models in Mathematical Reasoning](https://mlsanity.com/paper/2501.07301/) (2025)
- [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026)
- [Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control](https://mlsanity.com/paper/2110.01052/) (2021)
- [Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation](https://mlsanity.com/paper/2602.12125/) (2026)
- [Skywork Open Reasoner 1 Technical Report](https://mlsanity.com/paper/2505.22312/) (2025)
- [Categorizing Variants of Goodhart's Law](https://mlsanity.com/paper/1803.04585/) (2018)
- [Self-Distilled RLVR](https://mlsanity.com/paper/2604.03128/) (2026)
- [OpenClaw-RL: Train Any Agent Simply by Talking](https://mlsanity.com/paper/2603.10165/) (2026)
- [Scaling Reasoning Efficiently via Relaxed On-Policy Distillation](https://mlsanity.com/paper/2603.11137/) (2026)
- [Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing](https://mlsanity.com/paper/2604.02288/) (2026)
- [Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation](https://mlsanity.com/paper/2603.19220/) (2026)
- [Aligning Language Models from User Interactions](https://mlsanity.com/paper/2603.12273/) (2026)
- [Detecting Proxy Gaming in RL and LLM Alignment via Evaluator Stress Tests](https://mlsanity.com/paper/2507.05619/) (2025)
- [Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2601.18699/) (2026)
- [Anytime-Valid Answer Sufficiency Certificates for LLM Generation via Sequential Information Lift](https://mlsanity.com/paper/2510.06478/) (2025)

## Built on by (most cited first, 2 of 2)

- [Quality-Diversity Stress Tests for Process Reward Models:What Archive Coverage Can and Cannot Certify](https://mlsanity.com/paper/2608.08008/) (2026)
- [Auditing Proxy-Based Validation Across Text Spans](https://mlsanity.com/paper/2609.25808/) (2026)

## Abstract (arXiv)

Process reward models (PRMs) are widely used in language-model training with dense step-level supervision. They assume PRM scores are stable proxies for step correctness under label-preserving transformations. These transformations change reasoning structure but preserve final answers. We argue this assumption is not well validated. Such transformations can change how PRM scores relate to correctness signals, leading to different failure modes across models.To address this gap, we introduce EST-PRM, a stress-testing framework for dense process rewards. It applies three transformations: (1) step inflation, (2) dependency-aware step reordering, and (3) confidence markers. A vulnerability decomposition is defined that separates reward inflation from loss of correctness sensitivity. Five PRM-style models are evaluated on 4,687 reasoning chains from MATH-500, GSM8K, and PRMBench.The results indicate clear differences in vulnerability patterns across models. Math-Shepherd shows the strongest sensitivity to position perturbations, with a Pearson correlation drop of 0.152 ± 0.038 and a 32.8 ± 4.9% score inflation rate. Qwen2.5-Math-PRM is most affected by step inflation, reaching a 47.6 ± 4.3% inflation rate. Confidence-based perturbations also distort reward calibration, revealing inconsistencies in correctness estimation. Three mitigation strategies are evaluated, highlighting trade-offs between robustness coverage and false-positive rates.