# Beyond Outcome Verification: Verifiable Process Reward Models for Structured Reasoning

arXiv: https://arxiv.org/abs/2601.17223 · HTML: https://mlsanity.com/paper/2601.17223/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 9 of its 38 references that are in the map.
- First posted on arXiv 23 Jan 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)
- LLM alignment / RLHF / preference optimisation / [Process reward models for reasoning](https://mlsanity.com/alignment/process-reward-models-for-reasoning/)

## Builds on (most cited first, 9 of 9)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs](https://mlsanity.com/paper/2506.14245/) (2025)
- [Masked-and-Reordered Self-Supervision for Reinforcement Learning from Verifiable Rewards](https://mlsanity.com/paper/2511.17473/) (2025)

## Built on by (most cited first, 13 of 13)

- [AutoPyVerifier: Learning Compact Executable Verifiers for Large Language Model Outputs](https://mlsanity.com/paper/2604.22937/) (2026)
- [From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI](https://mlsanity.com/paper/2606.14502/) (2026)
- [CoT2-Meta: Budgeted Metacognitive Control for Test-Time Reasoning](https://mlsanity.com/paper/2603.28135/) (2026)
- [Enhancing Table Reasoning with Deterministic Table-State Rewards](https://mlsanity.com/paper/2601.22530/) (2026)
- [AutoForest: Automatically Generating Forest Plots from Biomedical Studies with End-to-End Evidence Extraction and Synthesis](https://mlsanity.com/paper/2606.02403/) (2026)
- [VeriBound: PAC-Bayesian Generalization Bounds for Process Reward Models Trained with Formal Verification Tools](https://mlsanity.com/paper/2606.20740/) (2026)
- [Internalizing Outcome Supervision into Process Supervision: A New Paradigm for Reinforcement Learning for Reasoning](https://mlsanity.com/paper/2605.05226/) (2026)
- [Shattering the Autoregressive Curse: Dynamic Epistemic Entropy Orchestrated Erasable Reinforcement Learning for LLMs](https://mlsanity.com/paper/2606.17735/) (2026)
- [Win by Silence: Deletion Non-Monotonicity, Autonomous Exploitation, and Typed-State Gating in LLM Plan Evaluation](https://mlsanity.com/paper/2607.12986/) (2026)
- [SBCO: Self-Supervised, Verifier-Grounded Harness Optimization For Planning Agents](https://mlsanity.com/paper/2608.10157/) (2026)
- [FinExam-10K: When Retrieval Helps Financial Reasoning?](https://mlsanity.com/paper/2608.28155/) (2026)
- [Physics as the label for measuring and correcting materials reasoning in multimodal models](https://mlsanity.com/paper/2609.12181/) (2026)
- [See, Measure, and Reason: Learning Visually Grounded Reasoning in Pathology](https://mlsanity.com/paper/2609.34277/) (2026)

## Abstract (arXiv)

Recent work on reinforcement learning with verifiable rewards (RLVR) has shown that large language models (LLMs) can be substantially improved using outcome-level verification signals, such as unit tests for code or exact-match checks for mathematics. In parallel, process supervision has long been explored as a way to shape the intermediate reasoning behaviour of LLMs, but existing approaches rely on neural judges to score chain-of-thought steps, leaving them vulnerable to opacity, bias, and reward hacking. To address this gap, we introduce Verifiable Process Reward Models (VPRMs), a reinforcement-learning framework in which intermediate reasoning steps are checked by deterministic, rule-based verifiers. We apply VPRMs to risk-of-bias assessment for medical evidence synthesis, a domain where guideline-defined criteria and rule-based decision paths enable programmatic verification of reasoning traces. Across multiple datasets, we find that VPRMs generate reasoning that adheres closely to domain rules and achieve substantially higher coherence between step-level decisions and final labels. Results show that VPRMs achieve up to 20% higher F1 than state-of-the-art models and 6.5% higher than verifiable outcome rewards, with substantial gains in evidence grounding and logical coherence.