# Verifiable Process Rewards for Agentic Reasoning

arXiv: https://arxiv.org/abs/2605.10325 · HTML: https://mlsanity.com/paper/2605.10325/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 14 of its 39 references that are in the map.
- First posted on arXiv 11 May 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)
- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022)
- [Group-in-Group Policy Optimization for LLM Agent Training](https://mlsanity.com/paper/2505.10978/) (2025)
- [FireAct: Toward Language Agent Fine-tuning](https://mlsanity.com/paper/2310.05915/) (2023)
- [OpenSpiel: A Framework for Reinforcement Learning in Games](https://mlsanity.com/paper/1908.09453/) (2019)
- [Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library](https://mlsanity.com/paper/2506.06122/) (2025)
- [GEM: A Gym for Agentic LLMs](https://mlsanity.com/paper/2510.01051/) (2025)

## Built on by (most cited first, 11 of 11)

- [Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress](https://mlsanity.com/paper/2608.19408/) (2026)
- [Learning from Environmental Feedback: Credit Assignment across Multiple Timescales for Agentic Reinforcement Learning](https://mlsanity.com/paper/2608.08255/) (2026)
- [TRACER: Per-Tool Context Retention for LLM Agents via Consequence-Attributed Reinforcement Learning](https://mlsanity.com/paper/2608.29363/) (2026)
- [Coverage, Not Targeting: A Structural Regime in Multi-Turn Agent Credit Assignment](https://mlsanity.com/paper/2609.02417/) (2026)
- [Learning to Refine Hidden States for Reliable LLM Reasoning](https://mlsanity.com/paper/2606.17524/) (2026)
- [Ignorance or Incompetence? Constructing Knowledge-Gated, Verifiable Tasks for LLM Agents](https://mlsanity.com/paper/2608.30322/) (2026)
- [ProCredit: From Outcome Rewards to Progress Credit in Agentic Reinforcement Learning](https://mlsanity.com/paper/2609.27532/) (2026)
- [RLHarness: Co-evolving Procedural Skills with Reinforcement Learning for Long-horizon Multimodal Reasoning](https://mlsanity.com/paper/2609.32326/) (2026)
- [From Outcomes to Strategies: Learning Strategy Utility for Mathematical Reasoning](https://mlsanity.com/paper/2609.32482/) (2026)
- [TRACE: Diagnosing Verifier Brittleness in Agentic Evaluation](https://mlsanity.com/paper/2610.11678/) (2026)
- [Tev: Scaling Long-Horizon Agentic Environments with Dense Progress Rewards](https://mlsanity.com/paper/2610.18868/) (2026)

## Abstract (arXiv)

Reinforcement learning from verifiable rewards (RLVR) has improved the reasoning capabilities of large language models (LLMs), yet existing approaches predominantly rely on sparse outcome-level feedback, posing a substantial credit-assignment challenge in long-horizon agentic tasks. In this work, we study structured agentic reasoning problems that admit symbolic or algorithmic task-solving oracles. We propose Verifiable Process Rewards (VPR), a training framework that repurposes such task oracles as action-level process verifiers, providing dense rewards for intermediate actions and enabling fine-grained credit assignment. We instantiate VPR in three distinct settings: a search-based oracle for Sokoban, a constraint-based oracle for Sudoku, and a posterior-based oracle for Minesweeper. To optimize policies with these rewards, we introduce oracle-guided state-group rollout and locally normalized optimization. Empirically, VPR-trained Qwen3-4B outperforms outcome-reward GRPO and VinePPO across all three training environments. When jointly trained on math data and all three game environments, VPR achieves the highest average general-reasoning performance and strongest agentic results among the compared training configurations. To further assess its applicability to more general agentic tool-use settings, we apply VPR-style action-level supervision to τ2-bench using an expert LLM as a reference policy and observe improved performance over outcome-reward GRPO on held-out tasks. Together, these results position VPR as a promising approach for leveraging reliable task oracles to improve agentic reasoning, while highlighting its dependence on the availability and reliability of task oracles and the challenge of extending intermediate verification to less structured, open-ended environments.