mlsanityData to 7 Oct 2026

ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents

arXiv 2601.12294 ↗First posted 18 Jan 2026cs.AI cs.SEcited by 13

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 2 more
Abstract (arXiv)

Reward-guided search methods have demonstrated strong potential in enhancing tool-using agents by effectively guiding sampling and exploration over complex action spaces. As a core design, those search methods utilize process reward models (PRMs) to provide step-level rewards, enabling more fine-grained monitoring. However, there is a lack of systematic and reliable evaluation benchmarks for PRMs in tool-using settings. In this paper, we introduce ToolPRMBench, a large-scale benchmark specifically designed to evaluate PRMs for tool-using agents. ToolPRMBench is built on top of several representative tool-using benchmarks and converts agent trajectories into step-level test cases. Each case contains the interaction history, a correct action, a plausible but incorrect alternative, and relevant tool metadata. We respectively utilize offline sampling to isolate local single-step errors and online sampling to capture realistic multi-step failures from full agent rollouts. A multi-LLM verification pipeline is proposed to reduce label noise and ensure data quality. We conduct extensive experiments across large language models, general PRMs, and tool-specialized PRMs on ToolPRMBench. The results reveal clear differences in PRM effectiveness and highlight the potential of specialized PRMs for tool-using. Code and data will be released at https://github.com/David-Li0406/ToolPRMBench.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 63 references that are in the map. First posted on arXiv 18 Jan 2026.

Builds on

36 of 63 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025ToolRM: Outcome Reward Models for Tool-Calling Large Language Models15 Sep
  2. 2025SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning10 Jun
  3. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning1 Apr
  4. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 32

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  9. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  10. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  11. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  12. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
20 more
  1. 2025DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decompositioncited by 270
  2. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 201
  3. 2025AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectoriescited by 99
  4. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
  5. 2024Self-Taught Evaluatorscited by 70
  6. 2024LLM With Tools: A Surveycited by 60
  7. 2025MCP-RADAR: A Multi-Dimensional Benchmark for Evaluating Tool Use Capabilities in Large Language Modelscited by 48
  8. 2025TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Usecited by 46
  9. 2023ToolTalk: Evaluating Tool-Usage in a Conversational Settingcited by 42
  10. 2025Web-Shepherd: Advancing PRMs for Reinforcing Web Agentscited by 41
  11. 2025Chain-of-Reasoning: Towards Unified Mathematical Reasoning in Large Language Models via a Multi-Paradigm Perspectivecited by 29
  12. 2025DRP: Distilled Reasoning Pruning with Skill-aware Step Decomposition for Efficient Large Reasoning Modelscited by 29
  13. 2025Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMscited by 24
  14. 2024BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignmentcited by 20
  15. 2026SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Modelscited by 20
  16. 2025Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillationcited by 15
  17. 2024WTU-EVAL: A Whether-or-Not Tool Usage Evaluation Benchmark for Large Language Modelscited by 12
  18. 2025Quantifying and Improving the Robustness of Retrieval-Augmented Language Models Against Spurious Features in Grounding Datacited by 10
  19. 2025SCALE: Towards Collaborative Content Analysis in Social Science with Large Language Model Agents and Human Interventioncited by 7
  20. 2025GuiLoMo: Allocating Expert Number and Rank for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectorscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

  1. 2026AndroidDaily: A Verifiable Benchmark for Mobile GUI Agents on Real-World Closed-Source Applicationscited by 8
  2. 2026SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Usecited by 7
  3. 2026Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysiscited by 4
  4. 2026TopoCurate:Modeling Interaction Topology for Tool-Use Agent Trainingcited by 3
  5. 2026Interactive Evaluation Requires a Design Sciencecited by 3
  6. 2026Locating Hidden Failures Makes Long-Horizon Agents More Reliablecited by 1
  7. 2026CRISP: Critical Step Perception for Training Efficient Deep Search Agentscited by 0
  8. 2026More Rejective, Not More Discriminative: The Unit of Verification in Pre-Execution LLM Oversightcited by 0
  9. 2026DENSE: Distilling Agent Trajectories into Evidence-Grounded Shortcut Trees for Self-Refinementcited by 0
  10. 2026On the Effectiveness of Kernel-Level Evidence for Agent Securitycited by 0
  11. 2026PartHackBench: Certified Equal-Progress Stress Tests for Partial-Credit Tool-Agent Evaluationcited by 0
  12. 2026TraceDance: An Automated System for Building Agent Behavior Benchmarks from Real-World Agent Deployment Tracescited by 0
1 more
  1. 2026When Does Correction Become Repair? Mechanistic Auditing of Internal Interventions in Tool-Using LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents
  2. 2026BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure
  3. 2025CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agent
  4. 2026OpaqueToolsBench: Learning Nuances of Tool Behavior Through Interaction
  5. 2025ToolComp: A Multi-Tool Reasoning & Process Supervision Benchmark
  6. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress
  7. 2026ParaRecover: A Process-Level Benchmark for Error Localization and Recovery in Parallel Tool-Use Agents
  8. 2026ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents