mlsanityData to 7 Oct 2026

ToolRM: Outcome Reward Models for Tool-Calling Large Language Models

arXiv 2509.11963 ↗First posted 15 Sep 2025cs.CLcited by 16

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 2 more
Abstract (arXiv)

As large language models (LLMs) increasingly interact with external tools, reward modeling for tool use has emerged as a critical yet underexplored area of research. Existing reward models, trained primarily on natural language outputs, struggle to evaluate tool-based reasoning and execution. To quantify this gap, we introduce FC-RewardBench, the first benchmark to systematically evaluate reward models in tool-calling scenarios. Our analysis shows that current reward models frequently miss key signals of effective tool use, highlighting the need for domain-specific modeling. We address this by proposing a training framework for outcome reward models using data synthesized from permissively licensed, open-weight LLMs. We introduce ToolRM - a suite of reward models for tool-use ranging from 1.7B to 14B parameters. Across diverse settings, these models consistently outperform general-purpose baselines. Notably, they achieve up to a 25% improvement with Best-of-N sampling, while also improving robustness to input noise, enabling effective data filtering, and supporting RL-training of policy models.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 48 of its 71 references that are in the map. First posted on arXiv 15 Sep 2025.

Builds on

48 of 71 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution25 Feb
  2. 2025Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning10 Feb
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  4. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  5. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  6. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 42

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2023Mistral 7Bcited by 3,065
  8. 2024Qwen2 Technical Reportcited by 2,076
  9. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  10. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  11. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  12. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
30 more
  1. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  2. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 377
  3. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  4. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  5. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 307
  6. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 214
  7. 2024ToolACE: Winning the Points of LLM Function Callingcited by 204
  8. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  9. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 174
  10. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  11. 2025ToRL: Scaling Tool-Integrated RLcited by 166
  12. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 158
  13. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  14. 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Modelscited by 129
  15. 2023Solving Math Word Problems by Combining Language Models With Symbolic Solverscited by 124
  16. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 108
  17. 2023CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Modelscited by 90
  18. 2024Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmarkcited by 62
  19. 2024Hammer: Robust Function-Calling for On-Device Language Models via Function Maskingcited by 53
  20. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  21. 2024Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Taskscited by 52
  22. 2025SMART: Self-Aware Agent for Tool Overuse Mitigationcited by 52
  23. 2024NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Callscited by 45
  24. 2023Aligning Large Language Models with Human: A Surveycited by 31
  25. 2023Toolink: Linking Toolkit Creation and Using through Chain-of-Solving on Open-Source Modelcited by 29
  26. 2024StepTool: Enhancing Multi-Step Tool Usage in LLMs via Step-Grained Reinforcement Learningcited by 22
  27. 2023Formally Specifying the High-Level Behavior of LLM-Based Agentscited by 15
  28. 2025iTool: Reinforced Fine-Tuning with Dynamic Deficiency Calibration for Advanced Tool Usecited by 7
  29. 2025CodeTool: Enhancing Programmatic Tool Invocation of LLMs via Process Supervisioncited by 6
  30. 2024SmartFlow: Robotic Process Automation using LLMscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 12.

16 papers.

Sort:

By area:

  1. 2025Feedback Descent: Open-Ended Text Optimization via Pairwise Comparisoncited by 42
  2. 2026SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Modelscited by 20
  3. 2025AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agentcited by 17
  4. 2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 16
  5. 2026ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agentscited by 13
  6. 2026Toward Efficient Agents: Memory, Tool learning, and Planningcited by 11
  7. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  8. 2026Agentic Uncertainty Reveals Agentic Overconfidencecited by 7
  9. 2026LOGIGEN: Logic-Driven Generation of Verifiable Agentic Taskscited by 5
  10. 2025ToolRM: Towards Agentic Tool-Use Reward Modelingcited by 4
  11. 2026What If We Allocate Test-Time Compute Adaptively?cited by 4
  12. 2026UniToolCall: Unifying Tool-Use Representation, Data, and Evaluation for LLM Agentscited by 3
4 more
  1. 2026RLAR: An Agentic Reward System for Multi-task Reinforcement Learning on Large Language Modelscited by 1
  2. 2026SCI-PRM: A Tool Aware Process Reward Model for Scientific Reasoning Verificationcited by 1
  3. 2026Eta Given Delta: Defining LLM Tool Efficiency With Marginal Tool Utilitycited by 0
  4. 2026When Does Correction Become Repair? Mechanistic Auditing of Internal Interventions in Tool-Using LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environments
  2. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settings
  3. 2024RewardBench: Evaluating Reward Models for Language Modeling
  4. 2024ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios
  5. 2023Tool-Augmented Reward Modeling
  6. 2025OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning
  7. 2025LongRM: Revealing and Unlocking the Context Boundary of Reward Modeling
  8. 2026PARM: Pipeline-Adapted Reward Model