Synthesize and Reward -- Reinforcement Learning for Multi-Step Tool Use in Live Environments
On the map: LLM agents and tool use → Reinforcement learning for LLM agents
Abstract (arXiv)
Training LLMs to orchestrate multi-step tool calls is held back by three coupled obstacles: realistic stateful execution environments are costly to build, synthetic training queries are often detached from the server's actual state (so the generated tool calls fail to execute), and recall-based RL rewards incentivize verbose tool-calling patterns. We present PROVE (Programmatic Rewards On Verified Environments), a framework with three contributions: (1) a library of 20 stateful MCP (Model Context Protocol) servers exposing 343 tools, enabling live-execution RL training with session-scoped state isolation; (2) a state-machine data synthesis pipeline that generates multi-turn tool-call trajectories grounded in live-sampled server state, so generated queries reference entities that actually exist; and (3) a multi-component programmatic reward with an adaptive efficiency penalty that counters the verbosity incentive of recall-based rewards. We train four models (Qwen3-4B, Qwen3-8B, Qwen2.5-7B, Granite-4.1-8B) with GRPO on the resulting ~13K training examples. On BFCL Multi-Turn, tau2-bench, and T-Eval, PROVE yields improvements of up to +10.2, +6.8, and +6.5 points respectively, demonstrating that this framework yields consistent gains on multi-step tool orchestration across two model families.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 14 of its 26 references that are in the map. First posted on arXiv 2 Jun 2026.
Builds on
14 of 26 references.
Closest ancestors · 4
Same direction, newest first
- 2025Tool Zero: Training Tool-Augmented LLMs via Pure RL from Scratch2 Nov
- 2025Tool-R1: Sample-Efficient Reinforcement Learning for Agentic Tool Use16 Sep
- 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMs15 Apr
- 2024StepTool: Enhancing Multi-Step Tool Usage in LLMs via Step-Grained Reinforcement Learning10 Oct
Other references · 10
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
- 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 111
- 2023T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Stepcited by 93
- 2025ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenariocited by 55
- 2025When2Call: When (not) to Call Toolscited by 42
- 2025Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translationcited by 37
- 2025ToolACE-MT: Non-Autoregressive Generation for Agentic Multi-Turn Interactioncited by 11
- 2026Simulating Complex Multi-Turn Tool Calling Interactions in Stateless Execution Environmentscited by 3
- 2026Training LLMs for Multi-Step Tool Orchestration with Constrained Data Synthesis and Graduated Rewardscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Controllable and Verifiable Tool-Use Data Synthesis for Agentic Reinforcement Learning
- 2026RODS: Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents
- 2026Critical-State RL: Diagnosing Trainable States for Multi-Turn Tool Use
- 2026On Effectiveness and Efficiency of Agentic Tool-calling and RL Training
- 2025RLFactory: A Plug-and-Play Reinforcement Learning Post-Training Framework for LLM Multi-Turn Tool-Use
- 2026Beyond Next-Token Prediction: An RLVR Proof of Concept for Tool-Use Agents on Atlassian Workflows
- 2026CM2: Reinforcement Learning with Checklist Rewards for Multi-Turn and Multi-Step Agentic Tool Use
- 2025ToolRL: Reward is All Tool Learning Needs