mlsanityData to 14 Oct 2026

Process-Supervised Reinforcement Learning for Interactive Multimodal Tool-Use Agents

arXiv 2509.14480 ↗First posted 17 Sep 2025cs.CL cs.AI cs.MAcited by 12

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Effective interactive tool use requires agents to master Tool Integrated Reasoning (TIR): a complex process involving multi-turn planning and long-context dialogue management. To train agents for this dynamic process, particularly in multi-modal contexts, we introduce a sandbox environment for reinforcement learning (RL) that supports interleaved speech-text rollouts. Our core strategy, Turn-level Adjudicated Reinforcement Learning (TARL), addresses the challenge of credit assignment in long-horizon tasks by employing a Large Language Model (LLM) as a judge to provide turn-level evaluation. To enhance exploration, we integrate a mixed-task training curriculum with mathematical reasoning problems. This unified approach boosts the task pass rate on the text-based τ-bench by over 6% compared to strong RL baselines. Crucially, we demonstrate our framework's suitability for fine-tuning a multi-modal foundation model for agentic tasks. By training a base multi-modal LLM on interleaved speech-text rollouts, we equip it with tool-use abilities, paving the way for more natural, voice-driven interactive agents.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 17 of its 36 references that are in the map. First posted on arXiv 17 Sep 2025.

Builds on

17 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Process Reward Models for LLM Agents: Practical Framework and Directions14 Feb

Other references · 16

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Let's Verify Step by Stepcited by 3,789
  6. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  7. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  8. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  9. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
  10. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
  11. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 415
  12. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
4 more
  1. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 167
  2. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertaintycited by 117
  3. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoningcited by 102
  4. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 95

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 10.

12 papers.

  1. 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Modelscited by 23
  2. 2025VoiceAgentBench: Are Voice Assistants ready for agentic tasks?cited by 19
  3. 2025Klear-AgentForge: Forging Agentic Intelligence through Posttraining Scalingcited by 9
  4. 2026OASES: Outcome-Aligned Search-Evaluation Co-Training for Agentic Searchcited by 7
  5. 2026Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix Itcited by 7
  6. 2026Credit Without Ground Truth: Auditing Step-Level Credit Assignment in LLM Agents Against Executed Replaycited by 7
  7. 2026Harmonizing Dense and Sparse Signals in Multi-turn RL: Dual-Horizon Credit Assignment for Industrial Sales Agentscited by 2
  8. 2026AgentCVR: Active Multi-Agent Cross-Video Reasoning via Script-Simulated Reinforcement Learningcited by 1
  9. 2026CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agentscited by 1
  10. 2026From Correctness to Preference: A Framework for Personalized Agentic Reinforcement Learningcited by 0
  11. 2026CityPlanner: A Sandbox Agent for Executable Urban Planningcited by 0
  12. 2026Of Errors and Echoes: Responsibility-Aware Credit Assignment for Tool-Use Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026IAPO: Input Attribution-Aware Policy Optimization for Tool Use in Small Multimodal Agents
  2. 2025Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning
  3. 2026On Effectiveness and Efficiency of Agentic Tool-calling and RL Training
  4. 2026ASTRA: Automated Synthesis of agentic Trajectories and Reinforcement Arenas
  5. 2025A Practitioner's Guide to Multi-turn Agentic Reinforcement Learning
  6. 2025Multi-Agent Tool-Integrated Policy Optimization
  7. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning
  8. 2025Agent-as-Tool: A Study on the Hierarchical Decision Making with Reinforcement Learning