Towards Better Agents for Multi-Turn User Interaction: The Next User Turn Is More Than Context
Not on the map. Why?
Abstract (arXiv)
User-facing tool agents must coordinate dialogue and tool use as user goals unfold over multiple turns. Yet interactive reinforcement learning typically reduces each rollout to a terminal reward, assigning the same credit to effective elicitation, errors, and later repair. The next user turn is more than context: it also provides noisy, temporally local evidence about the preceding user-to-user segment. We introduce Feedback-Aware Credit Assignment (textscFACA), which aligns each reaction with that segment, derives a locally normalized reaction advantage, and adds it to verified terminal outcome advantage without an extra critic or rollout. Against an outcome-only Interactive GRPO control matched in simulator, visible dialogue, initialization, rollout, and optimization, textscFACA improves the nine-domain τ-family average across three independently trained runs by 5.91 and 10.22 percentage points at 8B and 14B, respectively. Gains concentrate in Telecom; at 8B, randomizing reaction polarity removes the Telecom gain. The same ordering holds zero-shot on Pare-Bench and Co-Gym. These results demonstrate that next-turn user reactions provide actionable local credit for improving multi-turn user-interacting agents.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 37 of its 51 references that are in the map. First posted on arXiv 18 Aug 2026.
Builds on
37 of 51 references.
References · 37
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025Qwen3 Technical Reportcited by 8,820
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
- 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
- 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
- 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 497
- 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
- 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 167
- 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 118
- 2025Agent Lightning: Train ANY AI Agents with Reinforcement Learningcited by 109
- 2024SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agentscited by 93
25 more
- 2025Process Reward Models for LLM Agents: Practical Framework and Directionscited by 85
- 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignmentcited by 79
- 2025UserBench: An Interactive Gym Environment for User-Centric Agentscited by 76
- 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Modelscited by 75
- 2024Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agentscited by 74
- 2024Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaborationcited by 65
- 2025RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agentscited by 58
- 2025Simulating Environments with Reasoning Models for Agent Trainingcited by 57
- 2025Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agentscited by 48
- 2026τ-Knowledge: Evaluating Conversational Agents over Unstructured Knowledgecited by 39
- 2025Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translationcited by 37
- 2025MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool usecited by 32
- 2025UserRL: Training Interactive User-Centric Agent via Reinforcement Learningcited by 32
- 2026TIPS: Turn-Level Information-Potential Reward Shaping for Search-Augmented LLMscited by 24
- 2025SOTOPIA-Omega: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agentscited by 16
- 2026Proactive Agent Research Environment: Simulating Active Users to Evaluate Proactive Assistantscited by 15
- 2026InfoPO: Information-Driven Policy Optimization for User-Centric Agentscited by 12
- 2026SEAL: Synergistic Co-Evolution of Agents and Learning Environmentscited by 12
- 2025Reinforcement Learning from User Feedbackcited by 8
- 2026AT2PO: Agentic Turn-based Policy Optimization via Tree Searchcited by 8
- 2024Aligning Language Models Using Follow-up Likelihood as Reward Signalcited by 6
- 2025LAM SIMULATOR: Advancing Data Generation for Large Action Model Training via Online Exploration and Trajectory Feedbackcited by 6
- 2026Rethinking the Role of Entropy in Optimizing Tool-Use Behaviors for Large Language Model Agentscited by 5
- 2026Self-Induced Outcome Potential: Turn-Level Credit Assignment for Agents without Verifierscited by 4
- 2026EIBench: A Simulator-Based Benchmark and Turn-Credit RL for Emotion Managementcited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training
- 2026Self-Evolution for Multi-Turn Tool-Calling Agents via Divergence-Point Preference Learning
- 2025A Practitioner's Guide to Multi-turn Agentic Reinforcement Learning
- 2026Intent Speaks Louder: Controllable User Simulation Beyond Response Imitation
- 2026From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agents
- 2026On Effectiveness and Efficiency of Agentic Tool-calling and RL Training
- 2025Process-Supervised Reinforcement Learning for Interactive Multimodal Tool-Use Agents
- 2026RC-GRPO: Reward-Conditioned Group Relative Policy Optimization for Multi-Turn Tool Calling Agents