mlsanityData to 7 Oct 2026

UserBench: An Interactive Gym Environment for User-Centric Agents

arXiv 2507.22034 ↗First posted 29 Jul 2025cs.AI cs.CL cs.LGcited by 69

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Large Language Models (LLMs)-based agents have made impressive progress in reasoning and tool use, enabling them to solve complex tasks. However, their ability to proactively collaborate with users, especially when goals are vague, evolving, or indirectly expressed, remains underexplored. To address this gap, we introduce UserBench, a user-centric benchmark designed to evaluate agents in multi-turn, preference-driven interactions. UserBench features simulated users who start with underspecified goals and reveal preferences incrementally, requiring agents to proactively clarify intent and make grounded decisions with tools. Our evaluation of leading open- and closed-source LLMs reveals a significant disconnect between task completion and user alignment. For instance, models provide answers that fully align with all user intents only 20% of the time on average, and even the most advanced models uncover fewer than 30% of all user preferences through active interaction. These results highlight the challenges of building agents that are not just capable task executors, but true collaborative partners. UserBench offers an interactive environment to measure and advance this critical capability.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 33 of its 42 references that are in the map. First posted on arXiv 29 Jul 2025.

Builds on

33 of 42 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay4 Apr
  2. 2025MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents3 Mar
  3. 2024Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance16 Oct
  4. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systems5 Sep
  5. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun
  6. 2024Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents14 Feb
  7. 2024Investigate-Consolidate-Exploit: A General Strategy for Inter-Task Agent Self-Evolution25 Jan

Other references · 26

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  3. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  4. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  5. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  6. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  7. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  8. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  9. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  10. 2023Large Language Models as Tool Makerscited by 293
  11. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  12. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 272
14 more
  1. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  2. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  3. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 188
  4. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 142
  5. 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 124
  6. 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsetscited by 122
  7. 2025Acting Less is Reasoning More! Teaching Model to Act Efficientlycited by 80
  8. 2024Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questionscited by 70
  9. 2024Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Trainingcited by 53
  10. 2025SMART: Self-Aware Agent for Tool Overuse Mitigationcited by 52
  11. 2023ToolTalk: Evaluating Tool-Usage in a Conversational Settingcited by 42
  12. 2024Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agentscited by 35
  13. 2023Toolink: Linking Toolkit Creation and Using through Chain-of-Solving on Open-Source Modelcited by 29
  14. 2025ModelingAgent: Bridging LLMs and Mathematical Modeling for Real-World Challengescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 59.

69 papers.

Sort:

By area:

  1. 2026EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesiscited by 46
  2. 2025VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applicationscited by 45
  3. 2026Mind the Sim2Real Gap in User Simulation for Agentic Taskscited by 44
  4. 2026DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraintscited by 43
  5. 2025Training Proactive and Personalized LLM Agentscited by 42
  6. 2025CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agentscited by 32
  7. 2025UserRL: Training Interactive User-Centric Agent via Reinforcement Learningcited by 27
  8. 2025Enterprise Deep Research: Steerable Multi-Agent Deep Research for Enterprise Analyticscited by 20
  9. 2026AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Conditioncited by 19
  10. 2025RecoWorld: Building Simulated Environments for Agentic Recommender Systemscited by 18
  11. 2026HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?cited by 16
  12. 2026EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settingscited by 14
38 more
  1. 2026PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environmentscited by 13
  2. 2025InteractComp: Evaluating Search Agents With Ambiguous Queriescited by 12
  3. 2026When Users Change Their Mind: Evaluating Interruptible Agents in Long-Horizon Web Navigationcited by 12
  4. 2026Trajectory2Task: Training Robust Tool-Calling Agents with Synthesized Yet Verifiable Data for Complex User Intentscited by 11
  5. 2026PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystemscited by 10
  6. 2025MindEval: Benchmarking Language Models on Multi-turn Mental Health Supportcited by 9
  7. 2025COMPASS: Benchmarking Constrained Optimization in LLM Agentscited by 8
  8. 2026TRIP-Bench: A Benchmark for Long-Horizon Interactive Agents in Real-World Scenarioscited by 8
  9. 2026InfoPO: Information-Driven Policy Optimization for User-Centric Agentscited by 8
  10. 2026Towards Natural Personalization: Evaluating Long-Horizon Preference Following in Personalized User-LLM Interactionscited by 8
  11. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  12. 2026U-Fold: Dynamic Intent-Aware Context Folding for User-Centric Agentscited by 7
  13. 2026Drift-Bench: Diagnosing Cooperative Breakdowns in LLM Agents under Input Faults via Multi-Turn Interactioncited by 7
  14. 2026PrefIx: Understand and Adapt to User Preference in Human-Agent Interactioncited by 7
  15. 2026AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraintscited by 6
  16. 2026Beyond Perfect APIs: A Comprehensive Evaluation of LLM Agents Under Real-World API Complexitycited by 5
  17. 2026LHAW: Controllable Underspecification for Long-Horizon Taskscited by 5
  18. 2026Asymmetric Actor-Critic for Multi-turn LLM Agentscited by 4
  19. 2026ProactBench: Beyond What The User Asked Forcited by 4
  20. 2026CirrusBench: Evaluating LLM-based Agents Beyond Correctness in Real-World Cloud Service Environmentscited by 3
  21. 2026Interactive Evaluation Requires a Design Sciencecited by 3
  22. 2026Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimizationcited by 2
  23. 2026Code2Math: Can Your Code Agent Effectively Evolve Math Problems Through Exploration?cited by 2
  24. 2026Learning to Interrupt in Language-based Multi-agent Communicationcited by 2
  25. 2026PRIME: Training Free Proactive Reasoning via Iterative Memory Evolution for User-Centric Agentcited by 2
  26. 2026Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?cited by 2
  27. 2026AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Usecited by 2
  28. 2025ShortageSim: Simulating Drug Shortages under Information Asymmetrycited by 1
  29. 2026Long-term Task-oriented Agent: Proactive Long-term Intent Maintenance in Dynamic Environmentscited by 1
  30. 2026IntentRL: Training Proactive User-intent Agents for Open-ended Deep Research via Reinforcement Learningcited by 1
  31. 2026Beyond Words: Evaluating and Bridging Epistemic Divergence in User-Agent Interaction via Theory of Mindcited by 1
  32. 2026PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Modelscited by 1
  33. 2026UserHarness: Harnessing User Minds for Stronger Agent Theory-of-Mindcited by 1
  34. 2026WRIT: Write-Read Intensive Trajectory Synthesis for Multi-Turn User-Facing Agentscited by 1
  35. 2026CollabSkill: Evaluating Human-Agent Collaboration On Real-World Taskscited by 1
  36. 2026T1-Bench: Benchmarking Multi-Scenario Agents in Real-World Domainscited by 1
  37. 2026Trip+: Benchmarking Agents in Personalized Interactive Travel Planningcited by 1
  38. 2026Towards Better Agents for Multi-Turn User Interaction: The Next User Turn Is More Than Contextcited by 1

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026VitaBench 2.0: Evaluating Personalized and Proactive Agents in Long-Term User Interactions
  2. 2024Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration
  3. 2026UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training
  4. 2026π-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows
  5. 2026ProAgentBench: Evaluating LLM Agents for Proactive Assistance with Real-World Data
  6. 2026Interactive Task Alignment as a POMDP
  7. 2026ProACT: Towards Breakdown-Aware Proactive Agent in Multi-User Collaboration
  8. 2026RideWay: Benchmarking Efficient Task Completion for Tool-Using Language Agents