Rethinking Stateful Tool Use in Multi-Turn Dialogues: Benchmarks and Challenges

Existing benchmarks that assess Language Models (LMs) as Language Agents (LAs) for tool use primarily focus on stateless, single-turn interactions or partial evaluations, such as tool selection in a single turn, overlooking the inherent stateful nature of interactions in multi-turn applications. To fulfill this gap, we propose textttDialogTool, a multi-turn dialogue dataset with stateful tool interactions considering the whole life cycle of tool use, across six key tasks in three stages: 1) textittool creation; 2) textittool utilization: tool awareness, tool selection, tool execution; and 3) textitrole-consistent response: response generation and role play. Furthermore, we build textttVirtualMobile -- an embodied virtual mobile evaluation environment to simulate API calls and assess the robustness of the created APIsfootnoteWe will use tools and APIs alternatively, there are no significant differences between them in this paper.. Taking advantage of these artifacts, we conduct comprehensive evaluation on 13 distinct open- and closed-source LLMs and provide detailed analysis at each stage, revealing that the existing state-of-the-art LLMs still cannot perform well to use tools over long horizons.