# CONFETTI: Conversational Function-Calling Evaluation Through Turn-Level Interactions arXiv: https://arxiv.org/abs/2506.01859 ยท HTML: https://mlsanity.com/paper/2506.01859/ ## Key facts - Cited by 18 papers in the map (data through 14 Oct 2026). - Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. - Builds on 14 of its 24 references that are in the map. - First posted on arXiv 2 Jun 2025. ## Placed in - LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/) ## Builds on (most cited first, 14 of 14) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023) - [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023) - [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023) - [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023) - [Identifying the Risks of LM Agents with an LM-Emulated Sandbox](https://mlsanity.com/paper/2309.15817/) (2023) - [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://mlsanity.com/paper/2303.04048/) (2023) - [ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities](https://mlsanity.com/paper/2408.04682/) (2024) - [AlignScore: Evaluating Factual Consistency with a Unified Alignment Function](https://mlsanity.com/paper/2305.16739/) (2023) - [Evaluating Large Language Models as Generative User Simulators for Conversational Recommendation](https://mlsanity.com/paper/2403.09738/) (2024) - [A Sequence-to-Sequence Model for User Simulation in Spoken Dialogue Systems](https://mlsanity.com/paper/1607.00070/) (2016) - [Human-Centered Design Recommendations for LLM-as-a-Judge](https://mlsanity.com/paper/2407.03479/) (2024) - [PlatoLM: Teaching LLMs in Multi-Round Dialogue via a User Simulator](https://mlsanity.com/paper/2308.11534/) (2023) - [Contextual Dialogue Act Classification for Open-Domain Conversational Agents](https://mlsanity.com/paper/2005.13804/) (2020) ## Built on by (most cited first, 18 of 18) - [Beyond Static Tools: Test-Time Tool Evolution for Scientific Reasoning](https://mlsanity.com/paper/2601.07641/) (2026) - [XGrammar-2: Dynamic and Efficient Structured Generation Engine for Agentic LLMs](https://mlsanity.com/paper/2601.04426/) (2026) - [Ticket-Bench: A Kickoff for Multilingual and Regionalized Agent Evaluation](https://mlsanity.com/paper/2509.14477/) (2025) - [From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents](https://mlsanity.com/paper/2605.15104/) (2026) - [PARL-MT: Learning to Call Functions in Multi-Turn Conversation with Progress Awareness](https://mlsanity.com/paper/2509.23206/) (2025) - [Switchcraft: AI Model Router for Agentic Tool Calling](https://mlsanity.com/paper/2605.07112/) (2026) - [MTVA-Bench: Evaluating the Language Model Inside Cascaded Voice Agents](https://mlsanity.com/paper/2609.20152/) (2026) - [Food4All: An Agentic Framework and Benchmark for Food Resource Navigation with Adaptive User Understanding](https://mlsanity.com/paper/2510.18289/) (2025) - [The Bitter Lesson of Tool Calling](https://mlsanity.com/paper/2608.06370/) (2026) - [Enhancing Tool Calling in LLMs with the International Tool Calling Dataset](https://mlsanity.com/paper/2603.05515/) (2026) - [ToolWeave: Structured Synthesis of Complex Multi-Turn Tool-Calling Dialogues](https://mlsanity.com/paper/2605.12521/) (2026) - [Beyond Text-to-SQL: An Agentic LLM System for Governed Enterprise Analytics APIs](https://mlsanity.com/paper/2605.21027/) (2026) - [When Better Turns Do Not Make Better Agents: Diagnosing the Gap Between Next-Turn Metrics and Workflow Success](https://mlsanity.com/paper/2609.21187/) (2026) - [Tokengeist: Multi-Turn Attribution Tracing in Agentic Conversations](https://mlsanity.com/paper/2607.22610/) (2026) - [HoosierHelp: Benchmarking LLM Agents for Social Service Navigation](https://mlsanity.com/paper/2608.09946/) (2026) - [Can One Adapted Model Do It All? Fine-Tuning Strategy Selection for Customer Support LLMs](https://mlsanity.com/paper/2609.27262/) (2026) - [Evaluating LLMs as Agentic Workflow Executors in Real-World Customer Support Skills](https://mlsanity.com/paper/2610.14346/) (2026) - [RASSG: Repair-Aware Semantic Survival Gating for Streaming Tool Execution](https://mlsanity.com/paper/2610.14659/) (2026) ## Abstract (arXiv) We introduce Conversational Function-Calling Evaluation Through Turn-Level Interactions (CONFETTI), a conversational benchmark1 designed to evaluate the function-calling capabilities and response quality of large language models (LLMs). Current benchmarks lack comprehensive assessment of LLMs in complex conversational scenarios. CONFETTI addresses this gap through 109 human-simulated conversations, comprising 313 user turns and covering 86 APIs. These conversations explicitly target various conversational complexities, such as follow-ups, goal correction and switching, ambiguous and implicit goals. We perform off-policy turn-level evaluation using this benchmark targeting function-calling. Our benchmark also incorporates dialog act annotations to assess agent responses. We evaluate a series of state-of-the-art LLMs and analyze their performance with respect to the number of available APIs, conversation lengths, and chained function calling. Our results reveal that while some models are able to handle long conversations, and leverage more than 20+ APIs successfully, other models struggle with longer context or when increasing the number of APIs. We also report that the performance on chained function-calls is severely limited across the models. Overall, the top performing models on CONFETTI are Nova Pro (40.01%), Claude Sonnet v3.5 (35.46%) and Llama 3.1 405B (33.19%) followed by command-r-plus (31.18%) and Mistral-Large-2407 (30.07%).