# FunctionChat-Bench: Comprehensive Evaluation of Language Models' Generative Capabilities in Korean Tool-use Dialogs

arXiv: https://arxiv.org/abs/2411.14054 · HTML: https://mlsanity.com/paper/2411.14054/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use.
- Builds on 7 of its 9 references that are in the map.
- First posted on arXiv 21 Nov 2024.

## Placed in

- LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/)

## Builds on (most cited first, 7 of 7)

- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction](https://mlsanity.com/paper/2305.18752/) (2023)
- [MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use](https://mlsanity.com/paper/2310.03128/) (2023)
- [RestGPT: Connecting Large Language Models with Real-World RESTful APIs](https://mlsanity.com/paper/2306.06624/) (2023)
- [ToolTalk: Evaluating Tool-Usage in a Conversational Setting](https://mlsanity.com/paper/2311.10775/) (2023)

## Built on by (most cited first, 8 of 8)

- [Kanana: Compute-efficient Bilingual Language Models](https://mlsanity.com/paper/2502.18934/) (2025)
- [DiaTool-DPO: Multi-Turn Direct Preference Optimization for Tool-Augmented Large Language Models](https://mlsanity.com/paper/2504.02882/) (2025)
- [SLM-Based Agentic AI with P-C-G: Optimized for Korean Tool Use](https://mlsanity.com/paper/2509.19369/) (2025)
- [Latent Preference Modeling for Multi-Session Personalized Tool Calling](https://mlsanity.com/paper/2604.17886/) (2026)
- [Arabic Prompts with English Tools: A Benchmark](https://mlsanity.com/paper/2601.05101/) (2026)
- [Mi:dm K 2.5 Pro](https://mlsanity.com/paper/2603.18788/) (2026)
- [Multi-Step Tool-Calling over Korean Open Public APIs: A Benchmark and a Data-Synthesis Recipe](https://mlsanity.com/paper/2609.05395/) (2026)
- [Selection Is Retrieval, Abstention Is Not: On-Device Tool Routing over 70 Korean-English Actions](https://mlsanity.com/paper/2609.18672/) (2026)

## Abstract (arXiv)

This study investigates language models' generative capabilities in tool-use dialogs. We categorize the models' outputs in tool-use dialogs into four distinct types: Tool Call, Answer Completion, Slot Question, and Relevance Detection, which serve as aspects for evaluation. We introduce FunctionChat-Bench, comprising 700 evaluation items and automated assessment programs. Using this benchmark, we evaluate several language models that support function calling. Our findings indicate that while language models may exhibit high accuracy in single-turn Tool Call scenarios, this does not necessarily translate to superior generative performance in multi-turn environments. We argue that the capabilities required for function calling extend beyond generating tool call messages; they must also effectively generate conversational messages that engage the user.