mlsanityData to 7 Oct 2026

CallNavi, A Challenge and Empirical Study on LLM Function Calling and Routing

arXiv 2501.05255 ↗First posted 9 Jan 2025cs.SE cs.CLcited by 6

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

API-driven chatbot systems are increasingly integral to software engineering applications, yet their effectiveness hinges on accurately generating and executing API calls. This is particularly challenging in scenarios requiring multi-step interactions with complex parameterization and nested API dependencies. Addressing these challenges, this work contributes to the evaluation and assessment of AI-based software development through three key advancements: (1) the introduction of a novel dataset specifically designed for benchmarking API function selection, parameter generation, and nested API execution; (2) an empirical evaluation of state-of-the-art language models, analyzing their performance across varying task complexities in API function generation and parameter accuracy; and (3) a hybrid approach to API routing, combining general-purpose large language models for API selection with fine-tuned models and prompt engineering for parameter generation. These innovations significantly improve API execution in chatbot systems, offering practical methodologies for enhancing software design, testing, and operational workflows in real-world software engineering contexts.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 26 of its 44 references that are in the map. First posted on arXiv 9 Jan 2025.

Builds on

26 of 44 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024ToolACE: Winning the Points of LLM Function Calling2 Sep
  2. 2023Reverse Chain: A Generic-Rule for LLMs to Master Multi-API Planning6 Oct
  3. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  4. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases8 Jun
  5. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs14 Apr

Other references · 21

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  7. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  8. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  9. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  10. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  11. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  12. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
9 more
  1. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 108
  2. 2024Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generationcited by 99
  3. 2017Abstract Syntax Networks for Code Generation and Semantic Parsingcited by 92
  4. 2023Is ChatGPT the Ultimate Programming Assistant -- How far is it?cited by 72
  5. 2024"We Need Structured Output": Towards User-centered Constraints on Large Language Model Outputcited by 46
  6. 2024CodeAgent: Autonomous Communicative Agents for Code Reviewcited by 33
  7. 2021Revisiting, Benchmarking and Exploring API Recommendation: How Far Are We?cited by 21
  8. 2024IFAR: Multi-Perspective and Multi-Level Causal Discovery with LLMscited by 9
  9. 2017Programming Bots by Synthesizing Natural Language Expressions into API Invocationscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  2. 2025Towards Reliable Benchmarking: A Contamination Free, Controllable Evaluation Framework for Multi-step LLM Function Callingcited by 13
  3. 2025Doc2Agent: Scalable Generation of Tool-Using Agents from API Documentationcited by 4
  4. 2026AgentChaos: Chaos Engineering for Agent Systems via Programmatic Fault Injectioncited by 2
  5. 2026SAGE: Tool-Augmented LLM Task Solving Strategies in Scalable Multi-Agent Environmentscited by 1
  6. 2026The Remarkable Effectiveness of Providing AI Agents with Natural Language Tools: A Replication Study Validating NLT Performance Across 14 Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Octopus: On-device language model for function calling of software APIs
  2. 2024NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls
  3. 2026Beyond Perfect APIs: A Comprehensive Evaluation of LLM Agents Under Real-World API Complexity
  4. 2024Benchmarking Floworks against OpenAI & Anthropic: A Novel Framework for Enhanced LLM Function Calling
  5. 2024Designing Algorithms Empowered by Language Models: An Analytical Framework, Case Studies, and Insights
  6. 2025Evaluating LLMs on Sequential API Call Through Automated Test Generation
  7. 2024Creating an LLM-based AI-agent: A high-level methodology towards enhancing LLMs with APIs
  8. 2024Harnessing LLMs for API Interactions: A Framework for Classification and Synthetic Data Generation