mlsanityData to 14 Oct 2026

Live API-Bench: 2500+ Live APIs for Testing Multi-Step Tool Calling

arXiv 2506.11266 ↗First posted 12 Jun 2025cs.SE cs.AIcited by 4

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Large language models (LLMs) increasingly rely on external tools and APIs to execute complex tasks specified in natural language. Evaluating such tool calling capabilities in realistic enterprise settings is challenging: APIs are often proprietary, heterogeneous, and difficult to share, limiting reproducible benchmarks. To address this, we introduce Live API Bench, a comprehensive benchmark constructed by transforming NL2SQL datasets into interactive API environments. Our pipeline converts SQL queries from BIRD SQL into executable API sequences across three formulations SLOT, SEL, and REST covering minimal general purpose operations, domain specific multi step tasks, and function oriented RESTful interactions, respectively. The benchmark spans 11 databases with over 2,500 invocable tools, paired with human authored queries, ground truth API sequences, and verified final answers. Live API Bench enables systematic evaluation of core challenges in tool use, including error handling, sequential reasoning, parameter generation, response parsing, and robustness across diverse domains. We evaluate 10 LLMs and 4 ReACT agents, observing low task completion rates (7 to 47pct), which improve modestly to 50pct under interactive agent settings, highlighting substantial scope for improving LLM tool calling performance. We release all code and data associated with this paper.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 25 of its 48 references that are in the map. First posted on arXiv 12 Jun 2025.

Builds on

25 of 48 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario17 Jan
  2. 2024NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls4 Sep
  3. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets26 Jun
  4. 2024Octopus: On-device language model for function calling of software APIs2 Apr
  5. 2024API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMs23 Feb
  6. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  7. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases8 Jun
  8. 2023Gorilla: Large Language Model Connected with Massive APIs24 May
  9. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs14 Apr

Other references · 16

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  3. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  4. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  5. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  6. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  7. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 687
  8. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
  9. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 363
  10. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 158
  11. 2024DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?cited by 125
  12. 2024Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Modelscited by 96
4 more
  1. 2024AutoKaggle: A Multi-Agent Framework for Autonomous Data Science Competitionscited by 70
  2. 2021Revisiting, Benchmarking and Exploring API Recommendation: How Far Are We?cited by 21
  3. 2024APITestGenie: Automated API Test Generation through Generative AIcited by 7
  4. 2025RefTool: Reference-Guided Tool Creation for Knowledge-Intensive Reasoningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystemscited by 11
  2. 2026Agent Lifecycle Toolkit (ALTK): Reusable Middleware Components for Robust AI Agentscited by 0
  3. 2026MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimescited by 0
  4. 2026Toward Interactive Understanding of Code APIscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios
  2. 2025Evaluating LLMs on Sequential API Call Through Automated Test Generation
  3. 2024ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents
  4. 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models
  5. 2025The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution
  6. 2025MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers
  7. 2026APIFlow-Bench: Measuring Whether Agents Survive Long, Dependent API Workflows
  8. 2024Harnessing LLMs for API Interactions: A Framework for Classification and Synthetic Data Generation