mlsanityData to 7 Oct 2026

SEAL: Suite for Evaluating API-use of LLMs

arXiv 2409.15523 ↗First posted 23 Sep 2024cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have limitations in handling tasks that require real-time access to external APIs. While several benchmarks like ToolBench and APIGen have been developed to assess LLMs' API-use capabilities, they often suffer from issues such as lack of generalizability, limited multi-step reasoning coverage, and instability due to real-time API fluctuations. In this paper, we introduce SEAL, an end-to-end testbed designed to evaluate LLMs in real-world API usage. SEAL standardizes existing benchmarks, integrates an agent system for testing API retrieval and planning, and addresses the instability of real-time APIs by introducing a GPT-4-powered API simulator with caching for deterministic evaluations. Our testbed provides a comprehensive evaluation pipeline that covers API retrieval, API calls, and final responses, offering a reliable framework for structured performance comparison in diverse real-world scenarios. SEAL is publicly available, with ongoing updates for new benchmarks.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 31 of its 33 references that are in the map. First posted on arXiv 23 Sep 2024.

Builds on

31 of 33 references.

References · 31

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  5. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  6. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  7. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  8. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
  9. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  10. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  11. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  12. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 272
19 more
  1. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  2. 2024Tool Learning with Large Language Models: A Surveycited by 221
  3. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  4. 2024AI Agents That Mattercited by 199
  5. 2023RestGPT: Connecting Large Language Models with Real-World RESTful APIscited by 129
  6. 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Modelscited by 129
  7. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  8. 2024AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Callscited by 121
  9. 2024Small LLMs Are Weak Tool Learners: A Multi-LLM Agentcited by 92
  10. 2024Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarioscited by 85
  11. 2023Lemur: Harmonizing Natural Language and Code for Language Agentscited by 71
  12. 2024MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learningcited by 66
  13. 2023TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usagecited by 54
  14. 2023ChatCoT: Tool-Augmented Chain-of-Thought Reasoning on Chat-based Large Language Modelscited by 43
  15. 2023TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Systemscited by 39
  16. 2024m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Taskscited by 38
  17. 2024TOOLVERIFIER: Generalization to New Tools via Self-Verificationcited by 23
  18. 2024STRIDE: A Tool-Assisted LLM Agent Framework for Strategic and Interactive Decision-Makingcited by 22
  19. 2024APIGen: Generative API Method Recommendationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025DICE-BENCH: Evaluating the Tool-Use Capabilities of Large Language Models in Multi-Round, Multi-Party Dialoguescited by 6
  2. 2026RAG Strategies for Natural Language-Based SQL Query and REST API Call Generationcited by 0