mlsanityData to 14 Oct 2026

Benchmarking Web API Integration Code Generation

arXiv 2509.20172 ↗First posted 24 Sep 2025cs.SE cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

API integration is a cornerstone of our digital infrastructure, enabling software systems to connect and interact. However, as shown by many studies, writing or generating correct code to invoke APIs, particularly web APIs, is challenging. Although large language models (LLMs) have become popular in software development, their effectiveness in automating the generation of web API integration code remains unexplored. In order to address this, we present WAPIIBench, a dataset and evaluation pipeline designed to assess the ability of LLMs to generate web API invocation code. Our experiments with several open-source LLMs reveal that generating API invocations poses a significant challenge, resulting in hallucinated endpoints, incorrect argument usage, and other errors. None of the evaluated open-source models was able to solve more than 40% of the tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 70 references that are in the map. First posted on arXiv 24 Sep 2025.

Builds on

31 of 70 references.

References · 31

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  6. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  7. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  8. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  9. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  10. 2023StarCoder: may the source be with you!cited by 799
  11. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  12. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
19 more
  1. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  2. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 431
  3. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  4. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  5. 2017Lexically Constrained Decoding for Sequence Generation Using Grid Beam Searchcited by 237
  6. 2025Survey on Evaluation of LLM-based Agentscited by 204
  7. 2022Do Users Write More Insecure Code with AI Assistants?cited by 192
  8. 2022Out of the BLEU: how should we assess quality of the Code Generation models?cited by 65
  9. 2024Bugs in Large Language Models Generated Code: An Empirical Studycited by 60
  10. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 53
  11. 2023Unveiling Memorization in Code Modelscited by 30
  12. 2024On Mitigating Code LLM Hallucinations with API Documentationcited by 21
  13. 2024When to Stop? Towards Efficient Code Generation in LLMs with Excess Token Preventioncited by 21
  14. 2023Private-Library-Oriented Code Generation with Large Language Modelscited by 10
  15. 2024Applying RLAIF for Code Generation with API-usage in Lightweight LLMscited by 10
  16. 2022Dialog2API: Task-Oriented Dialogue with API Description and Example Programscited by 9
  17. 2025ToolCoder: A Systematic Code-Empowered Tool Learning Framework for Large Language Modelscited by 8
  18. 2021Compositional Generalization for Natural Language Interfaces to Web APIscited by 7
  19. 2023Pop Quiz! Do Pre-trained Code Models Possess Knowledge of Correct API Names?cited by 3

Built on by

Not cited yet.