# Revisiting, Benchmarking and Exploring API Recommendation: How Far Are We?

arXiv: https://arxiv.org/abs/2112.12653 · HTML: https://mlsanity.com/paper/2112.12653/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2021.

## Built on by (most cited first, 21 of 21)

- [Survey on Evaluation of LLM-based Agents](https://mlsanity.com/paper/2503.16416/) (2025)
- [AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents](https://mlsanity.com/paper/2410.24024/) (2024)
- [ToolNet: Connecting Large Language Models with Massive Tools via Tool Graph](https://mlsanity.com/paper/2403.00839/) (2024)
- [Dataflow-Guided Retrieval Augmentation for Repository-Level Code Completion](https://mlsanity.com/paper/2405.19782/) (2024)
- [MCP-Zero: Active Tool Discovery for Autonomous LLM Agents](https://mlsanity.com/paper/2506.01056/) (2025)
- [Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey](https://mlsanity.com/paper/2506.11102/) (2025)
- [ToolVQA: A Dataset for Multi-step Reasoning VQA with External Tools](https://mlsanity.com/paper/2508.03284/) (2025)
- [Compositional API Recommendation for Library-Oriented Code Generation](https://mlsanity.com/paper/2402.19431/) (2024)
- [Tools in the Loop: Quantifying Uncertainty of LLM Question Answering Systems That Use Tools](https://mlsanity.com/paper/2505.16113/) (2025)
- [NewTerm: Benchmarking Real-Time New Terms for Large Language Models with Annual Updates](https://mlsanity.com/paper/2410.20814/) (2024)
- [CallNavi, A Challenge and Empirical Study on LLM Function Calling and Routing](https://mlsanity.com/paper/2501.05255/) (2025)
- [Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Aware](https://mlsanity.com/paper/2505.05057/) (2025)
- [Translating Regulatory Clauses into Executable Codes for Building Design Checking via Large Language Model Driven Function Matching and Composing](https://mlsanity.com/paper/2308.08728/) (2023)
- [Knowledge-aware Alert Aggregation in Large-scale Cloud Systems: a Hybrid Approach](https://mlsanity.com/paper/2403.06485/) (2024)
- [Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Quality](https://mlsanity.com/paper/2503.05860/) (2025)
- [Live API-Bench: 2500+ Live APIs for Testing Multi-Step Tool Calling](https://mlsanity.com/paper/2506.11266/) (2025)
- [PORTS: Preference-Optimized Retrievers for Tool Selection with Large Language Models](https://mlsanity.com/paper/2607.05441/) (2026)
- [Tool Retrieval Bridge: Aligning Vague Instructions with Retriever Preferences via Bridge Model](https://mlsanity.com/paper/2604.07816/) (2026)
- [MetaToolAgent: Towards Generalizable Tool Usage in LLMs through Meta-Learning](https://mlsanity.com/paper/2601.12680/) (2026)
- [Explore-Construct-Filter: An Automated Framework for Rich and Reliable API Knowledge Graph Construction](https://mlsanity.com/paper/2502.13412/) (2025)
- [Beyond APIs: Probing the Limits of MLLMs in Physical Tool Use](https://mlsanity.com/paper/2606.10803/) (2026)

## Abstract (arXiv)

Application Programming Interfaces (APIs), which encapsulate the implementation of specific functions as interfaces, greatly improve the efficiency of modern software development. As numbers of APIs spring up nowadays, developers can hardly be familiar with all the APIs, and usually need to search for appropriate APIs for usage. So lots of efforts have been devoted to improving the API recommendation task. However, it has been increasingly difficult to gauge the performance of new models due to the lack of a uniform definition of the task and a standardized benchmark. For example, some studies regard the task as a code completion problem; while others recommend relative APIs given natural language queries. To reduce the challenges and better facilitate future research, in this paper, we revisit the API recommendation task and aim at benchmarking the approaches. Specifically, the paper groups the approaches into two categories according to the task definition, i.e., query-based API recommendation and code-based API recommendation. We study 11 recently-proposed approaches along with 4 widely-used IDEs. One benchmark named as APIBench is then built for the two respective categories of approaches. Based on APIBench, we distill some actionable insights and challenges for API recommendation. We also achieve some implications and directions for improving the performance of recommending APIs, including data source selection, appropriate query reformulation, low resource setting, and cross-domain adaptation.