# When LLMs Meet API Documentation: Can Retrieval Augmentation Aid Code Generation Just as It Helps Developers?

arXiv: https://arxiv.org/abs/2503.15231 · HTML: https://mlsanity.com/paper/2503.15231/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in RAG for code, one of 15 directions in Retrieval-augmented generation.
- Builds on 21 of its 49 references that are in the map.
- First posted on arXiv 19 Mar 2025.

## Placed in

- Retrieval-augmented generation / [RAG for code](https://mlsanity.com/rag/rag-for-code/)

## Builds on (most cited first, 21 of 21)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://mlsanity.com/paper/2306.04751/) (2023)
- [RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation](https://mlsanity.com/paper/2303.12570/) (2023)
- [Text and Code Embeddings by Contrastive Pre-Training](https://mlsanity.com/paper/2201.10005/) (2022)
- [The Power of Noise: Redefining Retrieval for RAG Systems](https://mlsanity.com/paper/2401.14887/) (2024)
- [Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs](https://mlsanity.com/paper/2312.05934/) (2023)
- [Dense X Retrieval: What Retrieval Granularity Should We Use?](https://mlsanity.com/paper/2312.06648/) (2023)
- [Large Language Model-Aware In-Context Learning for Code Generation](https://mlsanity.com/paper/2310.09748/) (2023)
- [Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applications](https://mlsanity.com/paper/2503.01245/) (2025)
- [EVOR: Evolving Retrieval for Code Generation](https://mlsanity.com/paper/2402.12317/) (2024)
- [On Mitigating Code LLM Hallucinations with API Documentation](https://mlsanity.com/paper/2407.09726/) (2024)
- [DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generation](https://mlsanity.com/paper/2408.13204/) (2024)
- [An Empirical Study of Retrieval-Augmented Code Generation: Challenges and Opportunities](https://mlsanity.com/paper/2501.13742/) (2025)
- [From Informal to Formal -- Incorporating and Evaluating LLMs on Natural Language Requirements to Verifiable Formal Proofs](https://mlsanity.com/paper/2501.16207/) (2025)
- [Retrieval-Augmented Test Generation: How Far Are We?](https://mlsanity.com/paper/2409.12682/) (2024)
- [Ivy: Templated Deep Learning for Inter-Framework Portability](https://mlsanity.com/paper/2102.02886/) (2021)

## Built on by (most cited first, 13 of 13)

- [Retrieval-Augmented Code Generation: A Survey with Focus on Repository-Level Approaches](https://mlsanity.com/paper/2510.04905/) (2025)
- [FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents](https://mlsanity.com/paper/2504.13128/) (2025)
- [Across Programming Language Silos: A Study on Cross-Lingual Retrieval-augmented Code Generation](https://mlsanity.com/paper/2506.03535/) (2025)
- [kRAIG: A Natural Language-Driven Agent for Automated DataOps Pipeline Generation](https://mlsanity.com/paper/2603.20311/) (2026)
- [Understanding Agent-Reactive Bugs at the Model-Harness Boundary: An Empirical Study of LLM Agent Issue Reports](https://mlsanity.com/paper/2607.15684/) (2026)
- [BIM Information Extraction Through LLM-based Adaptive Exploration](https://mlsanity.com/paper/2605.01698/) (2026)
- [LibRec: Benchmarking Retrieval-Augmented LLMs for Library Migration Recommendations](https://mlsanity.com/paper/2508.09791/) (2025)
- [RAG Strategies for Natural Language-Based SQL Query and REST API Call Generation](https://mlsanity.com/paper/2602.07086/) (2026)
- [BODHI: Precise OS Kernel Specification Inference](https://mlsanity.com/paper/2605.23931/) (2026)
- [Diagnosing Knowledge Gaps in LLM Tool Use: An Agentic Benchmark for Novel API Acquisition](https://mlsanity.com/paper/2606.03657/) (2026)
- [Detecting Architectural Drift in Safety-Critical Firmware through Runtime Trace Analysis](https://mlsanity.com/paper/2607.03135/) (2026)
- [Can Agents Design Libraries for Agents?](https://mlsanity.com/paper/2609.36730/) (2026)
- [Agent4Spec: An LLM-based Agentic System to Optimize API Specifications through Automated Validation and Market Analysis](https://mlsanity.com/paper/2610.13273/) (2026)

## Abstract (arXiv)

Retrieval-augmented generation (RAG) has increasingly shown its power in extending large language models' (LLMs') capability beyond their pre-trained knowledge. Existing works have shown that RAG can help with software development tasks such as code generation, code update, and test generation. Yet, the effectiveness of adapting LLMs to fast-evolving or less common API libraries using RAG remains unknown. To bridge this gap, we take an initial step to study this unexplored yet practical setting - when developers code with a less common library, they often refer to its API documentation; likewise, when LLMs are allowed to look up API documentation via RAG, to what extent can LLMs be advanced? To mimic such a setting, we select four less common open-source Python libraries with a total of 1017 eligible APIs. We study the factors that affect the effectiveness of using the documentation of less common API libraries as additional knowledge for retrieval and generation. Our intensive study yields interesting findings: (1) RAG helps improve LLMs' performance by 83%-220%. (2) Example code contributes the most to advance LLMs, instead of the descriptive texts and parameter lists in the API documentation. (3) LLMs could sometimes tolerate mild noises (typos in description or incorrect parameters) by referencing their pre-trained knowledge or document context. Finally, we suggest that developers pay more attention to the quality and diversity of the code examples in the API documentation. The study sheds light on future low-code software development workflows.