# Library Hallucinations in LLM-Generated Code: A Risk Analysis Grounded in Developer Queries

arXiv: https://arxiv.org/abs/2509.22202 · HTML: https://mlsanity.com/paper/2509.22202/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 83 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Builds on (most cited first, 30 of 43)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [Retrieval-Augmented Generation for AI-Generated Content: A Survey](https://mlsanity.com/paper/2402.19473/) (2024)
- [Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback](https://mlsanity.com/paper/2302.12813/) (2023)
- [Execution-Based Evaluation for Open-Domain Code Generation](https://mlsanity.com/paper/2212.10481/) (2022)
- [CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generation](https://mlsanity.com/paper/2206.06888/) (2022)
- [Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements](https://mlsanity.com/paper/2401.06766/) (2024)
- [NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation](https://mlsanity.com/paper/2112.02721/) (2021)
- [Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation](https://mlsanity.com/paper/2308.10335/) (2023)
- [Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERT](https://mlsanity.com/paper/2003.04985/) (2020)
- [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs](https://mlsanity.com/paper/2406.10279/) (2024)
- [A Survey on Evaluating Large Language Models in Code Generation Tasks](https://mlsanity.com/paper/2408.16498/) (2024)
- [Confabulation: The Surprising Value of Large Language Model Hallucinations](https://mlsanity.com/paper/2406.04175/) (2024)
- [A Survey on Large Language Model Hallucination via a Creativity Perspective](https://mlsanity.com/paper/2402.06647/) (2024)
- [CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification](https://mlsanity.com/paper/2405.00253/) (2024)
- [LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-based Code Completion](https://mlsanity.com/paper/2406.09834/) (2024)
- [Reasoning Robustness of LLMs to Adversarial Typographical Errors](https://mlsanity.com/paper/2411.05345/) (2024)
- [Coding Agents with Multimodal Browsing are Generalist Problem Solvers](https://mlsanity.com/paper/2506.03011/) (2025)
- [GitHub Typo Corpus: A Large-Scale Multilingual Dataset of Misspellings and Grammatical Errors](https://mlsanity.com/paper/1911.12893/) (2019)
- [Are Large Language Models a Threat to Digital Public Goods? Evidence from Activity on Stack Overflow](https://mlsanity.com/paper/2307.07367/) (2023)
- [On Mitigating Code LLM Hallucinations with API Documentation](https://mlsanity.com/paper/2407.09726/) (2024)
- [Self-Reflection Makes Large Language Models Safer, Less Biased, and Ideologically Neutral](https://mlsanity.com/paper/2406.10400/) (2024)
- [Auditing Prompt Caching in Language Model APIs](https://mlsanity.com/paper/2502.07776/) (2025)
- [A Study of LLMs' Preferences for Libraries and Programming Languages](https://mlsanity.com/paper/2503.17181/) (2025)
- [De-Hallucinator: Mitigating LLM Hallucinations in Code Generation Tasks via Iterative Grounding](https://mlsanity.com/paper/2401.01701/) (2024)
- [Importing Phantoms: Measuring LLM Package Hallucination Vulnerabilities](https://mlsanity.com/paper/2501.19012/) (2025)
- [ExploraCoder: Advancing code generation for multiple unseen APIs via planning and chained exploration](https://mlsanity.com/paper/2412.05366/) (2024)

## Built on by (most cited first, 1 of 1)

- [LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs](https://mlsanity.com/paper/2608.06041/) (2026)

## Abstract (arXiv)

Large language models (LLMs) now play a central role in code generation, yet they continue to hallucinate, frequently inventing non-existent libraries. Such library hallucinations are not just benign errors: they can mislead developers, break builds, and expose systems to supply chain threats such as slopsquatting. Despite growing awareness of these risks, there is limited understanding of how library hallucinations manifest under realistic usage conditions. To fill this gap, we present the first systematic study of how user-level prompt variations influence library hallucinations in LLM-generated code. Across seven diverse LLMs, we analyse library name hallucinations (invalid imports) and library member hallucinations (invalid calls from valid libraries), examining the effects of realistic developer language and controlled user mistakes, including misspellings and fabricated libraries or members. Our findings expose systemic vulnerabilities: one-character misspellings trigger hallucinations in up to 26% of tasks; fabricated library names are accepted in up to 99%; and time-based prompts induce hallucinations in up to 85%. Grounded in the highest-risk prompts identified in our study, we introduce LibHalluBench, a benchmark that enables a systematic and reproducible evaluation of these library hallucinations. Our findings underscore the fragility of LLMs to natural prompt variation and highlight the urgent need for safeguards against library-related hallucinations and their downstream risks.