Library Hallucinations in LLM-Generated Code: A Risk Analysis Grounded in Developer Queries
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) now play a central role in code generation, yet they continue to hallucinate, frequently inventing non-existent libraries. Such library hallucinations are not just benign errors: they can mislead developers, break builds, and expose systems to supply chain threats such as slopsquatting. Despite growing awareness of these risks, there is limited understanding of how library hallucinations manifest under realistic usage conditions. To fill this gap, we present the first systematic study of how user-level prompt variations influence library hallucinations in LLM-generated code. Across seven diverse LLMs, we analyse library name hallucinations (invalid imports) and library member hallucinations (invalid calls from valid libraries), examining the effects of realistic developer language and controlled user mistakes, including misspellings and fabricated libraries or members. Our findings expose systemic vulnerabilities: one-character misspellings trigger hallucinations in up to 26% of tasks; fabricated library names are accepted in up to 99%; and time-based prompts induce hallucinations in up to 85%. Grounded in the highest-risk prompts identified in our study, we introduce LibHalluBench, a benchmark that enables a systematic and reproducible evaluation of these library hallucinations. Our findings underscore the fragility of LLMs to natural prompt variation and highlight the urgent need for safeguards against library-related hallucinations and their downstream risks.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 43 of its 83 references that are in the map. First posted on arXiv 26 Sep 2025.
Builds on
43 of 83 references.
References · 43
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
- 2023Towards Understanding Sycophancy in Language Modelscited by 988
- 2024A Survey on Large Language Models for Code Generationcited by 646
- 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
- 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
- 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
- 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 89
- 2022CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generationcited by 87
- 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 82
- 2021NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentationcited by 72
- 2023Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generationcited by 72
31 more
- 2020Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERTcited by 69
- 2024We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMscited by 61
- 2024A Survey on Evaluating Large Language Models in Code Generation Taskscited by 43
- 2024Confabulation: The Surprising Value of Large Language Model Hallucinationscited by 42
- 2024A Survey on Large Language Model Hallucination via a Creativity Perspectivecited by 38
- 2024CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verificationcited by 32
- 2024Reasoning Robustness of LLMs to Adversarial Typographical Errorscited by 31
- 2024LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-based Code Completioncited by 29
- 2025Coding Agents with Multimodal Browsing are Generalist Problem Solverscited by 24
- 2019GitHub Typo Corpus: A Large-Scale Multilingual Dataset of Misspellings and Grammatical Errorscited by 22
- 2023Are Large Language Models a Threat to Digital Public Goods? Evidence from Activity on Stack Overflowcited by 22
- 2024On Mitigating Code LLM Hallucinations with API Documentationcited by 21
- 2024Self-Reflection Makes Large Language Models Safer, Less Biased, and Ideologically Neutralcited by 20
- 2025A Study of LLMs' Preferences for Libraries and Programming Languagescited by 20
- 2025Auditing Prompt Caching in Language Model APIscited by 19
- 2025Importing Phantoms: Measuring LLM Package Hallucination Vulnerabilitiescited by 15
- 2024De-Hallucinator: Mitigating LLM Hallucinations in Code Generation Tasks via Iterative Groundingcited by 14
- 2024ExploraCoder: Advancing code generation for multiple unseen APIs via planning and chained explorationcited by 11
- 2020Selecting third-party libraries: The practitioners' perspectivecited by 9
- 2025Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challengescited by 9
- 2025Studying How Configurations Impact Code Generation in LLMs: the Case of ChatGPTcited by 8
- 2025Which Prompting Technique Should I Use? An Empirical Investigation of Prompting Techniques for Software Engineering Taskscited by 8
- 2024Is ChatGPT a Good Software Librarian? An Exploratory Study on the Use of ChatGPT for Software Library Recommendationscited by 7
- 2024A Systematic Evaluation of Large Code Models in API Suggestion: When, Which, and Howcited by 7
- 2025Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Awarecited by 6
- 2020SpellBound: Defending Against Package Typosquattingcited by 4
- 2024CodeInsight: A Curated Dataset of Practical Coding Solutions from Stack Overflowcited by 4
- 2025Misspellings in Natural Language Processing: A surveycited by 3
- 2024The role of library versions in Developer-ChatGPT conversationscited by 2
- 2024"How do people decide?": A Model for Software Library Selectioncited by 2
- 2024The Dynamics of Innovation in Open Source Software Ecosystemscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.