# Importing Phantoms: Measuring LLM Package Hallucination Vulnerabilities

arXiv: https://arxiv.org/abs/2501.19012 · HTML: https://mlsanity.com/paper/2501.19012/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jan 2025.

## Built on by (most cited first, 15 of 15)

- [HalluHard: A Hard Multi-Turn Hallucination Benchmark](https://mlsanity.com/paper/2602.01031/) (2026)
- [Trust Me, Import This: Dependency Steering Attacks via Malicious Agent Skills](https://mlsanity.com/paper/2605.09594/) (2026)
- [Medicine on the Edge: Comparative Performance Analysis of On-Device LLMs for Clinical Reasoning](https://mlsanity.com/paper/2502.08954/) (2025)
- [Review of Hallucination Understanding in Large Language and Vision Models](https://mlsanity.com/paper/2510.00034/) (2025)
- [Library Hallucinations in LLM-Generated Code: A Risk Analysis Grounded in Developer Queries](https://mlsanity.com/paper/2509.22202/) (2025)
- [HFuzzer: Testing Large Language Models for Package Hallucinations via Phrase-based Fuzzing](https://mlsanity.com/paper/2509.23835/) (2025)
- [Harmless Yet Harmful: Neutral Prompting Attacks for Stealthy Hallucination Steering in Agent Skills](https://mlsanity.com/paper/2605.29354/) (2026)
- [An Empirical Analysis of Static Analysis Methods for Detection and Mitigation of Code Library Hallucinations](https://mlsanity.com/paper/2604.07755/) (2026)
- [Training a General Purpose Automated Red Teaming Model](https://mlsanity.com/paper/2604.23067/) (2026)
- [LLM Ghostbusters: Surgical Package Hallucination Suppression via Adaptive Unlearning](https://mlsanity.com/paper/2605.01047/) (2026)
- [The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort](https://mlsanity.com/paper/2605.17062/) (2026)
- [Empirical Analysis and Detection of Hallucinations in LLM-Generated Bug Report Summaries](https://mlsanity.com/paper/2605.24137/) (2026)
- [Cross Paraphrastic Invariance Learning for Hallucination Detection](https://mlsanity.com/paper/2606.08157/) (2026)
- [Beware of Agentic Botnets: Scalable Untargeted Promptware Attacks via Universal and Transferable Adversarial HalluSquatting](https://mlsanity.com/paper/2607.07433/) (2026)
- [The Innocent Courier: Covert Exfiltration Through Legitimate LLM Web Fetching](https://mlsanity.com/paper/2610.01768/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have become an essential tool in the programmer's toolkit, but their tendency to hallucinate code can be used by malicious actors to introduce vulnerabilities to broad swathes of the software supply chain. In this work, we analyze package hallucination behaviour in LLMs across popular programming languages examining both existing package references and fictional dependencies. By analyzing this package hallucination behaviour we find potential attacks and suggest defensive strategies to defend against these attacks. We discover that package hallucination rate is predicated not only on model choice, but also programming language, model size, and specificity of the coding task request. The Pareto optimality boundary between code generation performance and package hallucination is sparsely populated, suggesting that coding models are not being optimized for secure code. Additionally, we find an inverse correlation between package hallucination rate and the HumanEval coding benchmark, offering a heuristic for evaluating the propensity of a model to hallucinate packages. Our metrics, findings and analyses provide a base for future models, securing AI-assisted software development workflows against package supply chain attacks.