# Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions

arXiv: https://arxiv.org/abs/2605.06279 · HTML: https://mlsanity.com/paper/2605.06279/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 72 references that are in the map.
- First posted on arXiv 7 May 2026.

## Builds on (most cited first, 30 of 36)

- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation](https://mlsanity.com/paper/2409.20550/) (2024)
- [Instruction Tuning for Secure Code Generation](https://mlsanity.com/paper/2402.09497/) (2024)
- [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs](https://mlsanity.com/paper/2406.10279/) (2024)
- [A Survey on Large Language Models for Software Engineering](https://mlsanity.com/paper/2312.15223/) (2023)
- [Taxonomy of Attacks on Open-Source Software Supply Chains](https://mlsanity.com/paper/2204.04008/) (2022)
- [FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation](https://mlsanity.com/paper/2503.06680/) (2025)
- [Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study](https://mlsanity.com/paper/2310.02059/) (2023)
- [Constrained Decoding for Secure Code Generation](https://mlsanity.com/paper/2405.00218/) (2024)
- [LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-based Code Completion](https://mlsanity.com/paper/2406.09834/) (2024)
- [Do Developers Update Their Library Dependencies? An Empirical Study on the Impact of Security Advisories on Library Migration](https://mlsanity.com/paper/1709.04621/) (2017)
- [Large Language Model Supply Chain: A Research Agenda](https://mlsanity.com/paper/2404.12736/) (2024)
- [On Mitigating Code LLM Hallucinations with API Documentation](https://mlsanity.com/paper/2407.09726/) (2024)
- [CodeUpdateArena: Benchmarking Knowledge Editing on API Updates](https://mlsanity.com/paper/2407.06249/) (2024)
- [VersiCode: Towards Version-controllable Code Generation](https://mlsanity.com/paper/2406.07411/) (2024)
- [Give LLMs a Security Course: Securing Retrieval-Augmented Code Generation via Knowledge Injection](https://mlsanity.com/paper/2504.16429/) (2025)
- [Automating Dependency Updates in Practice: An Exploratory Study on GitHub Dependabot](https://mlsanity.com/paper/2206.07230/) (2022)
- [LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation](https://mlsanity.com/paper/2412.04478/) (2024)
- [Rethinking the Evaluation of Secure Code Generation](https://mlsanity.com/paper/2503.15554/) (2025)
- [Teaching an Old LLM Secure Coding: Localized Preference Optimization on Distilled Preferences](https://mlsanity.com/paper/2506.00419/) (2025)
- [Do LLMs Consider Security? An Empirical Study on Responses to Programming Questions](https://mlsanity.com/paper/2502.14202/) (2025)
- [Demystifying the Vulnerability Propagation and Its Evolution via Dependency Trees in the NPM Ecosystem](https://mlsanity.com/paper/2201.03981/) (2022)
- [GitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Models](https://mlsanity.com/paper/2411.05830/) (2024)
- [Empirical Analysis of Vulnerabilities Life Cycle in Golang Ecosystem](https://mlsanity.com/paper/2401.00515/) (2023)
- [Wolves in the Repository: A Software Engineering Analysis of the XZ Utils Supply Chain Attack](https://mlsanity.com/paper/2504.17473/) (2025)
- [A Comparative Study of Vulnerability Reporting by Software Composition Analysis Tools](https://mlsanity.com/paper/2108.12078/) (2021)
- [Pinning Is Futile: You Need More Than Local Dependency Versioning to Defend against Supply Chain Attacks](https://mlsanity.com/paper/2502.06662/) (2025)
- [A Survey of Bugs in AI-Generated Code](https://mlsanity.com/paper/2512.05239/) (2025)
- [AI Code in the Wild: Measuring Security Risks and Ecosystem Shifts of AI-Generated Code in Modern Software](https://mlsanity.com/paper/2512.18567/) (2025)

## Built on by (most cited first, 1 of 1)

- [Do AI Coding Assistants Check Before They Install? A Pre-Registered Demand-Side Audit of Trust Signals in the Research Software Supply Chain](https://mlsanity.com/paper/2609.07754/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are now largely involved in software development workflows, and the code they generate routinely includes third-party library (TPL) imports annotated with specific version identifiers. These version choices can carry security and compatibility risks, yet they have not been systematically studied. We present the first large-scale measurement study of version-level risk in LLM-generated Python code, evaluating 10 LLMs on PinTrace, a curated benchmark of 1,000 Stack Overflow programming tasks. LLMs tend to specify version identifiers when directly prompted at 26.83%-95.18%, while down to 6.45%-59.19% in creating a manifest file directly. Among the specified versions, 36.70%-55.70% of tasks contain at least one known CVE, and 62.75%-74.51% of them carry Critical or High severity ratings. In 72.27%-91.37% of cases, the associated CVEs were publicly disclosed before the model's knowledge cutoff. The statistics show all models converge on the same small set of risky release versions, indicating a systemic bias rather than isolated model error. Static compatibility rates range from 19.70% to 63.20%, with installation failure as the dominant cause. The dynamic test cases confirm the pattern by 6.49%-48.62% pass rates. Further experiments confirm that these failures are attributable to version selection rather than code quality, and that externally anchored version constraints substantially reduce both vulnerability exposure and compatibility failures. Our findings reveal LLM version selection as a first-class, previously overlooked risk surface in LLM-based development. We disclosed these findings to the community of the evaluated models, and several confirmed the issue. All the code and dataset have been released for open science at https://github.com/dw763j/PinTrace.