# An Empirical Study of Vulnerabilities in Python Packages and Their Detection

arXiv: https://arxiv.org/abs/2509.04260 · HTML: https://mlsanity.com/paper/2509.04260/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 73 references that are in the map.
- First posted on arXiv 4 Sep 2025.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Large Language Models for Code: Security Hardening and Adversarial Testing](https://mlsanity.com/paper/2302.05319/) (2023)
- [Deep Learning based Vulnerability Detection: Are We There Yet?](https://mlsanity.com/paper/2009.07235/) (2020)
- [Vulnerability Detection with Code Language Models: How Far Are We?](https://mlsanity.com/paper/2403.18624/) (2024)
- [Automated Vulnerability Detection in Source Code Using Deep Representation Learning](https://mlsanity.com/paper/1807.04320/) (2018)
- [DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection](https://mlsanity.com/paper/2304.00409/) (2023)
- [CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software](https://mlsanity.com/paper/2107.08760/) (2021)
- [Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study](https://mlsanity.com/paper/2310.02059/) (2023)
- [ReposVul: A Repository-Level High-Quality Vulnerability Dataset](https://mlsanity.com/paper/2401.13169/) (2024)
- [Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection](https://mlsanity.com/paper/2408.12986/) (2024)
- [PyCG: Practical Call Graph Generation in Python](https://mlsanity.com/paper/2103.00587/) (2021)
- [VUDENC: Vulnerability Detection with Deep Learning on a Natural Codebase for Python](https://mlsanity.com/paper/2201.08441/) (2022)
- [Toward Improved Deep Learning-based Vulnerability Detection](https://mlsanity.com/paper/2403.03024/) (2024)
- [How Well Do Large Language Models Serve as End-to-End Secure Code Agents for Python?](https://mlsanity.com/paper/2408.10495/) (2024)
- [An Empirical Study on Android-related Vulnerabilities](https://mlsanity.com/paper/1704.03356/) (2017)

## Built on by (most cited first, 2 of 2)

- [MultiVer: Zero-Shot Multi-Agent Vulnerability Detection](https://mlsanity.com/paper/2602.17875/) (2026)
- [From Evaluation to Optimisation: Hierarchy-Aware Training Signals for CWE Prediction in Python](https://mlsanity.com/paper/2607.21069/) (2026)

## Abstract (arXiv)

In the rapidly evolving software development landscape, Python stands out for its simplicity, versatility, and extensive ecosystem. Python packages, as units of organization, reusability, and distribution, have become a pressing concern, highlighted by the considerable number of vulnerability reports. As a scripting language, Python often cooperates with other languages for performance or interoperability. This adds complexity to the vulnerabilities inherent to Python packages, and the effectiveness of current vulnerability detection tools remains underexplored. This paper addresses these gaps by introducing PyVul, the first comprehensive benchmark suite of Python-package vulnerabilities. PyVul includes 1,157 publicly reported, developer-verified vulnerabilities, each linked to its affected packages. To accommodate diverse detection techniques, it provides annotations at both commit and function levels. An LLM-assisted data cleansing method is incorporated to improve label accuracy, achieving 100% commit-level and 94% function-level accuracy, establishing PyVul as the most precise large-scale Python vulnerability benchmark. We further carry out a distribution analysis of PyVul, which demonstrates that vulnerabilities in Python packages involve multiple programming languages and exhibit a wide variety of types. Moreover, our analysis reveals that multi-lingual Python packages are potentially more susceptible to vulnerabilities. Evaluation of state-of-the-art detectors using this benchmark reveals a significant discrepancy between the capabilities of existing tools and the demands of effectively identifying real-world security issues in Python packages. Additionally, we conduct an empirical review of the top-ranked CWEs observed in Python packages, to diagnose the fine-grained limitations of current detection tools and highlight the necessity for future advancements in the field.