# Many Hands Make Light Work: An LLM-based Multi-Agent System for Detecting Malicious PyPI Packages

arXiv: https://arxiv.org/abs/2601.12148 · HTML: https://mlsanity.com/paper/2601.12148/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 22 of its 43 references that are in the map.
- First posted on arXiv 17 Jan 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 22 of 22)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors](https://mlsanity.com/paper/2308.10848/) (2023)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks](https://mlsanity.com/paper/1909.03496/) (2019)
- [A Survey of Machine Learning for Big Code and Naturalness](https://mlsanity.com/paper/1709.06182/) (2017)
- [AutoAgents: A Framework for Automatic Agent Generation](https://mlsanity.com/paper/2309.17288/) (2023)
- [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024)
- [LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead](https://mlsanity.com/paper/2404.04834/) (2024)
- [Malware Detection by Eating a Whole EXE](https://mlsanity.com/paper/1710.09435/) (2017)
- [Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate](https://mlsanity.com/paper/2305.13160/) (2023)
- [Natural Attack for Pre-trained Models of Code](https://mlsanity.com/paper/2201.08698/) (2022)
- [Backstabber's Knife Collection: A Review of Open Source Software Supply Chain Attacks](https://mlsanity.com/paper/2005.09535/) (2020)
- [A Survey on LLM-based Multi-Agent System: Recent Advances and New Frontiers in Application](https://mlsanity.com/paper/2412.17481/) (2024)
- [Investigating ChatGPT's Potential to Assist in Requirements Elicitation Processes](https://mlsanity.com/paper/2307.07381/) (2023)
- [An Empirical Study of Malicious Code In PyPI Ecosystem](https://mlsanity.com/paper/2309.11021/) (2023)
- [DocAgent: A Multi-Agent System for Automated Code Documentation Generation](https://mlsanity.com/paper/2504.08725/) (2025)
- [Automated test generation to evaluate tool-augmented LLMs as conversational AI agents](https://mlsanity.com/paper/2409.15934/) (2024)
- [Killing Two Birds with One Stone: Malicious Package Detection in NPM and PyPI using a Single Model of Malicious Behavior Sequence](https://mlsanity.com/paper/2309.02637/) (2023)
- [A Large-Scale Security-Oriented Static Analysis of Python Packages in PyPI](https://mlsanity.com/paper/2107.12699/) (2021)
- [Teamwork makes the dream work: LLMs-Based Agents for GitHub README.MD Summarization](https://mlsanity.com/paper/2503.10876/) (2025)
- [A Machine Learning-Based Approach For Detecting Malicious PyPI Packages](https://mlsanity.com/paper/2412.05259/) (2024)
- [Detecting Malicious Source Code in PyPI Packages with LLMs: Does RAG Come in Handy?](https://mlsanity.com/paper/2504.13769/) (2025)

## Built on by (most cited first, 1 of 1)

- [Developing LLM-based Multi-Agent Systems in Software Engineering: A Mixed-Method Experience Report](https://mlsanity.com/paper/2608.11965/) (2026)

## Abstract (arXiv)

Malicious code in open-source repositories such as PyPI poses a growing threat to software supply chains. Traditional rule-based tools often overlook the semantic patterns in source code that are crucial for identifying adversarial components. Large language models (LLMs) show promise for software analysis, yet their use in interpretable and modular security pipelines remains limited. This paper presents LAMPS, a multi-agent system that employs collaborative LLMs to detect malicious PyPI packages. The system consists of four role-specific agents for package retrieval, file extraction, classification, and verdict aggregation, coordinated through the CrewAI framework. A prototype combines a fine-tuned CodeBERT model for classification with LLaMA-3 agents for contextual reasoning. LAMPS has been evaluated on two complementary datasets: D1, a balanced collection of 6,000 setup.py files, and D2, a realistic multi-file dataset with 1,296 files and natural class imbalance. On D1, LAMPS achieves 97.7% accuracy, surpassing MPHunter--one of the state-of-the-art approaches. On D2, it reaches 99.5% accuracy and 99.5% balanced accuracy, outperforming RAG-based approaches and fine-tuned single-agent baselines. McNemar's test confirmed these improvements as highly significant. The results demonstrate the feasibility of distributed LLM reasoning for malicious code detection and highlight the benefits of modular multi-agent designs in software supply chain security.