# CHASE: LLM Agents for Dissecting Malicious PyPI Packages

arXiv: https://arxiv.org/abs/2601.06838 · HTML: https://mlsanity.com/paper/2601.06838/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 19 of its 53 references that are in the map.
- First posted on arXiv 11 Jan 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 19 of 19)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models](https://mlsanity.com/paper/2305.04091/) (2023)
- [Backstabber's Knife Collection: A Review of Open Source Software Supply Chain Attacks](https://mlsanity.com/paper/2005.09535/) (2020)
- [Large Language Models for Code Analysis: Do LLMs Really Do Their Job?](https://mlsanity.com/paper/2310.12357/) (2023)
- [Small World with High Risks: A Study of Security Threats in the npm Ecosystem](https://mlsanity.com/paper/1902.09217/) (2019)
- [An Empirical Study of Malicious Code In PyPI Ecosystem](https://mlsanity.com/paper/2309.11021/) (2023)
- [On Benchmarking Code LLMs for Android Malware Analysis](https://mlsanity.com/paper/2504.00694/) (2025)
- [Killing Two Birds with One Stone: Malicious Package Detection in NPM and PyPI using a Single Model of Malicious Behavior Sequence](https://mlsanity.com/paper/2309.02637/) (2023)
- [Practical Automated Detection of Malicious npm Packages](https://mlsanity.com/paper/2202.13953/) (2022)
- [When Fuzzing Meets LLMs: Challenges and Opportunities](https://mlsanity.com/paper/2404.16297/) (2024)
- [A Benchmark Comparison of Python Malware Detection Approaches](https://mlsanity.com/paper/2209.13288/) (2022)
- [DONAPI: Malicious NPM Packages Detector using Behavior Sequence Knowledge Mapping](https://mlsanity.com/paper/2403.08334/) (2024)
- [Leveraging Large Language Models to Detect npm Malicious Packages](https://mlsanity.com/paper/2403.12196/) (2024)
- [MalGuard: Towards Real-Time, Accurate, and Actionable Detection of Malicious Packages in PyPI Ecosystem](https://mlsanity.com/paper/2506.14466/) (2025)
- [Towards Robust Detection of Open Source Software Supply Chain Poisoning Attacks in Industry Environments](https://mlsanity.com/paper/2409.09356/) (2024)
- [A Large-scale Fine-grained Analysis of Packages in Open-Source Software Ecosystems](https://mlsanity.com/paper/2404.11467/) (2024)

## Abstract (arXiv)

Modern software package registries like PyPI have become critical infrastructure for software development, but are increasingly exploited by threat actors distributing malicious packages with sophisticated multi-stage attack chains. While Large Language Models (LLMs) offer promising capabilities for automated code analysis, their application to security-critical malware detection faces fundamental challenges, including hallucination and context confusion, which can lead to missed detections or false alarms. We present CHASE (Collaborative Hierarchical Agents for Security Exploration), a high-reliability multi-agent architecture that addresses these limitations through a Plan-and-Execute coordination model, specialized Worker Agents focused on specific analysis aspects, and integration with deterministic security tools for critical operations. Our key insight is that reliability in LLM-based security analysis emerges not from improving individual model capabilities but from architecting systems that compensate for LLM weaknesses while leveraging their semantic understanding strengths. Evaluation on a dataset of 3,000 packages (500 malicious, 2,500 benign) demonstrates that CHASE achieves 98.4% recall with only 0.08% false positive rate, while maintaining a practical median analysis time of 4.5 minutes per package, making it suitable for operational deployment in automated package screening. Furthermore, we conducted a survey with cybersecurity professionals to evaluate the generated analysis reports, identifying their key strengths and areas for improvement. This work provides a blueprint for building reliable AI-powered security tools that can scale with the growing complexity of modern software supply chains. Our project page is available at https://t0d4.github.io/CHASE-AIware25/