# A Large-Scale Exploit Instrumentation Study of AI/ML Supply Chain Attacks in Hugging Face Models

arXiv: https://arxiv.org/abs/2410.04490 · HTML: https://mlsanity.com/paper/2410.04490/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 74 references that are in the map.
- First posted on arXiv 6 Oct 2024.

## Builds on (most cited first, 23 of 23)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Local Model Poisoning Attacks to Byzantine-Robust Federated Learning](https://mlsanity.com/paper/1911.11815/) (2019)
- [Weight Poisoning Attacks on Pre-trained Models](https://mlsanity.com/paper/2004.06660/) (2020)
- [Membership Inference Attacks on Machine Learning: A Survey](https://mlsanity.com/paper/2103.07853/) (2021)
- [Trojaning Language Models for Fun and Profit](https://mlsanity.com/paper/2008.00312/) (2020)
- [Prediction Poisoning: Towards Defenses Against DNN Model Stealing Attacks](https://mlsanity.com/paper/1906.10908/) (2019)
- [Code Prediction by Feeding Trees to Transformers](https://mlsanity.com/paper/2003.13848/) (2020)
- [Taxonomy of Attacks on Open-Source Software Supply Chains](https://mlsanity.com/paper/2204.04008/) (2022)
- [CodeFill: Multi-token Code Completion by Jointly Learning from Structure and Naming Sequences](https://mlsanity.com/paper/2202.06689/) (2022)
- [An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry](https://mlsanity.com/paper/2303.02552/) (2023)
- [DeepPayload: Black-box Backdoor Attack on Deep Learning Models through Neural Payload Injection](https://mlsanity.com/paper/2101.06896/) (2021)
- [Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Models](https://mlsanity.com/paper/2403.04786/) (2024)
- [Fast and Memory-Efficient Neural Code Completion](https://mlsanity.com/paper/2004.13651/) (2020)
- [Deep Learning Meets Software Engineering: A Survey on Pre-Trained Models of Source Code](https://mlsanity.com/paper/2205.11739/) (2022)
- [Compiling ONNX Neural Network Models Using MLIR](https://mlsanity.com/paper/2008.08272/) (2020)
- [M2TS: Multi-Scale Multi-Modal Approach Based on Transformer for Source Code Summarization](https://mlsanity.com/paper/2203.09707/) (2022)
- [DONAPI: Malicious NPM Packages Detector using Behavior Sequence Knowledge Mapping](https://mlsanity.com/paper/2403.08334/) (2024)
- [ODDFUZZ: Discovering Java Deserialization Vulnerabilities via Structure-Aware Directed Greybox Fuzzing](https://mlsanity.com/paper/2304.04233/) (2023)
- [MalModel: Hiding Malicious Payload in Mobile Deep Learning Models with Black-box Backdoor Attack](https://mlsanity.com/paper/2401.02659/) (2024)
- [ONNXPruner: ONNX-Based General Model Pruning Adapter](https://mlsanity.com/paper/2404.08016/) (2024)
- [An In-depth Study of Java Deserialization Remote-Code Execution Exploits and Vulnerabilities](https://mlsanity.com/paper/2208.08173/) (2022)
- [Seneca: Taint-Based Call Graph Construction for Java Object Deserialization](https://mlsanity.com/paper/2311.00943/) (2023)
- [Improving Java Deserialization Gadget Chain Mining via Overriding-Guided Object Generation](https://mlsanity.com/paper/2303.07593/) (2023)

## Built on by (most cited first, 14 of 14)

- [Fox in the Henhouse: Supply-Chain Backdoor Attacks Against Reinforcement Learning](https://mlsanity.com/paper/2505.19532/) (2025)
- [PickleBall: Secure Deserialization of Pickle-based Machine Learning Models (Extended Report)](https://mlsanity.com/paper/2508.15987/) (2025)
- [On the (In)Security of Loading Machine Learning Models](https://mlsanity.com/paper/2509.06703/) (2025)
- [Inference-Time Backdoors via Chat Templates: From LLM Supply Chains to Agentic System Compromise](https://mlsanity.com/paper/2602.04653/) (2026)
- [Broken on Arrival: Silently Defective LLM Artifacts in Public Model Registries and How to Catch Them](https://mlsanity.com/paper/2609.05881/) (2026)
- [Securing the AI Supply Chain: What Can We Learn From Developer-Reported Security Issues and Solutions of AI Projects?](https://mlsanity.com/paper/2512.23385/) (2025)
- [Toward Reliable, Safe, and Secure LLMs for Scientific Applications](https://mlsanity.com/paper/2603.18235/) (2026)
- [Cyber-Capable AI Agents: Vulnerabilities, Evaluation Containment, and Defensive Response](https://mlsanity.com/paper/2607.25379/) (2026)
- [When Prompts Become Payloads: A Framework for Mitigating SQL Injection Attacks in Large Language Model-Driven Applications](https://mlsanity.com/paper/2605.10176/) (2026)
- [Lifecycle-Aware Dynamic Analysis for Secure ML Model Execution](https://mlsanity.com/paper/2606.19023/) (2026)
- [A Security-Oriented Lifecycle Model for Large Language Model Systems](https://mlsanity.com/paper/2608.03626/) (2026)
- [BadEngram: Backdoor Attack on Gated Memory Components in LLMs](https://mlsanity.com/paper/2609.13478/) (2026)
- [SINGED: Correct Outputs Do Not Certify Safe Execution in LLM Agents](https://mlsanity.com/paper/2609.35889/) (2026)
- [GrayShield: Bit-Level Sanitization for Transformer Model Supply-Chain Security](https://mlsanity.com/paper/2610.04319/) (2026)

## Abstract (arXiv)

The development of machine learning (ML) techniques has led to ample opportunities for developers to develop and deploy their own models. Hugging Face serves as an open source platform where developers can share and download other models in an effort to make ML development more collaborative. In order for models to be shared, they first need to be serialized. Certain Python serialization methods are considered unsafe, as they are vulnerable to object injection. This paper investigates the pervasiveness of these unsafe serialization methods across Hugging Face, and demonstrates through an exploitation approach, that models using unsafe serialization methods can be exploited and shared, creating an unsafe environment for ML developers. We investigate to what extent Hugging Face is able to flag repositories and files using unsafe serialization methods, and develop a technique to detect malicious models. Our results show that Hugging Face is home to a wide range of potentially vulnerable models.