# On Security Weaknesses and Vulnerabilities in Deep Learning Systems

arXiv: https://arxiv.org/abs/2406.08688 · HTML: https://mlsanity.com/paper/2406.08688/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 80 references that are in the map.
- First posted on arXiv 12 Jun 2024.

## Builds on (most cited first, 15 of 15)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Caffe: Convolutional Architecture for Fast Feature Embedding](https://mlsanity.com/paper/1408.5093/) (2014)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [A Comprehensive Study on Deep Learning Bug Characteristics](https://mlsanity.com/paper/1906.01388/) (2019)
- [Engineering AI Systems: A Research Agenda](https://mlsanity.com/paper/2001.07522/) (2020)
- [A Software Engineering Perspective on Engineering Machine Learning Systems: State of the Art and Challenges](https://mlsanity.com/paper/2012.07919/) (2020)
- [An Empirical Study on Deployment Faults of Deep Learning Based Mobile Applications](https://mlsanity.com/paper/2101.04930/) (2021)
- [Design Smells in Deep Learning Programs: An Empirical Study](https://mlsanity.com/paper/2107.02279/) (2021)
- [Security and Machine Learning in the Real World](https://mlsanity.com/paper/2007.07205/) (2020)
- [Security for Machine Learning-based Software Systems: a survey of threats, practices and challenges](https://mlsanity.com/paper/2201.04736/) (2022)
- [On managing vulnerabilities in AI/ML systems](https://mlsanity.com/paper/2101.10865/) (2021)
- [IvySyn: Automated Vulnerability Discovery in Deep Learning Frameworks](https://mlsanity.com/paper/2209.14921/) (2022)
- [Characterizing and Understanding Software Security Vulnerabilities in Machine Learning Libraries](https://mlsanity.com/paper/2203.06502/) (2022)
- [An Empirical Study on Tensor Shape Faults in Deep Learning Systems](https://mlsanity.com/paper/2106.02887/) (2021)

## Abstract (arXiv)

The security guarantee of AI-enabled software systems (particularly using deep learning techniques as a functional core) is pivotal against the adversarial attacks exploiting software vulnerabilities. However, little attention has been paid to a systematic investigation of vulnerabilities in such systems. A common situation learned from the open source software community is that deep learning engineers frequently integrate off-the-shelf or open-source learning frameworks into their ecosystems. In this work, we specifically look into deep learning (DL) framework and perform the first systematic study of vulnerabilities in DL systems through a comprehensive analysis of identified vulnerabilities from Common Vulnerabilities and Exposures (CVE) and open-source DL tools, including TensorFlow, Caffe, OpenCV, Keras, and PyTorch. We propose a two-stream data analysis framework to explore vulnerability patterns from various databases. We investigate the unique DL frameworks and libraries development ecosystems that appear to be decentralized and fragmented. By revisiting the Common Weakness Enumeration (CWE) List, which provides the traditional software vulnerability related practices, we observed that it is more challenging to detect and fix the vulnerabilities throughout the DL systems lifecycle. Moreover, we conducted a large-scale empirical study of 3,049 DL vulnerabilities to better understand the patterns of vulnerability and the challenges in fixing them. We have released the full replication package at https://github.com/codelzz/Vulnerabilities4DLSystem. We anticipate that our study can advance the development of secure DL systems.