# Shallow or Deep? An Empirical Study on Detecting Vulnerabilities using Deep Learning

arXiv: https://arxiv.org/abs/2103.11940 · HTML: https://mlsanity.com/paper/2103.11940/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Mar 2021.

## Built on by (most cited first, 7 of 7)

- [An Empirical Study of Deep Learning Models for Vulnerability Detection](https://mlsanity.com/paper/2212.08109/) (2022)
- [A Survey on Data-driven Software Vulnerability Assessment and Prioritization](https://mlsanity.com/paper/2107.08364/) (2021)
- [A Survey of Source Code Representations for Machine Learning-Based Cybersecurity Tasks](https://mlsanity.com/paper/2403.10646/) (2024)
- [What are the Machine Learning best practices reported by practitioners on Stack Exchange?](https://mlsanity.com/paper/2301.10516/) (2023)
- [Data-Driven AI Model Signal-Awareness Enhancement and Introspection](https://mlsanity.com/paper/2111.05827/) (2021)
- [An Empirical Study of the Imbalance Issue in Software Vulnerability Detection](https://mlsanity.com/paper/2602.12038/) (2026)
- [From Lab to Reality: A Practical Evaluation of Deep Learning Models and LLMs for Vulnerability Detection](https://mlsanity.com/paper/2512.10485/) (2025)

## Abstract (arXiv)

Deep learning (DL) techniques are on the rise in the software engineering research community. More and more approaches have been developed on top of DL models, also due to the unprecedented amount of software-related data that can be used to train these models. One of the recent applications of DL in the software engineering domain concerns the automatic detection of software vulnerabilities. While several DL models have been developed to approach this problem, there is still limited empirical evidence concerning their actual effectiveness especially when compared with shallow machine learning techniques. In this paper, we partially fill this gap by presenting a large-scale empirical study using three vulnerability datasets and five different source code representations (i.e., the format in which the code is provided to the classifiers to assess whether it is vulnerable or not) to compare the effectiveness of two widely used DL-based models and of one shallow machine learning model in (i) classifying code functions as vulnerable or non-vulnerable (i.e., binary classification), and (ii) classifying code functions based on the specific type of vulnerability they contain (or "clean", if no vulnerability is there). As a baseline we include in our study the AutoML utility provided by the Google Cloud Platform. Our results show that the experimented models are still far from ensuring reliable vulnerability detection, and that a shallow learning classifier represents a competitive baseline for the newest DL-based models.