# Using ChatGPT as a Static Application Security Testing Tool

arXiv: https://arxiv.org/abs/2308.14434 · HTML: https://mlsanity.com/paper/2308.14434/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 32 references that are in the map.
- First posted on arXiv 28 Aug 2023.

## Builds on (most cited first, 11 of 11)

- [Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks](https://mlsanity.com/paper/1909.03496/) (2019)
- [Deep Learning based Vulnerability Detection: Are We There Yet?](https://mlsanity.com/paper/2009.07235/) (2020)
- [Automated Vulnerability Detection in Source Code Using Deep Representation Learning](https://mlsanity.com/paper/1807.04320/) (2018)
- [DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection](https://mlsanity.com/paper/2304.00409/) (2023)
- [LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluations](https://mlsanity.com/paper/2303.09384/) (2023)
- [VulBERTa: Simplified Source Code Pre-Training for Vulnerability Detection](https://mlsanity.com/paper/2205.12424/) (2022)
- [An Empirical Study of Deep Learning Models for Vulnerability Detection](https://mlsanity.com/paper/2212.08109/) (2022)
- [Evaluation of ChatGPT Model for Vulnerability Detection](https://mlsanity.com/paper/2304.07232/) (2023)
- [VELVET: a noVel Ensemble Learning approach to automatically locate VulnErable sTatements](https://mlsanity.com/paper/2112.10893/) (2021)
- [VUDENC: Vulnerability Detection with Deep Learning on a Natural Codebase for Python](https://mlsanity.com/paper/2201.08441/) (2022)
- [Graph Neural Networks: a bibliometrics overview](https://mlsanity.com/paper/2201.01188/) (2022)

## Built on by (most cited first, 8 of 8)

- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [Large Language Models for Cyber Security: A Systematic Literature Review](https://mlsanity.com/paper/2405.04760/) (2024)
- [From Vulnerabilities to Remediation: A Systematic Literature Review of LLMs in Code Security](https://mlsanity.com/paper/2412.15004/) (2024)
- [Do LLMs Consider Security? An Empirical Study on Responses to Programming Questions](https://mlsanity.com/paper/2502.14202/) (2025)
- [An Insight into Security Code Review with LLMs: Capabilities, Obstacles, and Influential Factors](https://mlsanity.com/paper/2401.16310/) (2024)
- [CodeCloak: A Method for Evaluating and Mitigating Code Leakage by LLM Code Assistants](https://mlsanity.com/paper/2404.09066/) (2024)
- [Towards Effective Complementary Security Analysis using Large Language Models](https://mlsanity.com/paper/2506.16899/) (2025)

## Abstract (arXiv)

In recent years, artificial intelligence has had a conspicuous growth in almost every aspect of life. One of the most applicable areas is security code review, in which a lot of AI-based tools and approaches have been proposed. Recently, ChatGPT has caught a huge amount of attention with its remarkable performance in following instructions and providing a detailed response. Regarding the similarities between natural language and code, in this paper, we study the feasibility of using ChatGPT for vulnerability detection in Python source code. Toward this goal, we feed an appropriate prompt along with vulnerable data to ChatGPT and compare its results on two datasets with the results of three widely used Static Application Security Testing tools (Bandit, Semgrep and SonarQube). We implement different kinds of experiments with ChatGPT and the results indicate that ChatGPT reduces the false positive and false negative rates and has the potential to be used for Python source code vulnerability detection.